
		eXtended PC Font Format (xpsf) -- draft specifications
		======================================================

		    (C) 1997,98,99 Yann Dirson <dirson@debian.org>


IMPORTANT NOTICE:

  This document is a draft for a file-format.  Please do not use it
yet to implement any software.  You may do what you want with it
according to the GPL, version 2, except from redistributing a modified
version with the same name without permission.


0. Changes

1999/05/06: dropped the format_version field, obsoleted by bitfields.
1999/05/06: redesigned the bitfield mechanism to use section ID's.
1999/05/06: misc reformulations
1998/10/08: replaced version-based compatibility mechanism with bitfields
	as suggested by hpa.
1998/10/08: reduced size of section-id from u16 to u8.
1998/10/08: compressed syntax description by using type definitions inside
	structure definitions.
1998/10/08: added section for copyright and license.
1998/10/08: fixed section-ids: obsolete xpsf_console_override_map was
	still allocated.
1998/10/08: mentionned VGA pseudo-9bit-wide characters.
1998/10/08: proposed a 4-byte magic.
1998/10/08: don't allow a glyph to have no UTF8 mapping.
1998/10/08: use UTF8 instead of UCS2 to describe glyphs.
1998/08/06: `=' in the description grammar was ambiguous - replaced one of
	the 2 meanings with `:'.
1998/08/06: changed how integer types are named.
1998/08/06: added note that virtual fonts may not be necessary.
1998/04/20: updated author's e-mail.
1998/04/20: changed ref from libconsole to libcfont.
1998/04/20: suggested more width for magic.
1998/04/20: changed section name from xpsf_font_data to xpsf_fixed_font_data.
1998/04/20: minor clarifications.
1997/09/27: removed console-maps sections - I finally understand what they
	are for.
1997/09/04: initial draft release.
	

1. Summary

 The future XPSF format is intended to be a replacement for all
current font-file formats used with the console utilities ('kbd' and
`console-tools').  For this, it should have all features from PSF,
cpi, and cp file-formats, and more.

 Among these:
 
a * Should have a magic number for easy file-format identification.
b * Should be easily extendable (esp. should feature some kind of file-format
  version nb), while remaining as readable as possible for any program knowing
  of an old version.  If however the file could be misinterpreted by a program
  not knowing about the new features, there should be a mean to warn the program.
b2* Similarly, backward compatibility has to be ensured, but the existence in
  older version of the file-format of obsolete section-types should not force
  a new program to support these obsolete sections.
c * Multiple font-sizes (both in height and width) in one file.
d * Unicode-Glyph map shared by all contained font-sizes.
e * File-format should be machine-independant.
f * Allow more than UCS2 for unimap (eg. UCS4).
g * File format should be described by a sort of context-free grammar (ie. no
  chained data).
h * All sections of a file should tell their length, in order to skip them
  easily (idea from Amiga's IFF file format).
i * Font-size specific unicode-map overrides, for cases where eg. some
  characters are (hopefully temporarily) not available at one size).
j * Maybe provide virtual fonts support, to avoid character-description
  duplication.  For example, building individual isoXX/codepages by mapping
  them into a larger font with a custom encoding will save the storage place
  needed by all those duplicated characters.  This one may not be necessary,
  and may be dropped in the final spec.
k * Provide enough info, whatever redundant, to allow just-sufficient
  malloc's while reading.
l * Make it possible to use textmode-VGA pseudo-9bit fonts.
m * Copyright and redistribution info MUST be included.  Too many fonts are
  available on the net without such information, and are thus legally unusable.

 Functions to manipulate files in the XPSF format will be included in
libcfont, as a reference implementation.  Tools to handle them and
convert to/from XPSF will be provided with the console utilities.


2. Possible solutions.

a * Suggest variation on PSF magic number (which is 0x36 0x04).  A 4-byte magic
  will lower the risk of misinterpreting random data as XPSF.
  0x36 0x05 0x84 0xBF seems unassigned.

b * Extendability can be achieved through file sectionning, with identifiers
  on sections telling their contents.  Upward readability can be achieved by
  additionnaly providing the length of each header and associated data part
  at the very start of each header; backward compatibility will also use
  header length.

    Sections will be listed in this specification, and will each be assigned
  a bit in a bitfield structure, according to their section ID (a section of
  id S will use bit S in the bitfield - bitmask (1<<S)).  An XPSF file will
  have 2 instances of this bitfield structure, used to respectively declare
  "R/O compatibility" and "incompatibility" of a reader not knowing of these
  features:
    When a reader does not support a feature declared as "R/O compatibility"
  in a file it will know that it can read this file, at the expense of not
  understanding all of the info in it (it should then not attempt to modify
  this file), but will still correctly understand the infos it could read.
    When a reader does not support a feature tagged with "incompatibility",
  it will know that even when attempting to read the stuff it thinks it
  could understand, it would mis-interpret them because of the missing
  feature.
    Bitfields won't be included in the whole, but for each, the useful width
  in bytes will be specified, followed by the declared number of least signifant
  bytes of the field.  With only a byte to declare a length, that leaves 256*8
  = 2048 possible bits, which should be more than we'll ever need.

b2* Each section-type may become obsolete at any time; when it occurs it
  will be documented in a special section of the XPSF specs, telling the
  first version of the specs when the section was made obsolete.
    A reading program declaring to implement a given version of the file
  format will be allowed not to implement features obsolete wrt the version
  it declares, and the mechanism described above will allow to detect old
  files that should not be read.
   
c * Multiple font-sizes can be included as multiple sections of the
  "font-data" type, which will include information on size (both height and
  width, as some system - including Linux 2.2.x - allow widths different
  from 8).

d * Unicode mapping can be described in specific sections.  Each font_data
  section will have references to the corresponding map in the file.  This
  will even allow multiple maps in one file, FWIW.

e * Machine-independance: use "network byte order" as defined by the
  htons() and htonl() library calls.

f * UCS2, as shown in the PSF format, causes some waste of space for
  common encodings, mainly due to the fact that we need to use 0xFFFF as
  a separator.  Using UTF8 to describe the mappings will allow to use
  only 0xFF as a separator.  The ASCII set will give us a small benefit
  too, being encoded in 1 byte instead of 2, and most widly-used chars
  will stay on 2 bytes (latin, cyrillic, arabic, hebrew...), only more
  exotic chars will take one more byte.  This will also make it possible
  to use chars outside of the Basic Multilingal Plane (ie. UCS2).

g * No problem.

h * see `b'.

i * Can be defined as a section refering to the map to be overriden, and
  containing a list of override entries.

j * [may be removed at any time]
    A 1st step would be to provide virtual mapping into a single font
  (possibly larger).  The virtual-font section will provide a reference to a
  single font-file, and an array of size <font_size> of character-positions
  telling for each fontpos in the virtual font, in which fontpos the
  corresponding glyph should be found.  The virtual-font section should be
  handled by programs as as-many font-data sections that the target font-file
  contains, using the relevant font_height and font_width.
    To save space and have some flexibility, the references to the target
  characters will be allowed in several sizes ; the chosen size will be
  specified in the section as the number of bits.  It is expected that this
  size will be a power of 2 compatible with available computers (ie., in
  1997: 8, 16, 32), but this is not enforced, and programs may make use of
  other sizes, but then SHOULD NOT rely on third-party XPSF-aware programs to
  be able to read them.
    Should the need arise, as an XPSF-file can eventually contain several
  fonts with different mappings, fields could be added in the virtual-font
  to specify, if any, a mapping in the target file on which to restrict
  target font-data sections (ie. only target sections with this encoding will
  be seen by the virtual-font section).  But placing different mappings in the
  same XPSF file is discouraged, unless you really have use of it; probably
  override-maps should be sufficient for most usages.

k * a "count" field has to be available before every variable-size array.

l * these can be encoded as standard 9-bit fonts.  The reading
  software will check for VGA-ification and take care of possibly
  reordering the chars.

m * a specific section type describe copyright and license.  It is the mandatory
  first section in an XPSF file.  We distinguish the copyright string, such as
  "(c) 1997,98 John Doe and others", and the license, such as "GPL" or a more
  descriptive text.  These strings shall be UTF8-encoded to ensure they won't
  be erroneously displayed using an incompatible charset.
    Maybe it should be allowed for an OS or a software package to install a
  separate text file describing the license.  In this case the `license' field in
  the XPSF file should be an absolute path to this file (to be distinguished
  by the leading `/' (slash) character), and the default behavior of the reading
  software if this file is not available should be to fail to read the font,
  with an explicit message.  A 'force' option can be provided to force
  installation of the font.



3. Draft specifications

 The file format is described here in sort-of EBNF notation.
Upper-case WORDS represent terminal symbols, ie. (mostly) C types;
lower-case words represent non-terminal symbols, ie. symbols defined
in terms of other symbols.
 sym1 sym2	is sym1 followed by sym2
 sym1 | sym2	is either sym1 or sym2
 [sym]		is an optional symbol
 {sym}		is a symbol that can be repeated 0 or more times
 {sym}*N	is a symbol that must be repeated N times
 Comments are introduced with a # sign.

 The integer types are specified using their length expressed in decimal
notation, prefixed with `u' for unsigned types, or `s' for signed types.
Eg, `u16' denotes a "16-bit unsigned integer".

#

xpsf_file :
		xpsf_header
		xpsf_section		# id = xpsf_copyright
		{xpsf_section}

xpsf_header :
		magic_id
		header_length : u16
		font_size
		nb_sections : u16
		features_incompat : large_bitfield
		features_ro_compat : large_bitfield

# magic number proposal:
magic_id :
		u8 = 0x36
		u8 = 0x05
		u8 = 0x84
		u8 = 0xBF

font_size :
		u32		# number of chars in font - enough for UCS4 ;)
				# note that as data_length is a u32 too, not all
				# 16M chars of such a font would fit in one section.
				# A new file-format will probably be needed before that :)

large_bitfield :
		real_size_1: u8	# effective width in bytes, minus 1
		{BIT}*<8*(real_size_1+1)>

#

xpsf_section :
		xpsf_section_header
		xpsf_section_data

xpsf_section_header :
		header_length = 8
		data_length : u32
		section_id : u8

xpsf_section_data :
		  xpsf_copyright		# id = 0x00
		| xpsf_fixed_font_data		# id = 0x01
		| xpsf_UTF8_map			# id = 0x02
		| xpsf_UTF8_override_map	# id = 0x03
(		| xpsf_virtual_font_data	# id = 0x04)


# Copyright and distribution licence
# Appeared in release: 1

xpsf_copyright :
		copyright : string
		distribution_licence : string

string :
		string_length : u16
		{UTF8}*<string_length>
		
# A font data of a particular size
# Appeared in release: 1

xpsf_fixed_font_data :
		char_height : u8
		char_width : u8
		{{BIT}*<char_width>}*<char_height>
		UTF8_map_ref : file_offset

file_offset :
		u32		


# Unicode fontpos-to-UTF8 map
# Appeared in release: 1

xpsf_UTF8_map :
		{ UTF8_array psf_separator }*<font_size>

UTF8_array :
		UTF8			# at least one.
		{ UTF8 }		# any necessary number of times

UTF8 :
		u8			# at least 1 byte
		{u8}

psf_separator :
		u8 = 0xFF

# Unicode fontpos-to-UTF8 override-map
# Appeared in release: 1

xpsf_UTF8_override_map :
		parent_map : file_offset
		nb_entries : u16
		fontpos_size
		{ fontpos UTF8 {UTF8} psf_separator }*<nb_entries>

fontpos_size :
		u8			# typically 8, 16 or 32

fontpos :
		{BIT}*<fontpos_size>

# Virtual-font
# May not appear in release 1

xpsf_virtual_font_data :
		target_name_length : u16
		target_name
		fontpos_size
		{ fontpos }*<font_size>

target_name :
		{u8}*<target_name_length - 1>
		u8=0x00
