Unicode issues

[Manuel Mall <ma...@apache.org>]

After having delved into the UAX#14 and SHY issues I am interested in 
compiling a FOP UNICODE issues list. That is a list of things that 
still require work to make FOP Unicode compliant. Obviously the best 
place for such a list is the wiki. But before doing this I am 
interested in some even more informal collection of items on this 
mailing list. Here is a start:

1. The biggest ticket item is probably writing modes and bidi. This is 
clearly a big subproject on its own and outside of the scope I am 
attempting to cover.

2. Unicode text boundaries (UAX#29) especially word boundaries. Do we 
need this? It does not determine the word breaks to which the word 
spacing property is applied to as this is determined by the 
treat-as-word-space property. It could be used to determine the words 
for hyphenation.

3. Normalisation (UAX#15): Do we need this? Do we need to feed words in 
some normalised form to the hyphenation. Other uses for this?

4. Treatment of combining forms: What should / must we do with those 
character combinations?

5. Formatting control: Word joiners etc.. These need at least be 
discarded and not given to the renderers. Obviously proper handling 
when it comes breaking and similar decisions is required for full 
conformance.

Anything else that comes to mind? Please add / comment. If it is enough 
material I'll put it on the wiki.

Thanks

Manuel

Re: Unicode issues

[Simon Pepping <sp...@leverkruid.eu>]

On Sun, Jan 14, 2007 at 11:05:33PM +0100, J.Pietschmann wrote:
> There are libraries which already implement UAX#15 properly, e.g. icu4j,
> but especially icu4j is a rather large blob of a jar. I think Unicode
> normalization should be handled like PDF encryption: do it if the
> library is available, otherwise emit a warning and simply skip the
> step.

That is a good idea. I think we should regard ICU4J as a base library
and not implement anything which is available in it. We may expect
people who want decent Unicode functionality to have the ICU libraries
installed.

Simon

-- 
Simon Pepping
home page: http://www.leverkruid.eu

Re: Unicode issues

["J.Pietschmann" <j3...@yahoo.de>]

Simon Pepping wrote:
> Aren't ligatures a feature of the font,

Yes and no. While the font may provide a glyph, it is the
responsibility of the content rendering code to decide whether
a ligature should be used. Deciding whether a ligature is
applicable is not necessarily trivial, for example there is
usually a ligature for the "ffi" character triple (especially
for slanted fonts), but the word "auffinden" uses only the "fi"
ligature because "auf-" is a prefix for the verb "finden". The
oe ligature in swedish (or danish) has even weirder rules.

The renderer may try to fake a ligature if the font doesn't have
a glyph, or it may just ignore the issue.

> NFC is the standard for the web. Does that carry any weight?

Given that this choice is buried deep in the FOP engine, far
away from any interface intended for average users, I'd say
we can choose whatever we think suites best. It's only a
concern for hyphenation pattern generation and the FOP dev
team.

J.Pietschmann

Re: Unicode issues

[Simon Pepping <sp...@leverkruid.eu>]

On Mon, Jan 15, 2007 at 04:42:12PM +0100, J.Pietschmann wrote:
 
> As for Ligatures and character shaping: an algorithm for automatically
> detecting ligature points may use a pattern lookup similar to the
> pattern based hyphenation. The pattern dictionary should store only
> either NFD or NFC forms, for the same reason this is advisable for
> hyphenation.

Aren't ligatures a feature of the font, e.g. the GSUB table of an Open
Type font? That is, one font may have a specific ligature, while
another font does not.

> We should choose either NFD or NFC as a canonical representation for
> hyphenation patters (and, in the future, for similar things), so that
> hyphenation patterns containing umlauts can be found regardless of
> the representation of the umlaut in the source file. Currently, we
> don't care much, which works but may break suddenly.
> There is obviously a slight space vs. run time tradeoff (NFC ought to
> be more compact but NFC'ing the source text may be more expensive
> than NFD'ing).

NFC is the standard for the web. Does that carry any weight?

Simon

-- 
Simon Pepping
home page: http://www.leverkruid.eu

Re: Unicode issues

["J.Pietschmann" <j3...@yahoo.de>]

Manuel Mall wrote:
>> Font selection in combination with character substitution. Ligatures
>> and character shaping.
> 
> Joerg, can you elaborate on this for me please. 

Fonts may contain glyphs for precomposed Unicode characters, or they
may not. If a list of fonts is searched for a glyph of a character,
it may be useful to look for
- glyphs for the encoded value (which needs the "Grapheme Cluster
   Boundaries" stuff from UAX#29)
- glyphs for the fully decomposed form (UAX#15 NFD)
- glyphs for maximal composition (UAX#15 NFC)

As for Ligatures and character shaping: an algorithm for automatically
detecting ligature points may use a pattern lookup similar to the
pattern based hyphenation. The pattern dictionary should store only
either NFD or NFC forms, for the same reason this is advisable for
hyphenation.

> In unicode an 'umlaut' can be 
> represented as 1 or 2 codepoints. What in your opinion should fop do 
> either a codepoint which can be split into two or vice versa?

We should choose either NFD or NFC as a canonical representation for
hyphenation patters (and, in the future, for similar things), so that
hyphenation patterns containing umlauts can be found regardless of
the representation of the umlaut in the source file. Currently, we
don't care much, which works but may break suddenly.
There is obviously a slight space vs. run time tradeoff (NFC ought to
be more compact but NFC'ing the source text may be more expensive
than NFD'ing).

> I noticed that PDF prints a # for a word joiner for example.

Ouch!

> That's why I 
> thought that most Cf code points should be dealt with in layout and not 
> be passed to the renderers.

It depends on the features of the target format. After all, PDF viewers
do kerning and some paragraph typesetting (e.g. line centering) by
themselves if properly instructed. The SVG flow text also has some
"somewhat higher level" functionality, which users might prefer to be
used. Unfortunately, all this has potential to complicate the FOP
layout.

J.Pietschmann

Re: Unicode issues

[Manuel Mall <ma...@apache.org>]

On Monday 15 January 2007 07:05, J.Pietschmann wrote:
> Manuel Mall wrote:
> > 2. Unicode text boundaries (UAX#29) especially word boundaries. Do
> > we need this? It does not determine the word breaks to which the
> > word spacing property is applied to as this is determined by the
> > treat-as-word-space property. It could be used to determine the
> > words for hyphenation.
>
> Probably not worth the trouble. If the pattern based hyphenator is
> used, words for hyphenation are determined by the character classes
> declared there. Well, unless someone changes this.
>
> > 3. Normalisation (UAX#15): Do we need this? Do we need to feed
> > words in some normalised form to the hyphenation.
>
> Yes, most definitely. This is a major factor in keeping the pattern
> based hyphenator both reasonably robust and small for languages which
> use combined characters, obviously.
>
> > Other uses for this?
>
> Font selection in combination with character substitution. Ligatures
> and character shaping.

Joerg, can you elaborate on this for me please. I am in no way an expert 
on Unicode and any hints are useful. To keep it simple (for us :-)) 
lets stay with German as an example. In unicode an 'umlaut' can be 
represented as 1 or 2 codepoints. What in your opinion should fop do 
either a codepoint which can be split into two or vice versa?

>
> There are libraries which already implement UAX#15 properly, e.g.
> icu4j, but especially icu4j is a rather large blob of a jar. I think
> Unicode normalization should be handled like PDF encryption: do it if
> the library is available, otherwise emit a warning and simply skip
> the step.
>
> Maybe BIDI can be done the same way, using the Java 1.4ff libs  while
> keeping some 1.3 compatibility (just without BIDI).
>
> > 4. Treatment of combining forms: What should / must we do with
> > those character combinations?
> >
> > 5. Formatting control: Word joiners etc.. These need at least be
> > discarded and not given to the renderers.
>
> This depends on the renderer. I'm not sure what PDF would do with it,
> but RTF definitely should get them. While the RTF spec doesn't
> mention anything about this topic, I'd say RTF visualization is done
> using advanced renderers, which should take care of character shaping
> etc. itself.

RTF is a special case renderer as it bypasses layout so it wouldn't be 
affected any way by what I suggest as these are layout functions. I 
noticed that PDF prints a # for a word joiner for example. That's why I 
thought that most Cf code points should be dealt with in layout and not 
be passed to the renderers.

>
> J.Pietschmann

Manuel

Re: Unicode issues

["J.Pietschmann" <j3...@yahoo.de>]

Manuel Mall wrote:
> 2. Unicode text boundaries (UAX#29) especially word boundaries. Do we 
> need this? It does not determine the word breaks to which the word 
> spacing property is applied to as this is determined by the 
> treat-as-word-space property. It could be used to determine the words 
> for hyphenation.

Probably not worth the trouble. If the pattern based hyphenator is used,
words for hyphenation are determined by the character classes declared
there. Well, unless someone changes this.

> 3. Normalisation (UAX#15): Do we need this? Do we need to feed words in 
> some normalised form to the hyphenation.

Yes, most definitely. This is a major factor in keeping the pattern
based hyphenator both reasonably robust and small for languages which
use combined characters, obviously.

> Other uses for this?

Font selection in combination with character substitution. Ligatures
and character shaping.

There are libraries which already implement UAX#15 properly, e.g. icu4j,
but especially icu4j is a rather large blob of a jar. I think Unicode
normalization should be handled like PDF encryption: do it if the
library is available, otherwise emit a warning and simply skip the
step.

Maybe BIDI can be done the same way, using the Java 1.4ff libs  while
keeping some 1.3 compatibility (just without BIDI).

> 4. Treatment of combining forms: What should / must we do with those 
> character combinations?
> 
> 5. Formatting control: Word joiners etc.. These need at least be 
> discarded and not given to the renderers.

This depends on the renderer. I'm not sure what PDF would do with it,
but RTF definitely should get them. While the RTF spec doesn't
mention anything about this topic, I'd say RTF visualization is done
using advanced renderers, which should take care of character shaping
etc. itself.

J.Pietschmann