Unicode normalization WAS: svn commit: r1455576 - in /incubator/bloodhound/branches/bep_0003_multiproduct/bloodhound_multiproduct/multiproduct: api.py hooks.py web_ui.py

[Olemis Lang <ol...@gmail.com>]

Now that we have shaped Bloodhound MP into a more robust solution , I
bring this thread to front once again .

On 3/13/13, Branko Čibej <br...@wandisco.com> wrote:
> On 13.03.2013 17:17, Olemis Lang wrote:
>> On 3/13/13, Branko Čibej <br...@wandisco.com> wrote:
[...]
>>> Not to mention that those 4 variants of i have 6 different Unicode
>>> representations. You do *not* want to deal with Unicode normalization
>>> issues in primary keys.
>>>

Definitely sure .

>> Like I just said in my previous message ... we already deal with
>> unicode values in primary keys , so that belongs in the past ...
>

Even if unit tests revealed that product models do support unicode
prefix, functional test cases highlighted that hooks did not ,
therefore they are not actually working in practice .

> Just make triply sure that Trac core actually does normalize the keys
> before writing them to the database.

The patches for #463 have been written with these goals in mind :

  - Equivalence of compatible unicode strings (e.g. NFD) with NFC form
  - Upgrade hooks to deal with unicode prefix

> Otherwise I'd consider that a
> serious bug, because it leaves room for having two identical-looking
> primary keys with different bit values.

  - ... ensure that canonically equivalent unicode strings are bound to the
    NFC form , therefore preventing duplicated prefix issue
    * because, yes , that will definitely happen
    * but note that afaict it's still possible to have identical-looking
      «compatible» product PKs with different bit values

see https://issues.apache.org/bloodhound/ticket/463#comment:5

About this subject I have a few observations and questions and I'd
like to hear further opinions:

  - IMO non-NFC (e.g. NFD) PK URLs have to be redirected (HTTP 301)
    to match the location for the corresponding NFC string.
    * e.g. http://127.0.0.1:8714/products/Ame%CC%81lie
      would be redirected to
      http://127.0.0.1:8714/products/Am%C3%A9lie
    * This is a bit difficult to achieve with current hooks, therefore it's
       still WiP ...
    * Is it ok to keep the non-NFC URL
  - An intermediate step of the NFC normalization process seems
    to be the NFD form .
    * Considering performance, isn't it better to use the later (i.e. NFD)
      as a reference?
  - Is it more accurate to match «compatible» PKs (i.e. NFKC, NFKD) ?

[...]
>
> As far as I can see, Trac core only normalizes the names of attachments.
> That's not enough.
>

You are definitely right . So far I have detected that this might be
an issue with milestone names and wiki pages . Does any other model
have a PK that might be affected by this kind of issues ?

[...]

-- 
Regards,

Olemis - @olemislc

Apache™ Bloodhound contributor
http://issues.apache.org/bloodhound
http://blood-hound.net

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

Re: Unicode normalization WAS: svn commit: r1455576 - in /incubator/bloodhound/branches/bep_0003_multiproduct/bloodhound_multiproduct/multiproduct: api.py hooks.py web_ui.py

[Olemis Lang <ol...@gmail.com>]

On 8/15/13, Branko Čibej <br...@wandisco.com> wrote:
> On 15.08.2013 17:53, Olemis Lang wrote:
>> - An intermediate step of the NFC normalization process seems to be
>> the NFD form . Considering performance, isn't it better to use the
>> later (i.e. NFD) as a reference?
>
> That is an implementation detail of the normalization library. Python
> for example (unicodedata.normalize) doesn't expose that. Although I have
> to say that, in my experience, the normal approach is indeed to convert
> to a stable decomposed form (NFD) first.
>

It seems so , that's why I asked because maybe if a call to
unicodedata.normalize under the hood performs the following
transformations (anything => NFD => NFC) as seems to be suggested by
the definition then relying upon NFD will not trigger the last
conversion step and maybe there will be a slight performance
improvement . Notice that one odf the consequences of adopting
bep:0003 is that product prefix (and therefore normalization) will be
used quite often almost everywhere for almost everything .

> In any case, performance should not be much of an issue here. As long as
> you decide on one normalization form for the database keys, the queries
> themselves will be relatively fast.

Even if I was thinking about conversions as mentioned above , it's
good to know this . I had not really thought about this, just assumed
that the impact at DB level would not be relevant.

> Normalization becomes expensive once
> you start playing with collation algorithms in the databases, and you
> certainly don't want to do that.
>

Test cases for this (if needed) will be written later.

>> - Is it more accurate to match «compatible» PKs (i.e. NFKC, NFKD) ? [...]
>
> That depends on the application. For text processing, the answer is a
> definite "no". For Bloodhound, it's not so clear that the compatibility
> form would be unacceptable. Although if I were a Russian or Greek
> speaker I'd take a dim view if Bloodhound converted my "А" (U+0410) or
> "Α" (U+0391) to "A" (U+0041).
>

For other applications ( e.g. free-text search ) I do see a reason for
considering NFK* forms . However for PKs I'm not really sure , mainly
because of this . Nevertheless

{{{#!py

>>> s = u'\u0410bc'
>>> print s
Аbc
>>> nfc = unicodedata.normalize('NFC', s)
>>> nfc
u'\u0410bc'
>>> nfd = unicodedata.normalize('NFD', s)
>>> nfd
u'\u0410bc'
>>> nfkd = unicodedata.normalize('NFKD', s)
>>> nfkd
u'\u0410bc'
>>> nfkc = unicodedata.normalize('NFKC', s)
>>> nfkc
u'\u0410bc'
>>> unicodedata.normalize('NFKC', u'Abc')
u'Abc'
>>> unicodedata.normalize('NFKC', u'Abc') == unicodedata.normalize('NFKC', s)
False
}}}

This makes me wonder whether they still belong in different
equivalence classes ...

-- 
Regards,

Olemis - @olemislc

Re: Unicode normalization WAS: svn commit: r1455576 - in /incubator/bloodhound/branches/bep_0003_multiproduct/bloodhound_multiproduct/multiproduct: api.py hooks.py web_ui.py

[Branko Čibej <br...@wandisco.com>]

On 15.08.2013 17:53, Olemis Lang wrote:
> - An intermediate step of the NFC normalization process seems to be
> the NFD form . Considering performance, isn't it better to use the
> later (i.e. NFD) as a reference?

That is an implementation detail of the normalization library. Python
for example (unicodedata.normalize) doesn't expose that. Although I have
to say that, in my experience, the normal approach is indeed to convert
to a stable decomposed form (NFD) first.

In any case, performance should not be much of an issue here. As long as
you decide on one normalization form for the database keys, the queries
themselves will be relatively fast. Normalization becomes expensive once
you start playing with collation algorithms in the databases, and you
certainly don't want to do that.

> - Is it more accurate to match «compatible» PKs (i.e. NFKC, NFKD) ? [...]

That depends on the application. For text processing, the answer is a
definite "no". For Bloodhound, it's not so clear that the compatibility
form would be unacceptable. Although if I were a Russian or Greek
speaker I'd take a dim view if Bloodhound converted my "А" (U+0410) or
"Α" (U+0391) to "A" (U+0041).

-- Brane


-- 
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com