You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2010/04/20 15:52:52 UTC
[jira] Commented: (LUCENE-1343) A replacement for
AsciiFoldingFilter that does a more thorough job of removing diacritical
marks or non-spacing modifiers.
[ https://issues.apache.org/jira/browse/LUCENE-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858891#action_12858891 ]
Robert Muir commented on LUCENE-1343:
-------------------------------------
By the way, I have been running this with the ASCIIFoldingFilter tests and ensuring its a superset (e.g. we have at least all their mappings).
But there are some bugs in ASCIIFoldingFilter that should be fixed:
For example, U+1E9B (LATIN SMALL LETTER LONG S WITH DOT ABOVE)
But in unicode. this is canonically equivalent to U+017F (LONG S) U+0307 (COMBINING DOT ABOVE)
AsciiFoldingFilter folds U+1E9B (LONG S WITH DOT) to an F
but it folds U+017F (LONG S) to an S
Unicode defines this character as a compatibility equivalent to S anyway, but its worse that ASCIIFoldingFilter is canonically inconsistent with itself.
> A replacement for AsciiFoldingFilter that does a more thorough job of removing diacritical marks or non-spacing modifiers.
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1343
> URL: https://issues.apache.org/jira/browse/LUCENE-1343
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.1
> Reporter: Robert Haschart
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-1343.patch, normalizer.jar, UnicodeCharUtil.java, UnicodeNormalizationFilter.java, UnicodeNormalizationFilterFactory.java, utr30.nrm
>
>
> The ISOLatin1AccentFilter takes Unicode characters that have diacritical marks and replaces them with a version of that character with the diacritical mark removed. For example é becomes e. However another equally valid way of representing an accented character in Unicode is to have the unaccented character followed by a non-spacing modifier character (like this: é ) The ISOLatin1AccentFilter doesn't handle the accents in decomposed unicode characters at all. Additionally there are some instances where a word will contain what looks like an accented character, that is actually considered to be a separate unaccented character such as Ł but which to make searching easier you want to fold onto the latin1 lookalike version L .
> The UnicodeNormalizationFilter can filter out accents and diacritical marks whether they occur as composed characters or decomposed characters, it can also handle cases where as described above characters that look like they have diacritics (but don't) are to be folded onto the letter that they look like ( Ł -> L )
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org