You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Hiroaki Kawai (JIRA)" <ji...@apache.org> on 2008/03/11 06:28:46 UTC
[jira] Created: (LUCENE-1215) Support of Unicode Collation
Support of Unicode Collation
----------------------------
Key: LUCENE-1215
URL: https://issues.apache.org/jira/browse/LUCENE-1215
Project: Lucene - Java
Issue Type: New Feature
Components: Analysis
Reporter: Hiroaki Kawai
Attachments: NormalizerTokenFilter.java
New in java 6, we have java.text.Normalizer that supports Unicode Standard Annex #15 normalization.
http://java.sun.com/javase/6/docs/api/java/text/Normalizer.html
http://www.unicode.org/unicode/reports/tr15/
The normalization defined has four variants of C, D, KC, KD. Canonical Decomposition or Compatibility Decomposition will be normalize the representation of a String, and the search result will be improved.
I'd like to submit a TokenFilter code supporting this feature! :-)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1215) Support of Unicode Collation
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved LUCENE-1215.
---------------------------------
Resolution: Duplicate
I am marking this resolved as a duplicate of LUCENE-2399.
The new ICU Normalizer2 offers high performance, and flexibility (you can integrate custom mappings, etc).
> Support of Unicode Collation
> ----------------------------
>
> Key: LUCENE-1215
> URL: https://issues.apache.org/jira/browse/LUCENE-1215
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Hiroaki Kawai
> Attachments: NormalizerTokenFilter.java
>
>
> New in java 6, we have java.text.Normalizer that supports Unicode Standard Annex #15 normalization.
> http://java.sun.com/javase/6/docs/api/java/text/Normalizer.html
> http://www.unicode.org/unicode/reports/tr15/
> The normalization defined has four variants of C, D, KC, KD. Canonical Decomposition or Compatibility Decomposition will be normalize the representation of a String, and the search result will be improved.
> I'd like to submit a TokenFilter code supporting this feature! :-)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1215) Support of Unicode Collation
Posted by "Andrew Lynch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12577312#action_12577312 ]
Andrew Lynch commented on LUCENE-1215:
--------------------------------------
This will be quite useful. I used the Normalizer to implement my own custom analyzer for https://issues.apache.org/jira/browse/LUCENE-1032.
There is actually a Normalizer equivalent in older versions of the Sun JDK, sun.text.Normalizer, but this obviously wouldn't end up being portable across VMs.
I ended up using reflection to determine the presence of Normalizer if it existed, then fell back to sun.text.Normalizer, then finally performing no normalization if neither could be found to preserve compatibility with non Java 6/ Sun JDKs.
> Support of Unicode Collation
> ----------------------------
>
> Key: LUCENE-1215
> URL: https://issues.apache.org/jira/browse/LUCENE-1215
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Hiroaki Kawai
> Attachments: NormalizerTokenFilter.java
>
>
> New in java 6, we have java.text.Normalizer that supports Unicode Standard Annex #15 normalization.
> http://java.sun.com/javase/6/docs/api/java/text/Normalizer.html
> http://www.unicode.org/unicode/reports/tr15/
> The normalization defined has four variants of C, D, KC, KD. Canonical Decomposition or Compatibility Decomposition will be normalize the representation of a String, and the search result will be improved.
> I'd like to submit a TokenFilter code supporting this feature! :-)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1215) Support of Unicode Collation
Posted by "Hiroaki Kawai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hiroaki Kawai updated LUCENE-1215:
----------------------------------
Attachment: NormalizerTokenFilter.java
> Support of Unicode Collation
> ----------------------------
>
> Key: LUCENE-1215
> URL: https://issues.apache.org/jira/browse/LUCENE-1215
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Hiroaki Kawai
> Attachments: NormalizerTokenFilter.java
>
>
> New in java 6, we have java.text.Normalizer that supports Unicode Standard Annex #15 normalization.
> http://java.sun.com/javase/6/docs/api/java/text/Normalizer.html
> http://www.unicode.org/unicode/reports/tr15/
> The normalization defined has four variants of C, D, KC, KD. Canonical Decomposition or Compatibility Decomposition will be normalize the representation of a String, and the search result will be improved.
> I'd like to submit a TokenFilter code supporting this feature! :-)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org