You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Tom Burton-West (JIRA)" <ji...@apache.org> on 2011/02/04 21:31:30 UTC
[jira] Created: (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
--------------------------------------------------------------------------------
Key: LUCENE-2906
URL: https://issues.apache.org/jira/browse/LUCENE-2906
Project: Lucene - Java
Issue Type: New Feature
Components: Analysis
Reporter: Tom Burton-West
Priority: Minor
The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177187#comment-13177187 ]
Robert Muir commented on LUCENE-2906:
-------------------------------------
Hi Uwe:
Many filters in lucene currently do things like this, and have forever (including StandardFilter).
In my opinion its ok, as its documented this filter works with StandardTokenizer and ICUTokenizer which use
the interned types.
So I would prefer if we discuss this on another issue.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2906:
--------------------------------
Attachment: LUCENE-2906.patch
one new test and a tweak: so that this filter never calls input.incrementToken() after it already returned false.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2906:
--------------------------------
Fix Version/s: 4.0
3.2
the prerequisite subtask is fixed, so we should be able to add this in 3.2 (supporting StandardTokenizer, UAX29URLEmailTokenizer, and ICUTokenizer) without having to change any of the tokenizers.
I'll update the patch.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Tom Burton-West
> Priority: Minor
> Fix For: 3.2, 4.0
>
> Attachments: LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Tom Burton-West (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990833#comment-12990833 ]
Tom Burton-West commented on LUCENE-2906:
-----------------------------------------
Sounds good to me.
The option to limit to "joined" text also sounds very useful.
Tom
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Tom Burton-West
> Priority: Minor
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-2906) Filter to process
output of ICUTokenizer and create overlapping bigrams for CJK
Posted by "Uwe Schindler (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177097#comment-13177097 ]
Uwe Schindler edited comment on LUCENE-2906 at 12/29/11 10:44 AM:
------------------------------------------------------------------
Hi Robert,
I had no time to review before, there is one small thing that should maybe fixed. Currently this finter relies on the fact that TypeAttribute strings are interned, as it compares by identity:
{code:java}
String type = typeAtt.type();
if (type == doHan || type == doHiragana || type == doKatakana || type == doHangul) {
{code}
This is documented nowhere that Strings in TypeAttribute need to be interned. We should maybe replace that check by a simple equals(). It seems that you already wanted to do that, as you added a sentinel value Object NO = new Object(). -With the above check this sentinel value is useless, a simple null would be enough-. *EDIT:* Sentinel value is also useful for not enabling bigramming is a Tokenizer sets "null" as TypeAttribute. When using equals() this sentinel makes real sense. The check is not costly. String.equals() already does an identity check for early exit, if the sentinel is used it will also quickly return false (if String.equals(sentinel) is used, it will return false on instanceof Check, if you call sentinel.equals(String) it will even be faster).
So I would change this check to:
{code:java}
String type = typeAtt.type();
if (doHan.equals(type) || doHiragana.equals(type) || doKatakana.equals(type) || doHangul.equals(type)) {
{code}
(this is the fastest check, if the doXXX is the sentinel, it's default Object.equals() will return false. If its a string, String.equals() will return true on identity very quick, but if it's not interned it will be slower. So we loose nothing but dont require useless interned strings.
was (Author: thetaphi):
Hi Robert,
I had no time to review before, there is one small thing that should maybe fixed. Currently this finter relies on the fact that TypeAttribute strings are interned, as it compares by identity:
{code:java}
String type = typeAtt.type();
if (type == doHan || type == doHiragana || type == doKatakana || type == doHangul) {
{code}
This is documented nowhere that Strings in TypeAttribute need to be interned. We should maybe replace that check by a simple equals(). It seems that you already wanted to do that, as you added a sentinel value Object NO = new Object(). With the above check this sentinel value is useless, a simple null would be enough. But when using equals() this sentinel makes sense. The check is not costly. String.equals() already does an identity check for early exit, if the sentinel is used it will also quickly return false (if String.equals(sentinel) is used, it will return false on instanceof Check, if you call sentinel.equals(String) it will even be faster).
So I would change this check to:
{code:java}
String type = typeAtt.type();
if (doHan.equals(type) || doHiragana.equals(type) || doKatakana.equals(type) || doHangul.equals(type)) {
{code}
(this is the fastest check, if the doXXX is the sentinel, it's default Object.equals() will return false. If its a string, String.equals() will return true on identity very quick, but if it's not interned it will be slower. So we loose nothing but dont require useless interned strings.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved LUCENE-2906.
---------------------------------
Resolution: Fixed
Committed, maybe in the future we enable this for StandardFilter (for good CJK behavior by default), but for now it seems good enough to have separate filters that handle the corner cases and all of unicode.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177097#comment-13177097 ]
Uwe Schindler commented on LUCENE-2906:
---------------------------------------
Hi Robert,
I had no time to review before, there is one small thing that should maybe fixed. Currently this finter relies on the fact that TypeAttribute strings are interned, as it compares by identity:
{code:java}
String type = typeAtt.type();
if (type == doHan || type == doHiragana || type == doKatakana || type == doHangul) {
{code}
This is documented nowhere that Strings in TypeAttribute need to be interned. We should maybe replace that check by a simple equals(). It seems that you already wanted to do that, as you added a sentinel value Object NO = new Object(). With the above check this sentinel value is useless, a simple null would be enough. But when using equals() this sentinel makes sense. The check is not costly. String.equals() already does an identity check for early exit, if the sentinel is used it will also quickly return false (if String.equals(sentinel) is used, it will return false on instanceof Check, if you call sentinel.equals(String) it will even be faster).
So I would change this check to:
{code:java}
String type = typeAtt.type();
if (doHan.equals(type) || doHiragana.equals(type) || doKatakana.equals(type) || doHangul.equals(type)) {
{code}
(this is the fastest check, if the doXXX is the sentinel, it's default Object.equals() will return false. If its a string, String.equals() will return true on identity very quick, but if it's not interned it will be slower. So we loose nothing but dont require useless interned strings.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2906:
--------------------------------
Attachment: LUCENE-2906.patch
here's a patch going in a slightly different direction (though we can still add some special icu-only stuff here).
instead the patch synchronizes the token types of ICUTokenizer with StandardTokenizer, adds the necessarily types to both, and then adds the bigramming logic to standardfilter.
this way, cjk works easily "out of box", for all of unicode (e.g. supplementaries) and plays well with other languages. i deprecated cjktokenizer in the patch and pulled out its special full-width filter into a separate tokenfilter.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Tom Burton-West
> Priority: Minor
> Attachments: LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991186#comment-12991186 ]
Robert Muir commented on LUCENE-2906:
-------------------------------------
{quote}
How will this differ from the SmartChineseAnalyzer?
{quote}
The SmartChineseAnalyzer is for Simplified Chinese only... this is about the
language-independent technique similar to what CJKAnalyzer does today.
{quote}
I doubt it but can this be in 3.1?
{quote}
Well i hate the way CJKAnalyzer treats things like supplementary characters (wrongly).
This is definitely a bug, and fixed here. Part of me wants to fix this as quickly as possible.
At the same time though, I would prefer 3.2... otherwise I would feel like I am rushing things.
I don't think 3.2 needs to come a year after 3.1... in fact since we have a stable branch I think its
stupid to make bugfix releases like 3.1.1 when we could just push out a new minor version (3.2) with
bugfixes instead. The whole branch is intended to be stable changes, so I think this is better use
of our time. But this is just my opinion, we can discuss it later on the list as one idea to promote
more rapid releases.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Tom Burton-West
> Priority: Minor
> Attachments: LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "DM Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991169#comment-12991169 ]
DM Smith commented on LUCENE-2906:
----------------------------------
Two questions:
How will this differ from the SmartChineseAnalyzer?
I doubt it but can this be in 3.1?
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Tom Burton-West
> Priority: Minor
> Attachments: LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2906:
---------------------------------------
Fix Version/s: (was: 3.4)
3.5
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Priority: Minor
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177201#comment-13177201 ]
Robert Muir commented on LUCENE-2906:
-------------------------------------
I created LUCENE-3669 for the broader interned-type issue
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171318#comment-13171318 ]
Robert Muir commented on LUCENE-2906:
-------------------------------------
As much as I would like this to work as the patch does (where its automatic from StandardFilter), I think its bad because its something we then have to commit to/deal with for a while (e.g. backwards compat).
So another idea is just to call it CJKFilter or something under the CJK package for now. We could still cutover CJKAnalyzer like the patch and then it finally works with supplementary characters too (which I think is really long needed).
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177101#comment-13177101 ]
Uwe Schindler commented on LUCENE-2906:
---------------------------------------
...alternatively, we could use a HashSet<String>. If the stringsin it are intered, the lookup is fast, too. The hashCode of Strings is precalculated in the String class. For four if checks it maybe not really different performance wise, but thats just another idea. The ctor would simply check the flags and add the type strings to the Set<String>.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990739#comment-12990739 ]
Robert Muir commented on LUCENE-2906:
-------------------------------------
I'll take it, ive done the unibigram approach already (maybe we can just have it as a separate filter option), so the bigram should be easy.
My original design, just lets you provide a BitSet of script codes. (this would be simple i think to parse from say a solr factory).
I think its also useful to have an option, for whether the filter should only do this for "joined" text or not (based on offsets). For CJK i think it makes sense to enforce this, so that it won't bigram across sentence boundaries. But for say the Tibetan language, where you have a syllable separator, you would want to turn this off.
Separately, if you want it to work "just like" CJKTokenizer, please be aware that by default, the unicode standard tokenizes Katakana to words (only hiragana and han are tokenized to codepoints). So in this case you would have to use a custom ruleset if you wanted katakana to be tokenized to codepoints instead of words, for later bigramming. I'm not sure you want to do this though... (in truth CJKTokenizer bigrams ANYTHING out of ascii, including a lot of things it shouldnt).
For hangul the same warning applies, but its more debatable, you might want to do this if you don't have a decompounder... but in my opinion this is past tokenization, and its the same problem you have with german, etc... the default tokenization is not "wrong".
In either case, if you decide to do that, it would be a pretty simple ruleset!
Let me know if this makes sense to you.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis
> Reporter: Tom Burton-West
> Priority: Minor
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2906:
--------------------------------
Attachment: LUCENE-2906.patch
patch removing all nocommits with additional tests.
I think its ready to commit.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-2906.patch, LUCENE-2906.patch, LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2906:
--------------------------------
Attachment: LUCENE-2906.patch
synced up to trunk... has a couple minor nocommits (mostly just some needed tests) I'll look at tomorrow morning.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-2906.patch, LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171356#comment-13171356 ]
Robert Muir commented on LUCENE-2906:
-------------------------------------
sorry to take so long Tom... ill round this out tonight.
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Tom Burton-West (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094902#comment-13094902 ]
Tom Burton-West commented on LUCENE-2906:
-----------------------------------------
Any chance this might get implemented for 3.4?
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Priority: Minor
> Fix For: 3.4, 4.0
>
> Attachments: LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-2906) Filter to process output of
ICUTokenizer and create overlapping bigrams for CJK
Posted by "Robert Muir (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir reassigned LUCENE-2906:
-----------------------------------
Assignee: Robert Muir
> Filter to process output of ICUTokenizer and create overlapping bigrams for CJK
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2906
> URL: https://issues.apache.org/jira/browse/LUCENE-2906
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Tom Burton-West
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-2906.patch
>
>
> The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org