You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Christian Moen (Created) (JIRA)" <ji...@apache.org> on 2012/03/22 11:00:22 UTC
[jira] [Created] (LUCENE-3901) Add katakana filter to better deal
with katakana spelling variants
Add katakana filter to better deal with katakana spelling variants
------------------------------------------------------------------
Key: LUCENE-3901
URL: https://issues.apache.org/jira/browse/LUCENE-3901
Project: Lucene - Java
Issue Type: New Feature
Components: modules/analysis
Reporter: Christian Moen
Fix For: 3.6, 4.0
Many Japanese katakana words end in a long sound that is sometimes optional.
For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Christian Moen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235746#comment-13235746 ]
Christian Moen commented on LUCENE-3901:
----------------------------------------
Find attached a patch for this.
The stemming is done by {{KuromojiKatakanaStemFilter}}, which has been added to {{KuromojiAnalyzer}} and a corresponding {{KuromojiKatakanaStemFilterFactory}} has been added to the {{text_ja}} field type in {{schema.xml}}.
Note that this stemming is now turned on by default and I think it makes good sense to do so. The minimum length of a token considered for stemming is configurable and I've made the default of 4 explicit in {{schema.xml}} to convey that it's there.
The stemmer only supports full-width katakana and should be used in combination with a {{CJKWidthFilter}} if stemming half-width characters is required and you're doing your wiring. Both {{text_ja}} and {{KuromojiAnalyzer}} takes care of this, and the default overall processing is the same.
There are some test cases in {{TestKuromojiKatakanaStemFilter}}, but I've added a case to {{TestKuromojiAnalyzer}} that demonstrates how the stemming works in combination with katakana compound splitting.
In Japanese, "manager" can be written both as マネージャー and マネージャ (and probably also マネジャー), and for the compound シニアプロジェクトマネージャー (senior project manager), we now get tokens シニア (senior) プロジェクト (project) マネージャ (manager), and we've stemmed the last token by removing the trailing ー. Kuromoji also makes the compound シニアプロジェクトマネージャ a synonym to シニア, and ー is also removed for the synonym compound.
Tests pass and I've also tested this end-to-end in a Solr trunk build.
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch, LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Christian Moen (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Moen resolved LUCENE-3901.
------------------------------------
Resolution: Fixed
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Assignee: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch, LUCENE-3901.patch, LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Christian Moen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235877#comment-13235877 ]
Christian Moen commented on LUCENE-3901:
----------------------------------------
Thanks a lot, Robert.
I'll do some more testing and hopefully I can commit this to {{trunk}} and {{branch_3x}} tomorrow.
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch, LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Christian Moen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Moen updated LUCENE-3901:
-----------------------------------
Attachment: LUCENE-3901.patch
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Christian Moen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237461#comment-13237461 ]
Christian Moen commented on LUCENE-3901:
----------------------------------------
Updated patch with minor whitespace changes to {{schema.xml}} and added an entry in {{CHANGES.txt}}.
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Assignee: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch, LUCENE-3901.patch, LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3901) Add katakana stem
filter to better deal with certain katakana spelling variants
Posted by "Christian Moen (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237475#comment-13237475 ]
Christian Moen edited comment on LUCENE-3901 at 3/24/12 10:48 AM:
------------------------------------------------------------------
Committed revision 1304727 on {{branch_3x}}. Fixed a small javadoc issue in revisions 1304728 and 1304741.
was (Author: cm):
Committed revision 1304727 on {{branch_3x}}. Fixed a small javadoc issue in 1304728.
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Assignee: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch, LUCENE-3901.patch, LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Christian Moen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Moen updated LUCENE-3901:
-----------------------------------
Summary: Add katakana stem filter to better deal with certain katakana spelling variants (was: Add katakana filter to better deal with katakana spelling variants)
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Fix For: 3.6, 4.0
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Christian Moen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Moen updated LUCENE-3901:
-----------------------------------
Attachment: LUCENE-3901.patch
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch, LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Christian Moen (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Moen reassigned LUCENE-3901:
--------------------------------------
Assignee: Christian Moen
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Assignee: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch, LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Christian Moen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Moen updated LUCENE-3901:
-----------------------------------
Attachment: LUCENE-3901.patch
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Assignee: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch, LUCENE-3901.patch, LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235753#comment-13235753 ]
Robert Muir commented on LUCENE-3901:
-------------------------------------
patch looks great!
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch, LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Christian Moen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237475#comment-13237475 ]
Christian Moen commented on LUCENE-3901:
----------------------------------------
Committed revision 1304727 on {{branch_3x}}.
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Assignee: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch, LUCENE-3901.patch, LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Christian Moen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235497#comment-13235497 ]
Christian Moen commented on LUCENE-3901:
----------------------------------------
Patch for this coming up shortly.
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Fix For: 3.6, 4.0
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3901) Add katakana stem
filter to better deal with certain katakana spelling variants
Posted by "Christian Moen (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237475#comment-13237475 ]
Christian Moen edited comment on LUCENE-3901 at 3/24/12 8:10 AM:
-----------------------------------------------------------------
Committed revision 1304727 on {{branch_3x}}. Fixed a small javadoc issue in 1304728.
was (Author: cm):
Committed revision 1304727 on {{branch_3x}}.
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Assignee: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch, LUCENE-3901.patch, LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3901) Add katakana stem filter to better
deal with certain katakana spelling variants
Posted by "Christian Moen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237462#comment-13237462 ]
Christian Moen commented on LUCENE-3901:
----------------------------------------
Committed revision 1304719 on {{trunk}}. Backporting to {{branch_3x}}.
> Add katakana stem filter to better deal with certain katakana spelling variants
> -------------------------------------------------------------------------------
>
> Key: LUCENE-3901
> URL: https://issues.apache.org/jira/browse/LUCENE-3901
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Christian Moen
> Assignee: Christian Moen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3901.patch, LUCENE-3901.patch, LUCENE-3901.patch
>
>
> Many Japanese katakana words end in a long sound that is sometimes optional.
> For example, パーティー and パーティ are both perfectly valid for "party". Similarly we have センター and センタ that are variants of "center" as well as サーバー and サーバ for "server".
> I'm proposing that we add a katakana stemmer that removes this long sound if the terms are longer than a configurable length. It's also possible to add the variant as a synonym, but I think stemming is preferred from a ranking point of view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org