You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2009/10/08 18:10:31 UTC
[jira] Created: (LUCENE-1963) ArabicAnalyzer: Lowercase before
Stopfilter
ArabicAnalyzer: Lowercase before Stopfilter
-------------------------------------------
Key: LUCENE-1963
URL: https://issues.apache.org/jira/browse/LUCENE-1963
Project: Lucene - Java
Issue Type: Improvement
Components: contrib/analyzers
Affects Versions: 2.9
Reporter: Robert Muir
Assignee: Robert Muir
Priority: Trivial
Attachments: LUCENE-1963.patch
ArabicAnalyzer lowercases text in case you have some non-Arabic text around.
It also allows you to set a custom stopword list (you might augment the Arabic list with some English ones, for example).
In this case its helpful for these non-Arabic stopwords, to lowercase before stopfilter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1963) ArabicAnalyzer: Lowercase before
Stopfilter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-1963:
--------------------------------
Attachment: LUCENE-1963_branch.patch
its been a few days, no one objected to applying this fix to the branch.
but I do not have permissions to commit to the branch... can someone commit this for me? Attached is the patch.
> ArabicAnalyzer: Lowercase before Stopfilter
> -------------------------------------------
>
> Key: LUCENE-1963
> URL: https://issues.apache.org/jira/browse/LUCENE-1963
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 2.9
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Fix For: 2.9.1, 3.0
>
> Attachments: LUCENE-1963.patch, LUCENE-1963.patch, LUCENE-1963_branch.patch
>
>
> ArabicAnalyzer lowercases text in case you have some non-Arabic text around.
> It also allows you to set a custom stopword list (you might augment the Arabic list with some English ones, for example).
> In this case its helpful for these non-Arabic stopwords, to lowercase before stopfilter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1963) ArabicAnalyzer: Lowercase before
Stopfilter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763562#action_12763562 ]
Robert Muir commented on LUCENE-1963:
-------------------------------------
bq. can you commit it to 2.9.1 too? (For those stuck on Java 1.4, there is no 3.0).
can someone comment on this one for me.
I don't think its too much of a stretch to consider this a bug, even if it does not affect Arabic text.
> ArabicAnalyzer: Lowercase before Stopfilter
> -------------------------------------------
>
> Key: LUCENE-1963
> URL: https://issues.apache.org/jira/browse/LUCENE-1963
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 2.9
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Fix For: 3.0
>
> Attachments: LUCENE-1963.patch, LUCENE-1963.patch
>
>
> ArabicAnalyzer lowercases text in case you have some non-Arabic text around.
> It also allows you to set a custom stopword list (you might augment the Arabic list with some English ones, for example).
> In this case its helpful for these non-Arabic stopwords, to lowercase before stopfilter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1963) ArabicAnalyzer: Lowercase before
Stopfilter
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764013#action_12764013 ]
Mark Miller commented on LUCENE-1963:
-------------------------------------
Your issue - if you can stretch it to bugish territory, I'd +1 it. I'd be wary of getting into porting features to 2.9.1 - but I wouldn't have a problem with this one myself.
> ArabicAnalyzer: Lowercase before Stopfilter
> -------------------------------------------
>
> Key: LUCENE-1963
> URL: https://issues.apache.org/jira/browse/LUCENE-1963
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 2.9
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Fix For: 3.0
>
> Attachments: LUCENE-1963.patch, LUCENE-1963.patch
>
>
> ArabicAnalyzer lowercases text in case you have some non-Arabic text around.
> It also allows you to set a custom stopword list (you might augment the Arabic list with some English ones, for example).
> In this case its helpful for these non-Arabic stopwords, to lowercase before stopfilter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1963) ArabicAnalyzer: Lowercase before
Stopfilter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-1963:
--------------------------------
Fix Version/s: 3.0
if no one objects, I'd like to commit this for 3.0 at the end of the day.
> ArabicAnalyzer: Lowercase before Stopfilter
> -------------------------------------------
>
> Key: LUCENE-1963
> URL: https://issues.apache.org/jira/browse/LUCENE-1963
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 2.9
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Fix For: 3.0
>
> Attachments: LUCENE-1963.patch
>
>
> ArabicAnalyzer lowercases text in case you have some non-Arabic text around.
> It also allows you to set a custom stopword list (you might augment the Arabic list with some English ones, for example).
> In this case its helpful for these non-Arabic stopwords, to lowercase before stopfilter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1963) ArabicAnalyzer: Lowercase before
Stopfilter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved LUCENE-1963.
---------------------------------
Resolution: Fixed
Committed revision 823534.
(if it is ok to apply this to 2.9 branch as DM requested, we should reopen)
> ArabicAnalyzer: Lowercase before Stopfilter
> -------------------------------------------
>
> Key: LUCENE-1963
> URL: https://issues.apache.org/jira/browse/LUCENE-1963
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 2.9
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Fix For: 3.0
>
> Attachments: LUCENE-1963.patch, LUCENE-1963.patch
>
>
> ArabicAnalyzer lowercases text in case you have some non-Arabic text around.
> It also allows you to set a custom stopword list (you might augment the Arabic list with some English ones, for example).
> In this case its helpful for these non-Arabic stopwords, to lowercase before stopfilter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1963) ArabicAnalyzer: Lowercase before
Stopfilter
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1963.
----------------------------------------
Resolution: Fixed
Committed on 2.9.x. Thanks Robert!
> ArabicAnalyzer: Lowercase before Stopfilter
> -------------------------------------------
>
> Key: LUCENE-1963
> URL: https://issues.apache.org/jira/browse/LUCENE-1963
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 2.9
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Fix For: 2.9.1, 3.0
>
> Attachments: LUCENE-1963.patch, LUCENE-1963.patch, LUCENE-1963_branch.patch
>
>
> ArabicAnalyzer lowercases text in case you have some non-Arabic text around.
> It also allows you to set a custom stopword list (you might augment the Arabic list with some English ones, for example).
> In this case its helpful for these non-Arabic stopwords, to lowercase before stopfilter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1963) ArabicAnalyzer: Lowercase before
Stopfilter
Posted by "DM Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763554#action_12763554 ]
DM Smith commented on LUCENE-1963:
----------------------------------
can you commit it to 2.9.1 too? (For those stuck on Java 1.4, there is no 3.0).
> ArabicAnalyzer: Lowercase before Stopfilter
> -------------------------------------------
>
> Key: LUCENE-1963
> URL: https://issues.apache.org/jira/browse/LUCENE-1963
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 2.9
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Fix For: 3.0
>
> Attachments: LUCENE-1963.patch, LUCENE-1963.patch
>
>
> ArabicAnalyzer lowercases text in case you have some non-Arabic text around.
> It also allows you to set a custom stopword list (you might augment the Arabic list with some English ones, for example).
> In this case its helpful for these non-Arabic stopwords, to lowercase before stopfilter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1963) ArabicAnalyzer: Lowercase before
Stopfilter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-1963:
--------------------------------
Attachment: LUCENE-1963.patch
here also update the javadocs to reflect the new order of what is going on in ArabicAnalyzer, to prevent any confusion to users.
> ArabicAnalyzer: Lowercase before Stopfilter
> -------------------------------------------
>
> Key: LUCENE-1963
> URL: https://issues.apache.org/jira/browse/LUCENE-1963
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 2.9
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Fix For: 3.0
>
> Attachments: LUCENE-1963.patch, LUCENE-1963.patch
>
>
> ArabicAnalyzer lowercases text in case you have some non-Arabic text around.
> It also allows you to set a custom stopword list (you might augment the Arabic list with some English ones, for example).
> In this case its helpful for these non-Arabic stopwords, to lowercase before stopfilter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1963) ArabicAnalyzer: Lowercase before
Stopfilter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-1963:
--------------------------------
Attachment: LUCENE-1963.patch
simple patch, but will need to warn in CHANGES.txt that folks should reindex, if they are using non-Arabic stopwords.
> ArabicAnalyzer: Lowercase before Stopfilter
> -------------------------------------------
>
> Key: LUCENE-1963
> URL: https://issues.apache.org/jira/browse/LUCENE-1963
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 2.9
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Attachments: LUCENE-1963.patch
>
>
> ArabicAnalyzer lowercases text in case you have some non-Arabic text around.
> It also allows you to set a custom stopword list (you might augment the Arabic list with some English ones, for example).
> In this case its helpful for these non-Arabic stopwords, to lowercase before stopfilter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1963) ArabicAnalyzer: Lowercase before
Stopfilter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-1963:
--------------------------------
Fix Version/s: 2.9.1
> ArabicAnalyzer: Lowercase before Stopfilter
> -------------------------------------------
>
> Key: LUCENE-1963
> URL: https://issues.apache.org/jira/browse/LUCENE-1963
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 2.9
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Fix For: 2.9.1, 3.0
>
> Attachments: LUCENE-1963.patch, LUCENE-1963.patch
>
>
> ArabicAnalyzer lowercases text in case you have some non-Arabic text around.
> It also allows you to set a custom stopword list (you might augment the Arabic list with some English ones, for example).
> In this case its helpful for these non-Arabic stopwords, to lowercase before stopfilter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Reopened: (LUCENE-1963) ArabicAnalyzer: Lowercase before
Stopfilter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir reopened LUCENE-1963:
---------------------------------
Mark, I think the problem is really that I overlooked this use case in LUCENE-1758, because Arabic is not case sensitive.
It won't affect the default usage of the Analyzer (where all the stopwords are in Arabic and lowercase is a no-op).
I am going to also set fix for 2.9.1 and give a day or two for people to comment if they disagree with applying to 2.9 branch.
> ArabicAnalyzer: Lowercase before Stopfilter
> -------------------------------------------
>
> Key: LUCENE-1963
> URL: https://issues.apache.org/jira/browse/LUCENE-1963
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/analyzers
> Affects Versions: 2.9
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Fix For: 2.9.1, 3.0
>
> Attachments: LUCENE-1963.patch, LUCENE-1963.patch
>
>
> ArabicAnalyzer lowercases text in case you have some non-Arabic text around.
> It also allows you to set a custom stopword list (you might augment the Arabic list with some English ones, for example).
> In this case its helpful for these non-Arabic stopwords, to lowercase before stopfilter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org