You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2009/11/14 11:27:39 UTC
[jira] Created: (LUCENE-2067) Czech Stemmer
Czech Stemmer
-------------
Key: LUCENE-2067
URL: https://issues.apache.org/jira/browse/LUCENE-2067
Project: Lucene - Java
Issue Type: New Feature
Components: contrib/analyzers
Reporter: Robert Muir
Priority: Minor
Fix For: 3.1
Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
In their measurements, it improves MAP 42%
The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2067) Czech Stemmer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783390#action_12783390 ]
Robert Muir commented on LUCENE-2067:
-------------------------------------
bq. well at least I figured out there must be something wrong
i appreciate the review... it is frustrating that you have to pay $ to view the paper right now.
on the other hand we are lucky when researchers that are this open about their experiments... saves a lot of work.
> Czech Stemmer
> -------------
>
> Key: LUCENE-2067
> URL: https://issues.apache.org/jira/browse/LUCENE-2067
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2067.patch, LUCENE-2067.patch, LUCENE-2067.patch, LUCENE-2067.patch
>
>
> Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
> This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
> In their measurements, it improves MAP 42%
> The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2067) Czech Stemmer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2067:
--------------------------------
Attachment: LUCENE-2067.patch
updated patch, fixed to use Version.LUCENE_31.
I also added a note that this StemFilter expects input to be in lowercase already, but wants the diacritical marks to still be there.
If you want to ascii fold do it after stemming.
> Czech Stemmer
> -------------
>
> Key: LUCENE-2067
> URL: https://issues.apache.org/jira/browse/LUCENE-2067
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2067.patch, LUCENE-2067.patch
>
>
> Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
> This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
> In their measurements, it improves MAP 42%
> The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2067) Czech Stemmer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2067:
--------------------------------
Attachment: LUCENE-2067.patch
> Czech Stemmer
> -------------
>
> Key: LUCENE-2067
> URL: https://issues.apache.org/jira/browse/LUCENE-2067
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2067.patch
>
>
> Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
> This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
> In their measurements, it improves MAP 42%
> The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2067) Czech Stemmer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2067:
--------------------------------
Attachment: LUCENE-2067.patch
bring patch up to trunk, and round out the tests and docs.
This one is simple, if no one objects I will commit it in a day or two.
> Czech Stemmer
> -------------
>
> Key: LUCENE-2067
> URL: https://issues.apache.org/jira/browse/LUCENE-2067
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2067.patch, LUCENE-2067.patch, LUCENE-2067.patch
>
>
> Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
> This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
> In their measurements, it improves MAP 42%
> The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2067) Czech Stemmer
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777894#action_12777894 ]
Simon Willnauer commented on LUCENE-2067:
-----------------------------------------
Good stuff Robert!
You might wanna link to https://issues.apache.org/jira/browse/LUCENE-2034 and subclass the StopawareAnalyzer.
btw. we could also rename it to SmartCzechAnalyzer :)
> Czech Stemmer
> -------------
>
> Key: LUCENE-2067
> URL: https://issues.apache.org/jira/browse/LUCENE-2067
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2067.patch
>
>
> Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
> This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
> In their measurements, it improves MAP 42%
> The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2067) Czech Stemmer
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783271#action_12783271 ]
Simon Willnauer commented on LUCENE-2067:
-----------------------------------------
Robert, I really like the readability in combination with the pseudo code in the paper! The tests looking very good. The only thing I do not see in the paper is
{code}
if (len > 1 && s[len - 2] == 'e') {
s[len - 2] = s[len - 1]; // e* > *
return len - 1;
}
{code}
durin the normalize stage. Maybe I miss something in the paper or this is some kind of optimisation. If so would you add a comment? - would make it easier!
I also see no reason why the CzechStemFilter should be none final.
> Czech Stemmer
> -------------
>
> Key: LUCENE-2067
> URL: https://issues.apache.org/jira/browse/LUCENE-2067
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2067.patch, LUCENE-2067.patch, LUCENE-2067.patch
>
>
> Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
> This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
> In their measurements, it improves MAP 42%
> The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2067) Czech Stemmer
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783389#action_12783389 ]
Simon Willnauer commented on LUCENE-2067:
-----------------------------------------
bq. make the stem filter final, and add explicit test for the mobile e rewrite
looks good to me! Go ahead and commit.
bq. Sorry for the confusion (pointing you at a slightly different algorithm)...
well at least I figured out there must be something wrong :)
> Czech Stemmer
> -------------
>
> Key: LUCENE-2067
> URL: https://issues.apache.org/jira/browse/LUCENE-2067
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2067.patch, LUCENE-2067.patch, LUCENE-2067.patch, LUCENE-2067.patch
>
>
> Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
> This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
> In their measurements, it improves MAP 42%
> The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2067) Czech Stemmer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2067:
--------------------------------
Attachment: LUCENE-2067.patch
make the stem filter final, and add explicit test for the mobile e rewrite
> Czech Stemmer
> -------------
>
> Key: LUCENE-2067
> URL: https://issues.apache.org/jira/browse/LUCENE-2067
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2067.patch, LUCENE-2067.patch, LUCENE-2067.patch, LUCENE-2067.patch
>
>
> Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
> This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
> In their measurements, it improves MAP 42%
> The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2067) Czech Stemmer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783276#action_12783276 ]
Robert Muir commented on LUCENE-2067:
-------------------------------------
Hi Simon, I think you are referring to the 2007 paper: http://clef-campaign.org/2007/working_notes/DolamicCLEF2007.pdf, which is very similar.
The 2009 paper (referenced by the code, but you need ACM account to get it), http://portal.acm.org/citation.cfm?id=1598600
describes this in pseudocode as:
{noformat}
if (word ends with "e*") then replace by "*" return;
{noformat}
Sorry for the confusion (pointing you at a slightly different algorithm)...
btw here is a description of what this rule does from http://www.seelrc.org/projects/casebooks/appendixcz.pdf
Many words in Czech have what we call mobile e. This is an e that appears (usually
before the last consonant in the stem) when there is no ending after the stem, but is absent
when there is a case ending.
Here are some examples:
* pes 'dog' Nsg : psi 'dogs' Npl
* otec 'father' Nsg : otce 'father' Gsg
* deska 'board' Nsg : desek 'boards' Gpl.
bq. I also see no reason why the CzechStemFilter should be none final.
I will make the stemfilter final, thanks!
> Czech Stemmer
> -------------
>
> Key: LUCENE-2067
> URL: https://issues.apache.org/jira/browse/LUCENE-2067
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2067.patch, LUCENE-2067.patch, LUCENE-2067.patch
>
>
> Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
> This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
> In their measurements, it improves MAP 42%
> The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2067) Czech Stemmer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777961#action_12777961 ]
Robert Muir commented on LUCENE-2067:
-------------------------------------
Simon, in this case the analyzer exists, this just adds a TokenFilter to it :)
but I agree, we should use StopawareAnalyzer for both CzechAnalyzer here, and LUCENE-2062 also.
> Czech Stemmer
> -------------
>
> Key: LUCENE-2067
> URL: https://issues.apache.org/jira/browse/LUCENE-2067
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2067.patch
>
>
> Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
> This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
> In their measurements, it improves MAP 42%
> The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-2067) Czech Stemmer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir reassigned LUCENE-2067:
-----------------------------------
Assignee: Robert Muir
> Czech Stemmer
> -------------
>
> Key: LUCENE-2067
> URL: https://issues.apache.org/jira/browse/LUCENE-2067
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2067.patch, LUCENE-2067.patch, LUCENE-2067.patch
>
>
> Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
> This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
> In their measurements, it improves MAP 42%
> The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2067) Czech Stemmer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved LUCENE-2067.
---------------------------------
Resolution: Fixed
Committed revision 885216.
> Czech Stemmer
> -------------
>
> Key: LUCENE-2067
> URL: https://issues.apache.org/jira/browse/LUCENE-2067
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2067.patch, LUCENE-2067.patch, LUCENE-2067.patch, LUCENE-2067.patch
>
>
> Currently, the CzechAnalyzer is merely stopwords, and there isn't a czech stemmer in snowball.
> This patch implements the light stemming algorithm described in: http://portal.acm.org/citation.cfm?id=1598600
> In their measurements, it improves MAP 42%
> The analyzer does not use this stemmer if LUCENE_VERSION <= 3.0, for back compat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org