You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2010/01/11 18:12:54 UTC
[jira] Created: (LUCENE-2203) improved snowball testing
improved snowball testing
-------------------------
Key: LUCENE-2203
URL: https://issues.apache.org/jira/browse/LUCENE-2203
Project: Lucene - Java
Issue Type: Test
Components: contrib/analyzers
Reporter: Robert Muir
Priority: Minor
Snowball project has test vocabulary files for each language in their svn repository, along with expected output.
We should use these tests to ensure all languages are working correctly, and it might be helpful in the future for identifying back breaks/changes if we ever want to upgrade snowball, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2203) improved snowball testing
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798737#action_12798737 ]
Robert Muir commented on LUCENE-2203:
-------------------------------------
its worth mentioning for the two broken languages: Finnish and Lovins, that they use some snowball operations none of the others do.
So I think its not gonna be too bad to get to the bottom of this.
> improved snowball testing
> -------------------------
>
> Key: LUCENE-2203
> URL: https://issues.apache.org/jira/browse/LUCENE-2203
> Project: Lucene - Java
> Issue Type: Test
> Components: contrib/analyzers
> Reporter: Robert Muir
> Priority: Minor
> Attachments: LUCENE-2203.patch
>
>
> Snowball project has test vocabulary files for each language in their svn repository, along with expected output.
> We should use these tests to ensure all languages are working correctly, and it might be helpful in the future for identifying back breaks/changes if we ever want to upgrade snowball, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2203) improved snowball testing
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799221#action_12799221 ]
Robert Muir commented on LUCENE-2203:
-------------------------------------
Hello, I ran some tests today, and the problems with Finnish and Lovins are bugs in snowball itself.
These two languages give correct results with their generated C code, but incorrect results with generated Java code.
I reported this to the snowball list, it is not a lucene problem. So I feel fine with leaving these commented out for now and also proceeding with performance improvements.
> improved snowball testing
> -------------------------
>
> Key: LUCENE-2203
> URL: https://issues.apache.org/jira/browse/LUCENE-2203
> Project: Lucene - Java
> Issue Type: Test
> Components: contrib/analyzers
> Reporter: Robert Muir
> Priority: Minor
> Attachments: LUCENE-2203.patch
>
>
> Snowball project has test vocabulary files for each language in their svn repository, along with expected output.
> We should use these tests to ensure all languages are working correctly, and it might be helpful in the future for identifying back breaks/changes if we ever want to upgrade snowball, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2203) improved snowball testing
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799703#action_12799703 ]
Robert Muir commented on LUCENE-2203:
-------------------------------------
i would like to commit this one at the end of the day, if no one objects.
Simon, you ok with this one? :)
> improved snowball testing
> -------------------------
>
> Key: LUCENE-2203
> URL: https://issues.apache.org/jira/browse/LUCENE-2203
> Project: Lucene - Java
> Issue Type: Test
> Components: contrib/analyzers
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Minor
> Attachments: LUCENE-2203.patch
>
>
> Snowball project has test vocabulary files for each language in their svn repository, along with expected output.
> We should use these tests to ensure all languages are working correctly, and it might be helpful in the future for identifying back breaks/changes if we ever want to upgrade snowball, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-2203) improved snowball testing
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir reassigned LUCENE-2203:
-----------------------------------
Assignee: Robert Muir
> improved snowball testing
> -------------------------
>
> Key: LUCENE-2203
> URL: https://issues.apache.org/jira/browse/LUCENE-2203
> Project: Lucene - Java
> Issue Type: Test
> Components: contrib/analyzers
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Minor
> Attachments: LUCENE-2203.patch
>
>
> Snowball project has test vocabulary files for each language in their svn repository, along with expected output.
> We should use these tests to ensure all languages are working correctly, and it might be helpful in the future for identifying back breaks/changes if we ever want to upgrade snowball, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2203) improved snowball testing
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved LUCENE-2203.
---------------------------------
Resolution: Fixed
Fix Version/s: 3.1
Committed revision 898950.
> improved snowball testing
> -------------------------
>
> Key: LUCENE-2203
> URL: https://issues.apache.org/jira/browse/LUCENE-2203
> Project: Lucene - Java
> Issue Type: Test
> Components: contrib/analyzers
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2203.patch
>
>
> Snowball project has test vocabulary files for each language in their svn repository, along with expected output.
> We should use these tests to ensure all languages are working correctly, and it might be helpful in the future for identifying back breaks/changes if we ever want to upgrade snowball, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2203) improved snowball testing
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2203:
--------------------------------
Attachment: LUCENE-2203.patch
attached is a patch that does an svn checkout of rev 500 (which is what we are using) of these test files, and has a test class to check every language.
the test for all languages is a total of 2 seconds on my computer (fast)
it takes about 10 seconds to do the svn checkout: since this stuff is BSD-licensed we could comment out the hook to do the checkout and instead actually commit the files if we want, i think as long as we do the proper NOTICE stuff, etc.
Please note two languages are commented out: Finnish and Lovins, these currently fail. I only investigated this enough to determine that it wasn't my LUCENE-2194 performance improvement commit that broke these, they were broken with the previous revision too. we should probably get to the bottom of these.
> improved snowball testing
> -------------------------
>
> Key: LUCENE-2203
> URL: https://issues.apache.org/jira/browse/LUCENE-2203
> Project: Lucene - Java
> Issue Type: Test
> Components: contrib/analyzers
> Reporter: Robert Muir
> Priority: Minor
> Attachments: LUCENE-2203.patch
>
>
> Snowball project has test vocabulary files for each language in their svn repository, along with expected output.
> We should use these tests to ensure all languages are working correctly, and it might be helpful in the future for identifying back breaks/changes if we ever want to upgrade snowball, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2203) improved snowball testing
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798800#action_12798800 ]
Robert Muir commented on LUCENE-2203:
-------------------------------------
Simon, these files are large (70MB) but so is the 65MB reuters corpus that the benchmark test downloads!!!!
> improved snowball testing
> -------------------------
>
> Key: LUCENE-2203
> URL: https://issues.apache.org/jira/browse/LUCENE-2203
> Project: Lucene - Java
> Issue Type: Test
> Components: contrib/analyzers
> Reporter: Robert Muir
> Priority: Minor
> Attachments: LUCENE-2203.patch
>
>
> Snowball project has test vocabulary files for each language in their svn repository, along with expected output.
> We should use these tests to ensure all languages are working correctly, and it might be helpful in the future for identifying back breaks/changes if we ever want to upgrade snowball, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2203) improved snowball testing
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798795#action_12798795 ]
Simon Willnauer commented on LUCENE-2203:
-----------------------------------------
Robert, those test seem to be very extensive - thats good!
But honestly think we should make those tests optional in some way. The files you are downloading are very large and might be an issues for some folks. The filesize is over 70MB which is a lot for a test. I need to thing about this a little and come up with some suggestions.
> improved snowball testing
> -------------------------
>
> Key: LUCENE-2203
> URL: https://issues.apache.org/jira/browse/LUCENE-2203
> Project: Lucene - Java
> Issue Type: Test
> Components: contrib/analyzers
> Reporter: Robert Muir
> Priority: Minor
> Attachments: LUCENE-2203.patch
>
>
> Snowball project has test vocabulary files for each language in their svn repository, along with expected output.
> We should use these tests to ensure all languages are working correctly, and it might be helpful in the future for identifying back breaks/changes if we ever want to upgrade snowball, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2203) improved snowball testing
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799868#action_12799868 ]
Simon Willnauer commented on LUCENE-2203:
-----------------------------------------
looks good to me. I haven't applied it but looks good though! +1 from my side
> improved snowball testing
> -------------------------
>
> Key: LUCENE-2203
> URL: https://issues.apache.org/jira/browse/LUCENE-2203
> Project: Lucene - Java
> Issue Type: Test
> Components: contrib/analyzers
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Minor
> Attachments: LUCENE-2203.patch
>
>
> Snowball project has test vocabulary files for each language in their svn repository, along with expected output.
> We should use these tests to ensure all languages are working correctly, and it might be helpful in the future for identifying back breaks/changes if we ever want to upgrade snowball, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2203) improved snowball testing
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799225#action_12799225 ]
Robert Muir commented on LUCENE-2203:
-------------------------------------
here is a link to the bug with an example: http://article.gmane.org/gmane.comp.search.snowball/1137
> improved snowball testing
> -------------------------
>
> Key: LUCENE-2203
> URL: https://issues.apache.org/jira/browse/LUCENE-2203
> Project: Lucene - Java
> Issue Type: Test
> Components: contrib/analyzers
> Reporter: Robert Muir
> Priority: Minor
> Attachments: LUCENE-2203.patch
>
>
> Snowball project has test vocabulary files for each language in their svn repository, along with expected output.
> We should use these tests to ensure all languages are working correctly, and it might be helpful in the future for identifying back breaks/changes if we ever want to upgrade snowball, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2203) improved snowball testing
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798957#action_12798957 ]
Uwe Schindler commented on LUCENE-2203:
---------------------------------------
The revision no. in the "svn co" works exactly like I proposed in LUCENE-2193 for the BW tests in lucene core.
> improved snowball testing
> -------------------------
>
> Key: LUCENE-2203
> URL: https://issues.apache.org/jira/browse/LUCENE-2203
> Project: Lucene - Java
> Issue Type: Test
> Components: contrib/analyzers
> Reporter: Robert Muir
> Priority: Minor
> Attachments: LUCENE-2203.patch
>
>
> Snowball project has test vocabulary files for each language in their svn repository, along with expected output.
> We should use these tests to ensure all languages are working correctly, and it might be helpful in the future for identifying back breaks/changes if we ever want to upgrade snowball, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org