You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2010/03/06 01:47:27 UTC
[jira] Created: (LUCENE-2298) Polish Analyzer
Polish Analyzer
---------------
Key: LUCENE-2298
URL: https://issues.apache.org/jira/browse/LUCENE-2298
Project: Lucene - Java
Issue Type: New Feature
Components: contrib/analyzers
Affects Versions: 3.1
Reporter: Robert Muir
Assignee: Robert Muir
Fix For: 3.1
Andrzej Bialecki has written a Polish stemmer and provided stemming tables for it under Apache License.
You can read more about it here: http://www.getopt.org/stempel/
In reality, the stemmer is general code and we could use it for more languages too perhaps.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2298) Polish Analyzer
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848270#action_12848270 ]
Dawid Weiss commented on LUCENE-2298:
-------------------------------------
The answer from the developer is: pick any license for from this list. So... GPL/LGPL are not compatible with Apache2. MPL seems not to be compatible for now (there's been Mozilla initiative to make it compatible, but I think this process is still active). What about ShareAlike? Does this address the license above?
http://www.apache.org/legal/resolved.html#cc-sa
"For any other type of CC-SA licensed work, please contact the Legal PMC. " I'll try to see which one of them the author had in mind...
> Polish Analyzer
> ---------------
>
> Key: LUCENE-2298
> URL: https://issues.apache.org/jira/browse/LUCENE-2298
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Affects Versions: 3.1
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-2298.patch, stemmer_20000.7z
>
>
> Andrzej Bialecki has written a Polish stemmer and provided stemming tables for it under Apache License.
> You can read more about it here: http://www.getopt.org/stempel/
> In reality, the stemmer is general code and we could use it for more languages too perhaps.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2298) Polish Analyzer
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848054#action_12848054 ]
Dawid Weiss commented on LUCENE-2298:
-------------------------------------
Staszek suggested that perhaps it would be convenient if this patch detected if another Polish stemming library for Polish is present in classpath and if so, used it. The library in mind is "morfologik-stemming", here:
http://sourceforge.net/projects/morfologik/
The code of this library is BSD-licensed and consists mainly of traversal of FSA automata. The stemmer is dictionary based, so it is nearly (ambiguities) 100% accurate for words in the dictionary and 0% accurate for non-dictionary words (returns null).
The problem with Morfologik is that its dictionary data is LGPL-ed, so it would have to be a separate download.
This is just a suggestion for discussion. I guess this functionality is limited to a very narrow audience anyway.
> Polish Analyzer
> ---------------
>
> Key: LUCENE-2298
> URL: https://issues.apache.org/jira/browse/LUCENE-2298
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Affects Versions: 3.1
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-2298.patch, stemmer_20000.7z
>
>
> Andrzej Bialecki has written a Polish stemmer and provided stemming tables for it under Apache License.
> You can read more about it here: http://www.getopt.org/stempel/
> In reality, the stemmer is general code and we could use it for more languages too perhaps.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2298) Polish Analyzer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848273#action_12848273 ]
Robert Muir commented on LUCENE-2298:
-------------------------------------
Dawid, thanks... so it is "pick your poison" after all.
FYI I created a JIRA issue so we can work this: LUCENE-2341
> Polish Analyzer
> ---------------
>
> Key: LUCENE-2298
> URL: https://issues.apache.org/jira/browse/LUCENE-2298
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Affects Versions: 3.1
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-2298.patch, stemmer_20000.7z
>
>
> Andrzej Bialecki has written a Polish stemmer and provided stemming tables for it under Apache License.
> You can read more about it here: http://www.getopt.org/stempel/
> In reality, the stemmer is general code and we could use it for more languages too perhaps.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2298) Polish Analyzer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2298:
--------------------------------
Attachment: LUCENE-2298.patch
Attached is an initial patch. i made the following changes to the original code:
* add generics and other java 5 things
* add junit tests
* add polish analyzer, with stopwords list from carrot2
* repackaging
you will need to extract the stemming table into resources/org.apache.lucene.analysis.pl folder, right where the Polish stopwords are. I will attach this separately.
> Polish Analyzer
> ---------------
>
> Key: LUCENE-2298
> URL: https://issues.apache.org/jira/browse/LUCENE-2298
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Affects Versions: 3.1
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-2298.patch
>
>
> Andrzej Bialecki has written a Polish stemmer and provided stemming tables for it under Apache License.
> You can read more about it here: http://www.getopt.org/stempel/
> In reality, the stemmer is general code and we could use it for more languages too perhaps.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2298) Polish Analyzer
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848139#action_12848139 ]
Dawid Weiss commented on LUCENE-2298:
-------------------------------------
I agree about classpath issues, they're a pain to debug. As for the license -- Morfologik's data is compiled from several sources, but you're right -- this clause has been problematic:
Licensed under GPL, LGPL, MPL and Creative Commons ShareAlike licenses.
Is there any legal sense in this? Is this an alternative (pick the one you want) or a combination (i.e. GPL swallows everything else)? I'll try to get in touch with the author and see what he has to say about it.
> Polish Analyzer
> ---------------
>
> Key: LUCENE-2298
> URL: https://issues.apache.org/jira/browse/LUCENE-2298
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Affects Versions: 3.1
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-2298.patch, stemmer_20000.7z
>
>
> Andrzej Bialecki has written a Polish stemmer and provided stemming tables for it under Apache License.
> You can read more about it here: http://www.getopt.org/stempel/
> In reality, the stemmer is general code and we could use it for more languages too perhaps.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2298) Polish Analyzer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848120#action_12848120 ]
Robert Muir commented on LUCENE-2298:
-------------------------------------
bq. The problem with Morfologik is that its dictionary data is LGPL-ed, so it would have to be a separate download.
Dawid, I looked at Morfologik's source code and it refers to this link: http://www.sjp.pl/slownik/en/
Is this correct? Because this appears to be licensed under GPL, LGPL, MPL and Creative Commons ShareAlike licenses.
bq. This is just a suggestion for discussion. I guess this functionality is limited to a very narrow audience anyway.
But if we can, why not provide two options?
In this case I think we should just open a separate JIRA issue for Morfologik.
Its really a different method than what this stemmer does anyway.
> Polish Analyzer
> ---------------
>
> Key: LUCENE-2298
> URL: https://issues.apache.org/jira/browse/LUCENE-2298
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Affects Versions: 3.1
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-2298.patch, stemmer_20000.7z
>
>
> Andrzej Bialecki has written a Polish stemmer and provided stemming tables for it under Apache License.
> You can read more about it here: http://www.getopt.org/stempel/
> In reality, the stemmer is general code and we could use it for more languages too perhaps.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2298) Polish Analyzer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848103#action_12848103 ]
Robert Muir commented on LUCENE-2298:
-------------------------------------
bq. Staszek suggested that perhaps it would be convenient if this patch detected if another Polish stemming library for Polish is present in classpath and if so, used it.
Hello, perhaps we can provide this as a separate option?
The problem with doing this is what to do if the classpath detection fails? would it fall back on the code here?
This magic could cause a lot of confusion, for example: indexing on a server with correct classpath, searching
on another where it is incorrect, or vice-versa.
In my opinion, for analysis stuff, if the environment isn't setup correctly, we should just throw an exception.
> Polish Analyzer
> ---------------
>
> Key: LUCENE-2298
> URL: https://issues.apache.org/jira/browse/LUCENE-2298
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Affects Versions: 3.1
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-2298.patch, stemmer_20000.7z
>
>
> Andrzej Bialecki has written a Polish stemmer and provided stemming tables for it under Apache License.
> You can read more about it here: http://www.getopt.org/stempel/
> In reality, the stemmer is general code and we could use it for more languages too perhaps.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2298) Polish Analyzer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848265#action_12848265 ]
Robert Muir commented on LUCENE-2298:
-------------------------------------
bq. Is there any legal sense in this? Is this an alternative (pick the one you want) or a combination (i.e. GPL swallows everything else)?
I looked at the files themselves etc and could not figure out the meaning of this.
I don't know if its fair to interpret it as "multi-licensed" or what, and I wouldn't want to do that.
But I do know several hunspell dictionaries are dual-licensed like this.
I suppose its so that they can be used in more open source projects.
bq. I'll try to get in touch with the author and see what he has to say about it.
Cool, thanks in advance for looking into this!
> Polish Analyzer
> ---------------
>
> Key: LUCENE-2298
> URL: https://issues.apache.org/jira/browse/LUCENE-2298
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Affects Versions: 3.1
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-2298.patch, stemmer_20000.7z
>
>
> Andrzej Bialecki has written a Polish stemmer and provided stemming tables for it under Apache License.
> You can read more about it here: http://www.getopt.org/stempel/
> In reality, the stemmer is general code and we could use it for more languages too perhaps.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2298) Polish Analyzer
Posted by "Dawid Weiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848648#action_12848648 ]
Dawid Weiss commented on LUCENE-2298:
-------------------------------------
The dictionary's author states that:
"Attribution-sa to ta, na której jest udostępniany słownik."
so we can pick the CC-SA license that Apache supposedly permits in the repositories. This is good news. Switching to LUCENE-2341 then.
> Polish Analyzer
> ---------------
>
> Key: LUCENE-2298
> URL: https://issues.apache.org/jira/browse/LUCENE-2298
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Affects Versions: 3.1
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-2298.patch, stemmer_20000.7z
>
>
> Andrzej Bialecki has written a Polish stemmer and provided stemming tables for it under Apache License.
> You can read more about it here: http://www.getopt.org/stempel/
> In reality, the stemmer is general code and we could use it for more languages too perhaps.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2298) Polish Analyzer
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2298:
--------------------------------
Attachment: stemmer_20000.7z
tables that must go in the resources folder under Polish (o.a.l.analysis.pl)
> Polish Analyzer
> ---------------
>
> Key: LUCENE-2298
> URL: https://issues.apache.org/jira/browse/LUCENE-2298
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Affects Versions: 3.1
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 3.1
>
> Attachments: LUCENE-2298.patch, stemmer_20000.7z
>
>
> Andrzej Bialecki has written a Polish stemmer and provided stemming tables for it under Apache License.
> You can read more about it here: http://www.getopt.org/stempel/
> In reality, the stemmer is general code and we could use it for more languages too perhaps.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org