You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Luca Cavanna (JIRA)" <ji...@apache.org> on 2012/04/24 15:55:34 UTC
[jira] [Created] (LUCENE-4019) Parsing Hunspell affix rules without
regexp condition
Luca Cavanna created LUCENE-4019:
------------------------------------
Summary: Parsing Hunspell affix rules without regexp condition
Key: LUCENE-4019
URL: https://issues.apache.org/jira/browse/LUCENE-4019
Project: Lucene - Java
Issue Type: Improvement
Components: modules/analysis
Affects Versions: 3.6
Reporter: Luca Cavanna
We found out that some recent Dutch hunspell dictionaries contain suffix or prefix rules like the following:
{code}
SFX Na N 1
SFX Na 0 ste
{code}
The rule on the second line doesn't contain the 5th parameter, which should be the condition (a regexp usually). You can usually see a '.' as condition, meaning always (for every character). As explained in LUCENE-3976 the readAffix method throws error. I wonder if we should treat the missing value as a kind of default value, like '.'. On the other hand I haven't found any information about this within the spec. Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4019) Parsing Hunspell affix rules
without regexp condition
Posted by "Chris Male (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284393#comment-13284393 ]
Chris Male commented on LUCENE-4019:
------------------------------------
Hi Luca,
Sorry for taking so long to get to this. Patch looks good and seems to fix the problem. I think we do need some way to force 'strict' parsing of the files. Do you think you can add a option for that? When strict parsing is enabled, lines without the expected number of elements cause an error.
We can even have this enabled by default so users have to explicitly say that they know the dictionary doesn't conform to our standard and are okay with us silently ignoring bad rules.
> Parsing Hunspell affix rules without regexp condition
> -----------------------------------------------------
>
> Key: LUCENE-4019
> URL: https://issues.apache.org/jira/browse/LUCENE-4019
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.6
> Reporter: Luca Cavanna
> Assignee: Chris Male
> Attachments: LUCENE-4019.patch
>
>
> We found out that some recent Dutch hunspell dictionaries contain suffix or prefix rules like the following:
> {code}
> SFX Na N 1
> SFX Na 0 ste
> {code}
> The rule on the second line doesn't contain the 5th parameter, which should be the condition (a regexp usually). You can usually see a '.' as condition, meaning always (for every character). As explained in LUCENE-3976 the readAffix method throws error. I wonder if we should treat the missing value as a kind of default value, like '.'. On the other hand I haven't found any information about this within the spec. Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-4019) Parsing Hunspell affix rules
without regexp condition
Posted by "Chris Male (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Male resolved LUCENE-4019.
--------------------------------
Resolution: Fixed
Fix Version/s: 5.0
4.0
Thanks Luca!
> Parsing Hunspell affix rules without regexp condition
> -----------------------------------------------------
>
> Key: LUCENE-4019
> URL: https://issues.apache.org/jira/browse/LUCENE-4019
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.6
> Reporter: Luca Cavanna
> Assignee: Chris Male
> Fix For: 4.0, 5.0
>
> Attachments: LUCENE-4019.patch, LUCENE-4019.patch, LUCENE-4019.patch
>
>
> We found out that some recent Dutch hunspell dictionaries contain suffix or prefix rules like the following:
> {code}
> SFX Na N 1
> SFX Na 0 ste
> {code}
> The rule on the second line doesn't contain the 5th parameter, which should be the condition (a regexp usually). You can usually see a '.' as condition, meaning always (for every character). As explained in LUCENE-3976 the readAffix method throws error. I wonder if we should treat the missing value as a kind of default value, like '.'. On the other hand I haven't found any information about this within the spec. Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-4019) Parsing Hunspell affix rules
without regexp condition
Posted by "Chris Male (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Male reassigned LUCENE-4019:
----------------------------------
Assignee: Chris Male
> Parsing Hunspell affix rules without regexp condition
> -----------------------------------------------------
>
> Key: LUCENE-4019
> URL: https://issues.apache.org/jira/browse/LUCENE-4019
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.6
> Reporter: Luca Cavanna
> Assignee: Chris Male
> Attachments: LUCENE-4019.patch
>
>
> We found out that some recent Dutch hunspell dictionaries contain suffix or prefix rules like the following:
> {code}
> SFX Na N 1
> SFX Na 0 ste
> {code}
> The rule on the second line doesn't contain the 5th parameter, which should be the condition (a regexp usually). You can usually see a '.' as condition, meaning always (for every character). As explained in LUCENE-3976 the readAffix method throws error. I wonder if we should treat the missing value as a kind of default value, like '.'. On the other hand I haven't found any information about this within the spec. Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4019) Parsing Hunspell affix rules
without regexp condition
Posted by "Luca Cavanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260612#comment-13260612 ]
Luca Cavanna commented on LUCENE-4019:
--------------------------------------
Robert, with "spec" I meant exactly your links :)
Actually it's clear that the affix header has 4 elements while each rule has at least 5 elements. I don't really know what hunspell does with that kind of malformed rules. Lucene just throws an error while loading the dictionary. Looking at the hunspell source code, I might be wrong but I suspect it just skips that specific rule with some warning. But honestly it's hard to believe that at least 4 dictionaries I tried contain mistaken rules, isn't it? I'll investigate more, thanks!
> Parsing Hunspell affix rules without regexp condition
> -----------------------------------------------------
>
> Key: LUCENE-4019
> URL: https://issues.apache.org/jira/browse/LUCENE-4019
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.6
> Reporter: Luca Cavanna
>
> We found out that some recent Dutch hunspell dictionaries contain suffix or prefix rules like the following:
> {code}
> SFX Na N 1
> SFX Na 0 ste
> {code}
> The rule on the second line doesn't contain the 5th parameter, which should be the condition (a regexp usually). You can usually see a '.' as condition, meaning always (for every character). As explained in LUCENE-3976 the readAffix method throws error. I wonder if we should treat the missing value as a kind of default value, like '.'. On the other hand I haven't found any information about this within the spec. Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4019) Parsing Hunspell affix rules without
regexp condition
Posted by "Luca Cavanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luca Cavanna updated LUCENE-4019:
---------------------------------
Attachment: LUCENE-4019.patch
Yeah, sorry for my mistakes, I corrected them.
And I added the line number to the ParseException.
Let me know if there's something more I can do!
> Parsing Hunspell affix rules without regexp condition
> -----------------------------------------------------
>
> Key: LUCENE-4019
> URL: https://issues.apache.org/jira/browse/LUCENE-4019
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.6
> Reporter: Luca Cavanna
> Assignee: Chris Male
> Attachments: LUCENE-4019.patch, LUCENE-4019.patch, LUCENE-4019.patch
>
>
> We found out that some recent Dutch hunspell dictionaries contain suffix or prefix rules like the following:
> {code}
> SFX Na N 1
> SFX Na 0 ste
> {code}
> The rule on the second line doesn't contain the 5th parameter, which should be the condition (a regexp usually). You can usually see a '.' as condition, meaning always (for every character). As explained in LUCENE-3976 the readAffix method throws error. I wonder if we should treat the missing value as a kind of default value, like '.'. On the other hand I haven't found any information about this within the spec. Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4019) Parsing Hunspell affix rules without
regexp condition
Posted by "Luca Cavanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luca Cavanna updated LUCENE-4019:
---------------------------------
Attachment: LUCENE-4019.patch
Hi Chris,
thanks for your feedback. Here is a new patch containing a new option in order to enable/disable the affix strict parsing, by default it is enabled. I updated the HunspellStemFilterFactory too in order to expose the new option to Solr.
> Parsing Hunspell affix rules without regexp condition
> -----------------------------------------------------
>
> Key: LUCENE-4019
> URL: https://issues.apache.org/jira/browse/LUCENE-4019
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.6
> Reporter: Luca Cavanna
> Assignee: Chris Male
> Attachments: LUCENE-4019.patch, LUCENE-4019.patch
>
>
> We found out that some recent Dutch hunspell dictionaries contain suffix or prefix rules like the following:
> {code}
> SFX Na N 1
> SFX Na 0 ste
> {code}
> The rule on the second line doesn't contain the 5th parameter, which should be the condition (a regexp usually). You can usually see a '.' as condition, meaning always (for every character). As explained in LUCENE-3976 the readAffix method throws error. I wonder if we should treat the missing value as a kind of default value, like '.'. On the other hand I haven't found any information about this within the spec. Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4019) Parsing Hunspell affix rules
without regexp condition
Posted by "Chris Male (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286474#comment-13286474 ]
Chris Male commented on LUCENE-4019:
------------------------------------
Hi Luca,
Thanks for taking a shot at this.
I wonder whether we can do improve the ParseException message? At the very least it should include the line that is causing the problem so people can find it. What would be even better is if we also included the line number. The latter is probably not so urgent, but it would be handy to have for other parsing errors too.
Also I think the changes to the Factory are wrong:
{code}
+ if(strictAffixParsing.equalsIgnoreCase(TRUE)) ignoreCase = true;
+ else if(strictAffixParsing.equalsIgnoreCase(FALSE)) ignoreCase = false;
{code}
> Parsing Hunspell affix rules without regexp condition
> -----------------------------------------------------
>
> Key: LUCENE-4019
> URL: https://issues.apache.org/jira/browse/LUCENE-4019
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.6
> Reporter: Luca Cavanna
> Assignee: Chris Male
> Attachments: LUCENE-4019.patch, LUCENE-4019.patch
>
>
> We found out that some recent Dutch hunspell dictionaries contain suffix or prefix rules like the following:
> {code}
> SFX Na N 1
> SFX Na 0 ste
> {code}
> The rule on the second line doesn't contain the 5th parameter, which should be the condition (a regexp usually). You can usually see a '.' as condition, meaning always (for every character). As explained in LUCENE-3976 the readAffix method throws error. I wonder if we should treat the missing value as a kind of default value, like '.'. On the other hand I haven't found any information about this within the spec. Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4019) Parsing Hunspell affix rules
without regexp condition
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260566#comment-13260566 ]
Robert Muir commented on LUCENE-4019:
-------------------------------------
I dont know if there is a real spec, more just what hunspell allows.
Furthermore i think some of these dictionaries are actually in ispell/myspell format
and hunspell is actually backwards compatible with them?
as far as a "spec" for all of these, good luck :)
when i was looking at this I looked at stuff like:
* http://pwet.fr/man/linux/fichiers_speciaux/hunspell
* http://www.openoffice.org/lingucomponent/affix.readme
> Parsing Hunspell affix rules without regexp condition
> -----------------------------------------------------
>
> Key: LUCENE-4019
> URL: https://issues.apache.org/jira/browse/LUCENE-4019
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.6
> Reporter: Luca Cavanna
>
> We found out that some recent Dutch hunspell dictionaries contain suffix or prefix rules like the following:
> {code}
> SFX Na N 1
> SFX Na 0 ste
> {code}
> The rule on the second line doesn't contain the 5th parameter, which should be the condition (a regexp usually). You can usually see a '.' as condition, meaning always (for every character). As explained in LUCENE-3976 the readAffix method throws error. I wonder if we should treat the missing value as a kind of default value, like '.'. On the other hand I haven't found any information about this within the spec. Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4019) Parsing Hunspell affix rules
without regexp condition
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260638#comment-13260638 ]
Robert Muir commented on LUCENE-4019:
-------------------------------------
its tough to know for sure. in general a lot of hunspell dictionaries cannot be parsed.
There are a ton of these, under many strange licenses and they are very large.
A "Test scaffolding" of sorts could probably be done to hunt out problems:
* download all dictionaries you can find
* for each one, use hunspell command-line tools like munch, unmunch (which applies all the rules), etc
to generate some sort of expected output in .txt format.
* for each one, do the same using the hunspell parsing here.
* compare results: when things differ, try to boil it down to a compact .aff/.dic, with a test case and fix and commit.
> Parsing Hunspell affix rules without regexp condition
> -----------------------------------------------------
>
> Key: LUCENE-4019
> URL: https://issues.apache.org/jira/browse/LUCENE-4019
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.6
> Reporter: Luca Cavanna
>
> We found out that some recent Dutch hunspell dictionaries contain suffix or prefix rules like the following:
> {code}
> SFX Na N 1
> SFX Na 0 ste
> {code}
> The rule on the second line doesn't contain the 5th parameter, which should be the condition (a regexp usually). You can usually see a '.' as condition, meaning always (for every character). As explained in LUCENE-3976 the readAffix method throws error. I wonder if we should treat the missing value as a kind of default value, like '.'. On the other hand I haven't found any information about this within the spec. Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4019) Parsing Hunspell affix rules without
regexp condition
Posted by "Luca Cavanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luca Cavanna updated LUCENE-4019:
---------------------------------
Attachment: LUCENE-4019.patch
Small patch: affix rules with less than 5 elements are now ignored. I added a specific test with a new affix file containing an example of rule shorter than it should be. Let me know if you prefer to add a warning when a rule is skipped. Hunspell does that only with a specific command line option.
> Parsing Hunspell affix rules without regexp condition
> -----------------------------------------------------
>
> Key: LUCENE-4019
> URL: https://issues.apache.org/jira/browse/LUCENE-4019
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.6
> Reporter: Luca Cavanna
> Attachments: LUCENE-4019.patch
>
>
> We found out that some recent Dutch hunspell dictionaries contain suffix or prefix rules like the following:
> {code}
> SFX Na N 1
> SFX Na 0 ste
> {code}
> The rule on the second line doesn't contain the 5th parameter, which should be the condition (a regexp usually). You can usually see a '.' as condition, meaning always (for every character). As explained in LUCENE-3976 the readAffix method throws error. I wonder if we should treat the missing value as a kind of default value, like '.'. On the other hand I haven't found any information about this within the spec. Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4019) Parsing Hunspell affix rules
without regexp condition
Posted by "Luca Cavanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269516#comment-13269516 ]
Luca Cavanna commented on LUCENE-4019:
--------------------------------------
Thank you Robert for the explanation!
In this specific case it's hard to understand the differences between hunspell and Lucene, since Lucene doesn't even parse the affix file.
I've been in contact with the authors of those Ducth dictionaries, as well as with the hunspell author. It turned out that those affix rules are wrong and hunspell actually ignores them. I think it's better to ignore them in Lucene too, rather than throwing an exception, which makes impossible to use those dictionaries at all.
> Parsing Hunspell affix rules without regexp condition
> -----------------------------------------------------
>
> Key: LUCENE-4019
> URL: https://issues.apache.org/jira/browse/LUCENE-4019
> Project: Lucene - Java
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 3.6
> Reporter: Luca Cavanna
>
> We found out that some recent Dutch hunspell dictionaries contain suffix or prefix rules like the following:
> {code}
> SFX Na N 1
> SFX Na 0 ste
> {code}
> The rule on the second line doesn't contain the 5th parameter, which should be the condition (a regexp usually). You can usually see a '.' as condition, meaning always (for every character). As explained in LUCENE-3976 the readAffix method throws error. I wonder if we should treat the missing value as a kind of default value, like '.'. On the other hand I haven't found any information about this within the spec. Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org