You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Dan Rosher (JIRA)" <ji...@apache.org> on 2008/11/24 16:21:44 UTC
[jira] Created: (SOLR-876) Add ability to optionally
splitOnNumerics WordDelimiterFilter/Factory
Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory
---------------------------------------------------------------------
Key: SOLR-876
URL: https://issues.apache.org/jira/browse/SOLR-876
Project: Solr
Issue Type: Improvement
Components: search
Reporter: Dan Rosher
Priority: Minor
Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory
Default behaviour is to splitOnNumerics as WordDelimiterFilter/Factory does now
I was having issues with e.g. Java/J2SE becoming split into tokens 'Java','J' '2' and 'SE' which isn't desired behavior in my instance, I wanted this to be tokens 'Java', 'J2SE'. Another option I thought about but not implemented was to have a protected list of words like solr.EnglishPorterFilterFactory
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-876) Add ability to optionally
splitOnNumerics WordDelimiterFilter/Factory
Posted by "Dan Rosher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dan Rosher updated SOLR-876:
----------------------------
Attachment: SOLR-876.patch
Thanks for reading over it Hoss, I think I originally made the patch against a previous version of the solr trunk, the test omissions weren't intentional, I should have done a better job of reading the patch!
I've made new constructors, and deprecated the current/older ones, not sure if this is the approach you recommend though.
> Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory
> ---------------------------------------------------------------------
>
> Key: SOLR-876
> URL: https://issues.apache.org/jira/browse/SOLR-876
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Dan Rosher
> Priority: Minor
> Attachments: SOLR-876.patch, SOLR-876.patch, SOLR-876.patch
>
>
> Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory
> Default behaviour is to splitOnNumerics as WordDelimiterFilter/Factory does now
> I was having issues with e.g. Java/J2SE becoming split into tokens 'Java','J' '2' and 'SE' which isn't desired behavior in my instance, I wanted this to be tokens 'Java', 'J2SE'. Another option I thought about but not implemented was to have a protected list of words like solr.EnglishPorterFilterFactory
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-876) Add ability to optionally
splitOnNumerics WordDelimiterFilter/Factory
Posted by "Dan Rosher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dan Rosher updated SOLR-876:
----------------------------
Attachment: SOLR-876.patch
Add ability to splitOnNumerics [default = 1 as is current behaviour.]
Also some additional tests in src/test/org/apache/solr/analysis/TestWordDelimiterFilter.java and changes to src/test/test-files/solr/conf/schema.xml
> Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory
> ---------------------------------------------------------------------
>
> Key: SOLR-876
> URL: https://issues.apache.org/jira/browse/SOLR-876
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Dan Rosher
> Priority: Minor
> Attachments: SOLR-876.patch
>
>
> Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory
> Default behaviour is to splitOnNumerics as WordDelimiterFilter/Factory does now
> I was having issues with e.g. Java/J2SE becoming split into tokens 'Java','J' '2' and 'SE' which isn't desired behavior in my instance, I wanted this to be tokens 'Java', 'J2SE'. Another option I thought about but not implemented was to have a protected list of words like solr.EnglishPorterFilterFactory
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SOLR-876) Add ability to optionally
splitOnNumerics WordDelimiterFilter/Factory
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man resolved SOLR-876.
---------------------------
Resolution: Fixed
Fix Version/s: 1.4
Assignee: Hoss Man
Thanks for the great patch Dan.
Committed revision 727677.
(i also added a few more tests to prove to myself that protwords was really working)
> Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory
> ---------------------------------------------------------------------
>
> Key: SOLR-876
> URL: https://issues.apache.org/jira/browse/SOLR-876
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Dan Rosher
> Assignee: Hoss Man
> Priority: Minor
> Fix For: 1.4
>
> Attachments: SOLR-876.patch, SOLR-876.patch, SOLR-876.patch
>
>
> Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory
> Default behaviour is to splitOnNumerics as WordDelimiterFilter/Factory does now
> I was having issues with e.g. Java/J2SE becoming split into tokens 'Java','J' '2' and 'SE' which isn't desired behavior in my instance, I wanted this to be tokens 'Java', 'J2SE'. Another option I thought about but not implemented was to have a protected list of words like solr.EnglishPorterFilterFactory
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-876) Add ability to optionally
splitOnNumerics WordDelimiterFilter/Factory
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651762#action_12651762 ]
Hoss Man commented on SOLR-876:
-------------------------------
Dan: reading over your patch it seems that several test* methods were removed from TestWordDelimiterFilter.java, as well as change some parts of the test to use the deprecated Token.termText() method (even though the test already uses Token.termBuffer())
Also: for back compatibility we should leave the existing WDF constructors alone, and either add a new one with the new args, or make the new options configured via setters.
> Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory
> ---------------------------------------------------------------------
>
> Key: SOLR-876
> URL: https://issues.apache.org/jira/browse/SOLR-876
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Dan Rosher
> Priority: Minor
> Attachments: SOLR-876.patch, SOLR-876.patch
>
>
> Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory
> Default behaviour is to splitOnNumerics as WordDelimiterFilter/Factory does now
> I was having issues with e.g. Java/J2SE becoming split into tokens 'Java','J' '2' and 'SE' which isn't desired behavior in my instance, I wanted this to be tokens 'Java', 'J2SE'. Another option I thought about but not implemented was to have a protected list of words like solr.EnglishPorterFilterFactory
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-876) Add ability to optionally
splitOnNumerics WordDelimiterFilter/Factory
Posted by "Dan Rosher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dan Rosher updated SOLR-876:
----------------------------
Attachment: SOLR-876.patch
Added in ability to 'protect' words against further tokenizing
> Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory
> ---------------------------------------------------------------------
>
> Key: SOLR-876
> URL: https://issues.apache.org/jira/browse/SOLR-876
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Dan Rosher
> Priority: Minor
> Attachments: SOLR-876.patch, SOLR-876.patch
>
>
> Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory
> Default behaviour is to splitOnNumerics as WordDelimiterFilter/Factory does now
> I was having issues with e.g. Java/J2SE becoming split into tokens 'Java','J' '2' and 'SE' which isn't desired behavior in my instance, I wanted this to be tokens 'Java', 'J2SE'. Another option I thought about but not implemented was to have a protected list of words like solr.EnglishPorterFilterFactory
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.