You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Tommaso Teofili (Created) (JIRA)" <ji...@apache.org> on 2012/01/21 23:41:39 UTC
[jira] [Created] (SOLR-3054) Add a TypeTokenFilterFactory
Add a TypeTokenFilterFactory
----------------------------
Key: SOLR-3054
URL: https://issues.apache.org/jira/browse/SOLR-3054
Project: Solr
Issue Type: New Feature
Components: Schema and Analysis
Reporter: Tommaso Teofili
Fix For: 3.6, 4.0
Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Uwe Schindler (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler resolved SOLR-3054.
---------------------------------
Resolution: Fixed
Committed trunk revision: 1234573 (main), 1234579 (removal of unneeded assureMatchVersion check)
Committed 3.x revision: 1234580
As noted, I removed the not-needed assureMatchVersionCheck(), as this filter does not use the matchVersion at all.
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch, SOLR-3054_2.patch, SOLR-3054_3.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Tommaso Teofili (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tommaso Teofili updated SOLR-3054:
----------------------------------
Attachment: SOLR-3054_3.patch
updated patch:
- enablePosIncr set to false
- fixed unit tests for throwing Exceptions rather than try/catching
- 'types' TypeTokenFF parameter is now mandatory (SolrException raised if not supplied)
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch, SOLR-3054_2.patch, SOLR-3054_3.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Tommaso Teofili (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190643#comment-13190643 ]
Tommaso Teofili commented on SOLR-3054:
---------------------------------------
bq. I looked up the other TokenFilters that filter tokens, unfortunately all of then default to enablePosIncr=false. I am not sure what the right solution is here? Consistency or correctness?
in the first patch I went for consistency but then your comment made me realize the enablePosIncr should be true by default. I mean, as a user I'd expect it to be true by default.
bq. I would only remove the try-catch blocks in the test methods and let the test method declare the exception. It then gets reported by JUnit with a failure automatically.
ok
bq. The question is, the wordset is initialized to be empty if missing. Does it make sense? I would maybe make the types file mandatory, as without the filter makes no sense.
right, need to fix that
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch, SOLR-3054_2.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Tommaso Teofili (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tommaso Teofili updated SOLR-3054:
----------------------------------
Attachment: SOLR-3054.patch
attached factory with unit test patch
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190641#comment-13190641 ]
Uwe Schindler commented on SOLR-3054:
-------------------------------------
I looked up the other TokenFilters that filter tokens, unfortunately all of then default to enablePosIncr=false. I am not sure what the right solution is here? Consistency or correctness? Robert, whats your opinion?
The rest of the patch looks fine, I would only remove the try-catch blocks in the test methods and let the test method declare the exception. It then gets reported by JUnit with a failure automatically.
The question is, the wordset is initialized to be empty if missing. Does it make sense? I would maybe make the types file mandatory, as without the filter makes no sense.
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch, SOLR-3054_2.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Uwe Schindler (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler reassigned SOLR-3054:
-----------------------------------
Assignee: Uwe Schindler
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190631#comment-13190631 ]
Uwe Schindler commented on SOLR-3054:
-------------------------------------
Hi Tommaso,
patch look good, a few things:
- I think the default should be to enable positionIncrements. I have no idea what the default for the other FilteringTFs is, but Solr has the correct posIncr handling in its default configs enabled.
- The MockResourceLoader is funny, but why do you need it? For the creation test, simply also use the supplied stoptype-files.
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190717#comment-13190717 ]
Robert Muir commented on SOLR-3054:
-----------------------------------
+1
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch, SOLR-3054_2.patch, SOLR-3054_3.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190694#comment-13190694 ]
Uwe Schindler commented on SOLR-3054:
-------------------------------------
OK, I think we should commit them with "false" as default for consistency (especially in 3.x), and upgrade 4.0 in a separate issue.
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch, SOLR-3054_2.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190691#comment-13190691 ]
Robert Muir commented on SOLR-3054:
-----------------------------------
{quote}
in the first patch I went for consistency but then your comment made me realize the enablePosIncr should be true by default. I mean, as a user I'd expect it to be true by default.
{quote}
I think we should open an issue to fix this in 4.0... I don't care if everything
in the defaults^H^H^H^Hexample has it set to true, the factories should also default to true.
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch, SOLR-3054_2.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Tommaso Teofili (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tommaso Teofili updated SOLR-3054:
----------------------------------
Attachment: SOLR-3054_2.patch
Thanks Uwe, I've fixed the patch as per your comments.
Also I've added that TypeTokenFF implements the ResourceLoaderAware interface.
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch, SOLR-3054_2.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190719#comment-13190719 ]
Uwe Schindler commented on SOLR-3054:
-------------------------------------
+1, I will commit this later!
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch, SOLR-3054_2.patch, SOLR-3054_3.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3054) Add a TypeTokenFilterFactory
Posted by "Tommaso Teofili (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190749#comment-13190749 ]
Tommaso Teofili commented on SOLR-3054:
---------------------------------------
Thanks Uwe
> Add a TypeTokenFilterFactory
> ----------------------------
>
> Key: SOLR-3054
> URL: https://issues.apache.org/jira/browse/SOLR-3054
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Tommaso Teofili
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: SOLR-3054.patch, SOLR-3054_2.patch, SOLR-3054_3.patch
>
>
> Create a TypeTokenFilterFactory to make the TypeTokenFilter (filtering tokens depending on token types, see LUCENE-3671) available in Solr too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org