You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2010/03/30 23:26:27 UTC
[jira] Created: (LUCENE-2358) rename KeywordMarkerTokenFilter
rename KeywordMarkerTokenFilter
-------------------------------
Key: LUCENE-2358
URL: https://issues.apache.org/jira/browse/LUCENE-2358
Project: Lucene - Java
Issue Type: Task
Components: Analysis
Reporter: Robert Muir
Priority: Trivial
Attachments: LUCENE-2358.patch
I would like to rename KeywordMarkerTokenFilter to KeywordMarkerFilter.
We havent released it yet, so its a good time to keep the name brief and consistent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2358) rename KeywordMarkerTokenFilter
Posted by "Steven Rowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851652#action_12851652 ]
Steven Rowe commented on LUCENE-2358:
-------------------------------------
Hi Robert,
I'm working on a change to ShingleFilter to not output "_" filler token unigrams (or generally, filler-only ngrams, to cover the case where position increment gaps exceed n).
I needed to be able to mark cached tokens as being filler tokens (or not) - a boolean attribute. After trying to write a new private-use attribute and failing (I didn't make both an interface and an implementation, I think - I should figure it out and improve the docs I guess), I found KeywordAttribute and have used it to mark whether or not a cached token is a filler token (keyword:yes => filler-token:yes).
Would it make sense to have a generalized boolean attribute, specialized for keywords or (fill-in-the-blank)? It's a small leap to say that "iskeyword" means true for whatever boolean attribute you want to carry, so this isn't really a big deal, but I thought I'd bring it up while you're thinking about naming this thing.
(This may be a can of worms: if there is a generic boolean attribute, should there be generic string/int/float/etc. attributes too?)
Steve
> rename KeywordMarkerTokenFilter
> -------------------------------
>
> Key: LUCENE-2358
> URL: https://issues.apache.org/jira/browse/LUCENE-2358
> Project: Lucene - Java
> Issue Type: Task
> Components: Analysis
> Reporter: Robert Muir
> Priority: Trivial
> Attachments: LUCENE-2358.patch
>
>
> I would like to rename KeywordMarkerTokenFilter to KeywordMarkerFilter.
> We havent released it yet, so its a good time to keep the name brief and consistent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2358) rename KeywordMarkerTokenFilter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2358:
--------------------------------
Attachment: LUCENE-2358.patch
attached is a patch (really svn move of KeywordMarkerTokenFilter and its test will be done)
> rename KeywordMarkerTokenFilter
> -------------------------------
>
> Key: LUCENE-2358
> URL: https://issues.apache.org/jira/browse/LUCENE-2358
> Project: Lucene - Java
> Issue Type: Task
> Components: Analysis
> Reporter: Robert Muir
> Priority: Trivial
> Attachments: LUCENE-2358.patch
>
>
> I would like to rename KeywordMarkerTokenFilter to KeywordMarkerFilter.
> We havent released it yet, so its a good time to keep the name brief and consistent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2358) rename KeywordMarkerTokenFilter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved LUCENE-2358.
---------------------------------
Resolution: Fixed
Committed revision 932856.
> rename KeywordMarkerTokenFilter
> -------------------------------
>
> Key: LUCENE-2358
> URL: https://issues.apache.org/jira/browse/LUCENE-2358
> Project: Lucene - Java
> Issue Type: Task
> Components: Analysis
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Attachments: LUCENE-2358.patch
>
>
> I would like to rename KeywordMarkerTokenFilter to KeywordMarkerFilter.
> We havent released it yet, so its a good time to keep the name brief and consistent.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2358) rename KeywordMarkerTokenFilter
Posted by "Steven Rowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851659#action_12851659 ]
Steven Rowe commented on LUCENE-2358:
-------------------------------------
Sorry for cluttering this issue...
{quote}
I'm not really sure the KeywordAttribute is the best fit here, because its purpose is for the token
to not be changed by some later filter. I'm not sure how your filter works (I would have to see the patch),
but I think using this attribute for this purpose could introduce some bugs?
I guess the key is that its not a private-use attribute really, these things are visible by all tokenstreams.
so stemmers etc will see your 'internal' attribute.
{quote}
Yep, you're right, I hadn't thought it through that far.
{quote}
bq. Would it make sense to have a generalized boolean attribute [...]?
I don't really think so. Since there can only be one of any attribute in the tokenstream, you would have
various TokenFilters clashing on how they interpret and use some generic boolean attribute!
{quote}
Um, yes, I should have realized that...
(Re-writing private FillerTokenAttribute! Hooray!)
> rename KeywordMarkerTokenFilter
> -------------------------------
>
> Key: LUCENE-2358
> URL: https://issues.apache.org/jira/browse/LUCENE-2358
> Project: Lucene - Java
> Issue Type: Task
> Components: Analysis
> Reporter: Robert Muir
> Priority: Trivial
> Attachments: LUCENE-2358.patch
>
>
> I would like to rename KeywordMarkerTokenFilter to KeywordMarkerFilter.
> We havent released it yet, so its a good time to keep the name brief and consistent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-2358) rename KeywordMarkerTokenFilter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir reassigned LUCENE-2358:
-----------------------------------
Assignee: Robert Muir
> rename KeywordMarkerTokenFilter
> -------------------------------
>
> Key: LUCENE-2358
> URL: https://issues.apache.org/jira/browse/LUCENE-2358
> Project: Lucene - Java
> Issue Type: Task
> Components: Analysis
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Trivial
> Attachments: LUCENE-2358.patch
>
>
> I would like to rename KeywordMarkerTokenFilter to KeywordMarkerFilter.
> We havent released it yet, so its a good time to keep the name brief and consistent.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2358) rename KeywordMarkerTokenFilter
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851656#action_12851656 ]
Robert Muir commented on LUCENE-2358:
-------------------------------------
{quote}
I needed to be able to mark cached tokens as being filler tokens (or not) - a boolean attribute. After trying to write a new private-use attribute and failing (I didn't make both an interface and an implementation, I think - I should figure it out and improve the docs I guess), I found KeywordAttribute and have used it to mark whether or not a cached token is a filler token (keyword:yes => filler-token:yes).
{quote}
I'm not really sure the KeywordAttribute is the best fit here, because its purpose is for the token
to not be changed by some later filter. I'm not sure how your filter works (I would have to see the patch),
but I think using this attribute for this purpose could introduce some bugs?
I guess the key is that its not a private-use attribute really, these things are visible by all tokenstreams.
so stemmers etc will see your 'internal' attribute.
{quote}
Would it make sense to have a generalized boolean attribute, specialized for keywords or (fill-in-the-blank)? It's a small leap to say that "iskeyword" means true for whatever boolean attribute you want to carry, so this isn't really a big deal, but I thought I'd bring it up while you're thinking about naming this thing.
(This may be a can of worms: if there is a generic boolean attribute, should there be generic string/int/float/etc. attributes too?)
{quote}
I don't really think so. Since there can only be one of any attribute in the tokenstream, you would have
various TokenFilters clashing on how they interpret and use some generic boolean attribute!
> rename KeywordMarkerTokenFilter
> -------------------------------
>
> Key: LUCENE-2358
> URL: https://issues.apache.org/jira/browse/LUCENE-2358
> Project: Lucene - Java
> Issue Type: Task
> Components: Analysis
> Reporter: Robert Muir
> Priority: Trivial
> Attachments: LUCENE-2358.patch
>
>
> I would like to rename KeywordMarkerTokenFilter to KeywordMarkerFilter.
> We havent released it yet, so its a good time to keep the name brief and consistent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org