You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Hoss Man (JIRA)" <ji...@apache.org> on 2007/12/18 21:03:43 UTC
[jira] Created: (LUCENE-1095) StopFilter should have option to incr
positionIncrement after stop word
StopFilter should have option to incr positionIncrement after stop word
-----------------------------------------------------------------------
Key: LUCENE-1095
URL: https://issues.apache.org/jira/browse/LUCENE-1095
Project: Lucene - Java
Issue Type: Improvement
Reporter: Hoss Man
I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1095) StopFilter should have option to
incr positionIncrement after stop word
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552903 ]
Doron Cohen commented on LUCENE-1095:
-------------------------------------
The problem Doug refers to is the effect of this on a Phrase Query.
I believe that if same analyzer with same stop filter and same option enables is used at search, the PhrasePositions would reflect the same increments due to stop word removals, and so phrase search should work.
I didn't check this though.
To be also verified for Span Queries.
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1095) StopFilter should have option to
incr positionIncrement after stop word
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552911 ]
Doron Cohen commented on LUCENE-1095:
-------------------------------------
I checked further.
Query parser currently ignores the position increment - it is calling PhraseQuery.add(Term) rather than add(Term,position).
So without changing this - either by default or optionally - the whole thing will not work.
(I think current QP behavior is wrong in this regard.)
Is there a good reason *not* to change QP to *always* pass the position to the PhraseQuery?
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-1095) StopFilter should have option to
incr positionIncrement after stop word
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen reassigned LUCENE-1095:
-----------------------------------
Assignee: Doron Cohen
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
> Assignee: Doron Cohen
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1095) StopFilter should have option to
incr positionIncrement after stop word
Posted by "Erik Hatcher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552905 ]
Erik Hatcher commented on LUCENE-1095:
--------------------------------------
I believe QueryParser has been fixed since that first change I made mentioned by Steven to account for positions returned from an Analyzer. So maybe all is well with fixing StopFilter now. Unit tests needed :)
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-1095) StopFilter should have
option to incr positionIncrement after stop word
Posted by "Steven Rowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552916 ]
steve_rowe edited comment on LUCENE-1095 at 12/18/07 2:29 PM:
---------------------------------------------------------------
{quote}
Is there a good reason *not* to change QP to *always* pass the position to the PhraseQuery?
{quote}
Backward compatibility :) In all seriousness, if QP's default behavior is changed, regardless of correctness, shouldn't it wait until 3.0?
Hoss's suggestion changes no defaults, so *caveat user* applies - no problem applying to trunk immediately. Maybe a different patch to change the default behavior in StopFilter and in QP for 3.0?
was (Author: steve_rowe):
{quote}
Is there a good reason *not* to change QP to *always* pass the position to the PhraseQuery?
{quote}
Backward compatibility :) In all seriousness, if QP's default behavior of is changed, regardless of correctness, shouldn't it wait until 3.0?
Hoss's suggestion changes no defaults, so *caveat user* applies - no problem applying to trunk immediately. Maybe a different patch to change the default behavior in StopFilter and in QP for 3.0?
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1095) StopFilter should have option to incr
positionIncrement after stop word
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen updated LUCENE-1095:
--------------------------------
Attachment: lucene-1095-pos-incr.patch
{quote}
... it does not assume incoming tokens to have position-increment of 1.
So it should work ok when stop filters and "expanders" are concatenated.
{quote}
Well, chaining stop filters didn't work in previous patch.
Attached patch fixed this.
Problem was with tokenizers reuse of tokens (next(Token)).
CharTokenizer, KeywordTokenizer and StandardTokenizer
were fixed to reset positionIncrement to 1.
The way I see it *Tokenizers must* do this and *Filters must not* do this.
If there's no objection to this observation I'll javadoc it in Tokenizer
for future implementors.
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
> Assignee: Doron Cohen
> Attachments: lucene-1095-pos-incr.patch, lucene-1095-pos-incr.patch
>
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1095) StopFilter should have option to
incr positionIncrement after stop word
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen resolved LUCENE-1095.
---------------------------------
Resolution: Fixed
Lucene Fields: [Patch Available] (was: [New])
Committed . (already yesterday actually, I was sure that I already resolved it...)
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
> Assignee: Doron Cohen
> Attachments: lucene-1095-pos-incr.patch, lucene-1095-pos-incr.patch, lucene-1095-pos-incr.patch
>
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1095) StopFilter should have option to
incr positionIncrement after stop word
Posted by "Steven Rowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552916 ]
Steven Rowe commented on LUCENE-1095:
-------------------------------------
{quote}
Is there a good reason *not* to change QP to *always* pass the position to the PhraseQuery?
{quote}
Backward compatibility :) In all seriousness, if QP's default behavior of is changed, regardless of correctness, shouldn't it wait until 3.0?
Hoss's suggestion changes no defaults, so *caveat user* applies - no problem applying to trunk immediately. Maybe a different patch to change the default behavior in StopFilter and in QP for 3.0?
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1095) StopFilter should have option to
incr positionIncrement after stop word
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554472 ]
Doron Cohen commented on LUCENE-1095:
-------------------------------------
Ok I separated the tokenizers fix to LUCENE-1101.
Patch here stands-alone (not depending on 1101), will
update it once 1101 gets committed.
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
> Assignee: Doron Cohen
> Attachments: lucene-1095-pos-incr.patch, lucene-1095-pos-incr.patch
>
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1095) StopFilter should have option to incr
positionIncrement after stop word
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen updated LUCENE-1095:
--------------------------------
Attachment: lucene-1095-pos-incr.patch
This patch enables increments optionally in QueryParser and StopFilter, as discussed above.
The default behavior has not changed, so there should be no back-comp issues.
The implementation in StopFilter differs slightly from the "older" reverted code in
that it does not assume incoming tokens to have position-increment of 1. So it
should work ok when stop filters and "expanders" are concatenated.
Added API:
{code}
QueryParser {
public boolean getApplyPositionIncrements();
public void setApplyPositionIncrements(boolean applyPositionIncrements);
}
{code}
And
{code}
StopFilter {
public boolean getApplyPositionIncrements();
public void setApplyPositionIncrements(boolean applyPositionIncrements);
public static boolean getDefaultApplyPositionIncrements()
public static void setDefaultApplyPositionIncrements(boolean defaultApplyPositionIncrements)
}
{code}
Note the two new static methods in StopFilter - these allow to control behavior system wide, and this way affect the behavior of classes that construct a StopFilter internally, like StopAnalyzer and StandardAnalyzer, without needing to pollute the API of many classes (all Snowball analyzers for instance) with new API.
Parse tests and search tests were added.
All tests pass.
Feedback is required:
* Comments on the proposed API:
** the static methods approach?
** perhaps "enable" instead of "apply"?
* Do you think this should be added now (it is ready I think) or would you rather to hold this for after 2.3?
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
> Assignee: Doron Cohen
> Attachments: lucene-1095-pos-incr.patch
>
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1095) StopFilter should have option to
incr positionIncrement after stop word
Posted by "Steven Rowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552901 ]
Steven Rowe commented on LUCENE-1095:
-------------------------------------
>From <http://mail-archives.apache.org/mod_mbox/lucene-java-user/200510.mbox/%3c435D44FA.6050702@syr.edu%3e>:
A patch implementing this approach was actually applied to
StopFilter.java in late 2003, but was reverted shortly afterward,
because this approach conflicts with the QueryParser and PhraseQuery
implementations.
See Doug Cutting's description of the problem with the position
increment modification approach here:
<http://mail-archives.apache.org/mod_mbox/lucene-java-dev/200312.mbox/%3c3FCFB3CA.9000103@lucene.com%3e>
See a colored diff of StopFilter.java, just before and after the
position increment modification patch was reverted, here:
<http://svn.apache.org/viewcvs.cgi/lucene/java/trunk/src/java/org/apache/lucene/analysis/StopFilter.java?rev=150152&r1=150150&r2=150152&diff_format=h>
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1095) StopFilter should have option to
incr positionIncrement after stop word
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552906 ]
Hoss Man commented on LUCENE-1095:
----------------------------------
I knew there must have been a good reason why it hadn't been done before, but there are some distinctions between then and now:
1) then the position incrementing wasn't optional, i wouldn't suggest making that some change, just making it easier for people who want to do this without having to write an entirely new Filter.
2) now PhraseQuery actually supports the notion of relative positions, so people can do the kind of thing Doug was talking about in the email you linked to ... but i won't go so far as to suggest we should make those changes by default in the QueryParser
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1095) StopFilter should have option to incr
positionIncrement after stop word
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen updated LUCENE-1095:
--------------------------------
Attachment: lucene-1095-pos-incr.patch
Updated patch:
* Token reuse fixes omitted (handled by LUCENE-1101).
* new feature methods renamed from "apply" to "enable".
All tests pass, default behavior unmodified, ready to commit.
> StopFilter should have option to incr positionIncrement after stop word
> -----------------------------------------------------------------------
>
> Key: LUCENE-1095
> URL: https://issues.apache.org/jira/browse/LUCENE-1095
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Hoss Man
> Assignee: Doron Cohen
> Attachments: lucene-1095-pos-incr.patch, lucene-1095-pos-incr.patch, lucene-1095-pos-incr.patch
>
>
> I've seen this come up on the mailing list a few times in the last month, so i'm filing a known bug/improvement arround it...
> StopFilter should have an option that if set, records how many stop words are "skipped" in a row, and then sets that value as the positionIncrement on the "next" token that StopFilter does return.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org