You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Chuck Williams <ch...@manawiz.com> on 2006/08/11 20:32:55 UTC
Strange behavior of positionIncrementGap
Hi All,
There is a strange treatment of positionIncrementGap in
DocumentWriter.invertDocument(). The gap is inserted between all
values of a field, except it is not inserted between values if the
prefix of the value list up to that point has not yet generated a token.
For example, if a field F has values A, B and C the following example
cases arise:
1. A and B both generate no tokens ==> no positionIncrementGaps are
generated
2. A has no tokens but B does ==> just the gap between B and C
3. A has tokens but B and C do not ==> both gaps between A and B, and
between B and C are generated
So, empty fields are treated anomalously. They are ignored for gap
purposes at the beginning of the field list, but included if they occur
later in the field list.
This issue caused a subtle bug in my bulk update operation because to
modify values and update the postings it must reanalyze them with
precisely the same positions used when they were originally indexed.
So, I had to match this previously unnoticed strange behavior.
I could post a patch to fix this, but am concerned it might introduce
upward incompatibilities in various implementations and applications
that are dependent on Lucene index format. If that is not a concern in
this case, please let me know and I'll post a patch. I at least wanted
to report it.
Chuck
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Strange behavior of positionIncrementGap
Posted by Chris Hostetter <ho...@fucit.org>.
: > 3) A _gap_ b _gap_ c _gap_ D ...results in: A _double_gap_ D
: >
: > ...is that the behavior you are seeing?
: >
: Almost. The only difference is that case 3 has 3 gaps, so it's A
: _triple_gap_ D.
sorry ... brain fart on my part.
: 1. My bulk update code was always generating the positionIncrementGap
: between all field values, so if there are 4 values it would always
: generate 3 gaps independent of whether or not the values generate
: This seems a natural behavior and is consistent with the use cases you
: describe (which are essentially the same reason I'm using gaps, and
: presumably the main purpose of gaps).
: Hoss, do you think it would be ok to fix given the potential upward
: incompatibility for index-format-dependent implementaitons?
it certainly seems like the "right thing to do" to me... i can't think of
any cases situations in which clients would be relying on the current
behavior.
Does anyone else see any reason why the current behavior is usefull?
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Strange behavior of positionIncrementGap
Posted by Chuck Williams <ch...@manawiz.com>.
Yonik Seeley wrote on 08/12/2006 05:08 AM:
> On 8/11/06, Chuck Williams <ch...@manawiz.com> wrote:
>> 1) a b C D ...results in: _gap_ _gap_ C _gap_ D
>> 2) a B C D ...results in: _gap_ B _gap_ C _gap_ D
>> 3) A b c D ...results in: A _gap_ _gap_ _gap_ D
>>
>> This seems a natural behavior and is consistent with the use cases you
>> describe (which are essentially the same reason I'm using gaps, and
>> presumably the main purpose of gaps).
>>
>> Hoss, do you think it would be ok to fix given the potential upward
>> incompatibility for index-format-dependent implementaitons?
>
> The proposed behavior seems fine to me...
> Is there any incompatibility other than the position gaps changing in
> the presence of fields empty after analysis?
No, the only change is the insertion of extra gaps between empty field
values at the beginning of a list of values for a given field name.
There are already such gaps between empty field values that occur
anywhere other than the beginning of the field value list. The idea is
to treat empty field values consistently, independent of whether they
occur at the beginning of the list or somewhere else in the list.
I'll post a patch for you guys to review, hopefully sometime this
weekend. The one downside of the patch is that it may require some
extra tracking data in DocumentWriter.invertDocument(). I think the
current behavior may have been coded as an expedient to reuse the
existing length tracking for the max-field length check.
Thanks,
Chuck
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Strange behavior of positionIncrementGap
Posted by Yonik Seeley <yo...@apache.org>.
On 8/11/06, Chuck Williams <ch...@manawiz.com> wrote:
> 1) a b C D ...results in: _gap_ _gap_ C _gap_ D
> 2) a B C D ...results in: _gap_ B _gap_ C _gap_ D
> 3) A b c D ...results in: A _gap_ _gap_ _gap_ D
>
> This seems a natural behavior and is consistent with the use cases you
> describe (which are essentially the same reason I'm using gaps, and
> presumably the main purpose of gaps).
>
> Hoss, do you think it would be ok to fix given the potential upward
> incompatibility for index-format-dependent implementaitons?
The proposed behavior seems fine to me...
Is there any incompatibility other than the position gaps changing in
the presence of fields empty after analysis?
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Strange behavior of positionIncrementGap
Posted by Chuck Williams <ch...@manawiz.com>.
Chris Hostetter wrote on 08/11/2006 09:08 AM:
> (using lower case
> to indicate no tokens produced and upper case to indicate tokens were
> produced) ...
>
> 1) a b C _gap_ D ...results in: C _gap_ D
> 2) a B _gap_ C _gap_ D ...results in: B _gap_ C _gap_ D
> 3) A _gap_ b _gap_ c _gap_ D ...results in: A _double_gap_ D
>
> ...is that the behavior you are seeing?
>
Almost. The only difference is that case 3 has 3 gaps, so it's A
_triple_gap_ D.
> Only case #3 seems "wrongish" to me there. ... i started to explain why i
> thought it made sense to go ahead and "fix this", where by fix i ment only
> insert one gap in case#3 ... and then realized i was acctually arguing in
> favor of the current behavior for case#3, here is why...
>
> based on the semi-frequently discussed usage of token gap sizes to
> denote sentence/paragraph/page boundaries for the purpose of sloppy
> phrase queries, it certianly seems worthwhile to fix to me (so that
> queries like "find Erik within 3 pages of Otis" still work even if one
> of those pages is blank ...
>
> ...that's when i realized the current behavior of case#3 is acctually
> important for accurate matching, otherwise a search for two words within a
> certain number of pages would have a false match if those pages were
> blank. case #1 seems fine, but case #2 seems like the "wrong" case to me
> know, becuase trying to find occurances of "B" on page #1 using a
> SpanFirst query will have false positives ... it seems like the
> positionIncrimentGap should always be called/used after any field value is
> added (even if the value results in no okens) before the next value is
> added (even if that value results in no tokens)
>
>
> Does this jive with what you were expecting, and the patch you were
> considering?
>
Precisely. The same concern about SpanFirstQuery also applies to case
1. My bulk update code was always generating the positionIncrementGap
between all field values, so if there are 4 values it would always
generate 3 gaps independent of whether or not the values generate
tokens. For your cases it generated:
1) a b C D ...results in: _gap_ _gap_ C _gap_ D
2) a B C D ...results in: _gap_ B _gap_ C _gap_ D
3) A b c D ...results in: A _gap_ _gap_ _gap_ D
This seems a natural behavior and is consistent with the use cases you
describe (which are essentially the same reason I'm using gaps, and
presumably the main purpose of gaps).
Hoss, do you think it would be ok to fix given the potential upward
incompatibility for index-format-dependent implementaitons?
Chuck
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Strange behavior of positionIncrementGap
Posted by Chris Hostetter <ho...@fucit.org>.
: For example, if a field F has values A, B and C the following example
: cases arise:
: 1. A and B both generate no tokens ==> no positionIncrementGaps are
: generated
: 2. A has no tokens but B does ==> just the gap between B and C
: 3. A has tokens but B and C do not ==> both gaps between A and B, and
: between B and C are generated
:
: So, empty fields are treated anomalously. They are ignored for gap
: purposes at the beginning of the field list, but included if they occur
: later in the field list.
Since positions are allways relative, i'm not sure i understnad how this
caused a problem in for you ... but I suspect it's because there's more to
what you describe ... in each of the 3 causes you outlined what happens if
there is a field value "D" which allways produces tokens? Based on your
description so far, i'm guessing the following scenerio (using lower case
to indicate no tokens produced and upper case to indicate tokens were
produced) ...
1) a b C _gap_ D ...results in: C _gap_ D
2) a B _gap_ C _gap_ D ...results in: B _gap_ C _gap_ D
3) A _gap_ b _gap_ c _gap_ D ...results in: A _double_gap_ D
...is that the behavior you are seeing?
Only case #3 seems "wrongish" to me there. ... i started to explain why i
thought it made sense to go ahead and "fix this", where by fix i ment only
insert one gap in case#3 ... and then realized i was acctually arguing in
favor of the current behavior for case#3, here is why...
based on the semi-frequently discussed usage of token gap sizes to
denote sentence/paragraph/page boundaries for the purpose of sloppy
phrase queries, it certianly seems worthwhile to fix to me (so that
queries like "find Erik within 3 pages of Otis" still work even if one
of those pages is blank ...
...that's when i realized the current behavior of case#3 is acctually
important for accurate matching, otherwise a search for two words within a
certain number of pages would have a false match if those pages were
blank. case #1 seems fine, but case #2 seems like the "wrong" case to me
know, becuase trying to find occurances of "B" on page #1 using a
SpanFirst query will have false positives ... it seems like the
positionIncrimentGap should always be called/used after any field value is
added (even if the value results in no okens) before the next value is
added (even if that value results in no tokens)
Does this jive with what you were expecting, and the patch you were
considering?
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org