You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Shai Erera <se...@gmail.com> on 2014/09/03 16:51:50 UTC
NearSpansOrdered semantics
Hi
I am playing with searching over annotated text - that is some stream of
text is indexed under one field, and I have annotations indexed under
another field. The annotations are indexed with custom positions that I
inject, based on the original position of the text.
So for example I might have a piece of text "brown fox" at positions 15,16
that is annotated as "nickname". In that case the annotation constitutes
two terms: annot:nick and annoth:/nick in positions 15 and 16 respectively.
I then implemented a variation of a span query which I call SpanWithinQuery
which allows me to search for "fox WITHIN nickname". The way it's
implemented is a SpanNearQuery on annot:nick and annot:/nick, and whenever
a match is found for 'fox', it makes sure that the position falls within
the range of the nick annotation. Quite simple.
The problem I ran into is when the annotation spans a single position, in
which case the start and end annotation are indexed in the same position.
So e.g. in the example above, if "brown" was further annotated as "color",
I'd have annot:color and annot:/color indexed in position 15. Now when I
execute a SNQ(annot:nick, annot:/nick, slop=MAX_INT, ordered=true), I get 0
results.
I tracked this down to NearSpansOrdered.docSpansOrdered(), which documents
and implements the check to require that if the spans start in the same
position, their end position must not be the same.
If you forget about all the preface I wrote above, I have a basic question
about the semantics of SNQ. I tried a simple case -- SNQ("fox", "fox") and
I get 0 results. Is this a bug in SNQ itself, or it's not built to find
spans that are identical in their start/end positions, and I should use
another variant of SpanQuery?
BTW, I'm aware that the way I play with the annotated text could probably
be done in other ways, e.g. index annot:color with a payload with the span
range, but for now I'd like to know if SNQ (or NearSpansOrdered) has a bug.
Shai
Re: NearSpansOrdered semantics
Posted by Paul Elschot <pa...@gmail.com>.
By the way, in case you need more complex positional relations between fields, have a look at LUCENE-5627 .
A spanwithinquery is also used in the test code there as an unordered spannear with 2 arguments and distance -1.
Regards,
Paul Elschot
On 3 september 2014 20:23:46 CEST, Shai Erera <se...@gmail.com> wrote:
>Thanks Hoss, this is very useful information, especially
>FieldMaskingSpanQuery :).
>
>So according to those issues, it seems others also think that end1 <=
>end2
>is also a "correct" behavior of SNQ (in case ordered=true, and
>start1==start2). Was there a reason these issues were not resolved
>(e.g.
>LUCENE-3120)? I'm asking if there was additional discussion where
>people
>raised objections against this, or was it just a matter of lack of
>time/focus?
>
>Shai
>
>
>On Wed, Sep 3, 2014 at 8:47 PM, Chris Hostetter
><ho...@fucit.org>
>wrote:
>
>> : I then implemented a variation of a span query which I call
>> SpanWithinQuery
>> : which allows me to search for "fox WITHIN nickname". The way it's
>> : implemented is a SpanNearQuery on annot:nick and annot:/nick, and
>> whenever
>> : a match is found for 'fox', it makes sure that the position falls
>within
>> : the range of the nick annotation. Quite simple.
>>
>> FWIW: i think what you are describing is just a special case of
>> FieldMaskingSpanQuery, might save you some custom code -- but that's
>a
>> tangent.
>>
>> : If you forget about all the preface I wrote above, I have a basic
>> question
>> : about the semantics of SNQ. I tried a simple case -- SNQ("fox",
>"fox")
>> and
>> : I get 0 results. Is this a bug in SNQ itself, or it's not built to
>find
>> : spans that are identical in their start/end positions, and I should
>use
>> : another variant of SpanQuery?
>>
>> I'm not lookin at the code, and i'ts been a looooong while since i
>thouht
>> about phrases/spans in depth, but i seem to recall this being a known
>> aspect of the behavior ... similar to how PhraseQuery behaves i
>think?
>>
>> since you have "ordered=true" then the 2nd span must come *after* the
>> first span, and "after" is defined based on the end position of hte
>spans
>> (aparently?)
>>
>> You'll find a couple issues in jira (both closed and open) discussing
>what
>> the "right" behavior of SpanNearQuery should be. Some particularly
>> on-point issues based on a quick skim...
>>
>> https://issues.apache.org/jira/browse/LUCENE-3120
>> https://issues.apache.org/jira/browse/LUCENE-3371
>> https://issues.apache.org/jira/browse/LUCENE-3229
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
Re: NearSpansOrdered semantics
Posted by Shai Erera <se...@gmail.com>.
I dug some more into this, and e.g. compared how does PhraseQuery behave on
these cases ('a' and 'b' are indexed at the same position):
"a b" - NO MATCH (expected)
"a b"~1+ (1+ - any slop >= 1) - MATCH
Because of the mathematics behind PQ, the slop determines the edit distance
to match 'a' and 'b'. So in slop=1, 'b' is moved one place, *on-to* 'a',
and since they're in the same position, it's a match. If for instance they
were indexed in successive positions, to match "b a" you'd need to do "b
a"~2: 1 move to get 'b' on 'a' and another to get 'b' after 'a'.
SpanNearQuery, UnOrdered behaves the same and matches "a b" with slop=1. In
Ordered mode, I guess the rationale was that someone requested "b follows
a"... there is also further evidence in SpanTermQuery, or actually
TermSpans that implement end() to return position+1 (i.e. the end span is
EXCLUSIVE).
We could add a flag/mode/whatever to allow users to extend SNQ and change
the logic to make the span range INCLUSIVE, thus matching "a b" even when
they are indexed at the same position. But that won't be enough, and we'll
need to add such logic e.g at least to TermSpans, so that its end() returns
'position' and not off-by-1.
I think for now I'll just implement my own SpanQuery, and then see if by
opening simple hooks (that don't complicate the API), I could reuse any of
the existing impls.
Shai
On Thu, Sep 4, 2014 at 7:51 PM, Chris Hostetter <ho...@fucit.org>
wrote:
>
> : So according to those issues, it seems others also think that end1 <=
> end2
> : is also a "correct" behavior of SNQ (in case ordered=true, and
>
> A more accurate way ot put it may be that *some* people felt like it was
> the lest counter-intuitive behavior they could think of, and everybody
> else either didn't have an opinion, didn't share an opinion, or didn't
> understand spans enough to know if they had an opinion.
>
> : start1==start2). Was there a reason these issues were not resolved (e.g.
> : LUCENE-3120)? I'm asking if there was additional discussion where people
> : raised objections against this, or was it just a matter of lack of
> : time/focus?
>
> you could probably find more wisps of discussion about spans and nearness
> in general in the list archives, but i don't remember anything really
> concrete beyond the thread mentioned in LUCENE-3120 (and the earlier
> thread linked ot from it) ... and the only reason i remember that is
> because it wsa easy to find from the jira...
>
>
> https://mail-archives.apache.org/mod_mbox/lucene-dev/201105.mbox/%3C614C529D389A5944B351F7DFB7594F240126DA5B@uksrpblkexb01.detica.com%3E
>
>
> https://mail-archives.apache.org/mod_mbox/lucene-dev/200612.mbox/%3C45788E3D.20004@istandfor.com%3E
>
> : > https://issues.apache.org/jira/browse/LUCENE-3120
> : > https://issues.apache.org/jira/browse/LUCENE-3371
> : > https://issues.apache.org/jira/browse/LUCENE-3229
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: NearSpansOrdered semantics
Posted by Chris Hostetter <ho...@fucit.org>.
: So according to those issues, it seems others also think that end1 <= end2
: is also a "correct" behavior of SNQ (in case ordered=true, and
A more accurate way ot put it may be that *some* people felt like it was
the lest counter-intuitive behavior they could think of, and everybody
else either didn't have an opinion, didn't share an opinion, or didn't
understand spans enough to know if they had an opinion.
: start1==start2). Was there a reason these issues were not resolved (e.g.
: LUCENE-3120)? I'm asking if there was additional discussion where people
: raised objections against this, or was it just a matter of lack of
: time/focus?
you could probably find more wisps of discussion about spans and nearness
in general in the list archives, but i don't remember anything really
concrete beyond the thread mentioned in LUCENE-3120 (and the earlier
thread linked ot from it) ... and the only reason i remember that is
because it wsa easy to find from the jira...
https://mail-archives.apache.org/mod_mbox/lucene-dev/201105.mbox/%3C614C529D389A5944B351F7DFB7594F240126DA5B@uksrpblkexb01.detica.com%3E
https://mail-archives.apache.org/mod_mbox/lucene-dev/200612.mbox/%3C45788E3D.20004@istandfor.com%3E
: > https://issues.apache.org/jira/browse/LUCENE-3120
: > https://issues.apache.org/jira/browse/LUCENE-3371
: > https://issues.apache.org/jira/browse/LUCENE-3229
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: NearSpansOrdered semantics
Posted by Shai Erera <se...@gmail.com>.
Thanks Hoss, this is very useful information, especially
FieldMaskingSpanQuery :).
So according to those issues, it seems others also think that end1 <= end2
is also a "correct" behavior of SNQ (in case ordered=true, and
start1==start2). Was there a reason these issues were not resolved (e.g.
LUCENE-3120)? I'm asking if there was additional discussion where people
raised objections against this, or was it just a matter of lack of
time/focus?
Shai
On Wed, Sep 3, 2014 at 8:47 PM, Chris Hostetter <ho...@fucit.org>
wrote:
> : I then implemented a variation of a span query which I call
> SpanWithinQuery
> : which allows me to search for "fox WITHIN nickname". The way it's
> : implemented is a SpanNearQuery on annot:nick and annot:/nick, and
> whenever
> : a match is found for 'fox', it makes sure that the position falls within
> : the range of the nick annotation. Quite simple.
>
> FWIW: i think what you are describing is just a special case of
> FieldMaskingSpanQuery, might save you some custom code -- but that's a
> tangent.
>
> : If you forget about all the preface I wrote above, I have a basic
> question
> : about the semantics of SNQ. I tried a simple case -- SNQ("fox", "fox")
> and
> : I get 0 results. Is this a bug in SNQ itself, or it's not built to find
> : spans that are identical in their start/end positions, and I should use
> : another variant of SpanQuery?
>
> I'm not lookin at the code, and i'ts been a looooong while since i thouht
> about phrases/spans in depth, but i seem to recall this being a known
> aspect of the behavior ... similar to how PhraseQuery behaves i think?
>
> since you have "ordered=true" then the 2nd span must come *after* the
> first span, and "after" is defined based on the end position of hte spans
> (aparently?)
>
> You'll find a couple issues in jira (both closed and open) discussing what
> the "right" behavior of SpanNearQuery should be. Some particularly
> on-point issues based on a quick skim...
>
> https://issues.apache.org/jira/browse/LUCENE-3120
> https://issues.apache.org/jira/browse/LUCENE-3371
> https://issues.apache.org/jira/browse/LUCENE-3229
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: NearSpansOrdered semantics
Posted by Chris Hostetter <ho...@fucit.org>.
: I then implemented a variation of a span query which I call SpanWithinQuery
: which allows me to search for "fox WITHIN nickname". The way it's
: implemented is a SpanNearQuery on annot:nick and annot:/nick, and whenever
: a match is found for 'fox', it makes sure that the position falls within
: the range of the nick annotation. Quite simple.
FWIW: i think what you are describing is just a special case of
FieldMaskingSpanQuery, might save you some custom code -- but that's a
tangent.
: If you forget about all the preface I wrote above, I have a basic question
: about the semantics of SNQ. I tried a simple case -- SNQ("fox", "fox") and
: I get 0 results. Is this a bug in SNQ itself, or it's not built to find
: spans that are identical in their start/end positions, and I should use
: another variant of SpanQuery?
I'm not lookin at the code, and i'ts been a looooong while since i thouht
about phrases/spans in depth, but i seem to recall this being a known
aspect of the behavior ... similar to how PhraseQuery behaves i think?
since you have "ordered=true" then the 2nd span must come *after* the
first span, and "after" is defined based on the end position of hte spans
(aparently?)
You'll find a couple issues in jira (both closed and open) discussing what
the "right" behavior of SpanNearQuery should be. Some particularly
on-point issues based on a quick skim...
https://issues.apache.org/jira/browse/LUCENE-3120
https://issues.apache.org/jira/browse/LUCENE-3371
https://issues.apache.org/jira/browse/LUCENE-3229
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org