You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Gregory Tarr <Gr...@detica.com> on 2011/05/09 13:28:53 UTC
SpanNearQuery - inOrder parameter
I attach a junit test which shows strange behaviour of the inOrder
parameter on the SpanNearQuery constructor, using Lucene 2.9.4.
My understanding of this parameter is that true forces the order and
false doesn't care about the order.
Using true always works. However using false works fine when the terms
in the query are distinct, but if they are equivalent, e.g. searching
for "john john", I do not get the expected results. The workaround seems
to be to always use true for queries with repeated terms.
Any help?
Thanks
Greg
<<TestSpanNearQueryInOrder.java>>
Please consider the environment before printing this email.
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.
Re: SpanNearQuery - inOrder parameter
Posted by Tom Hill <so...@worldware.com>.
Since no one else is jumping in, I'll say that I suspect that the span
query code does not bother to check to see if two of the terms are the
same.
I think that would account for the behavior you are seeing. Since the
second SpanTermQuery would match the same term the first one did.
Note that I'm not familiar with the span query code, so this is just a
quick deduction.
Not sure how easy it would be to add this duplicate term detection, if
that's the problem.
Tom
On Tue, May 10, 2011 at 5:58 AM, Gregory Tarr <Gr...@detica.com> wrote:
> Anyone able to help me with the problem below?
>
> Thanks
>
> Greg
>
> -----Original Message-----
> From: Gregory Tarr [mailto:Gregory.tarr@detica.com]
> Sent: 09 May 2011 12:33
> To: java-user@lucene.apache.org
> Subject: RE: SpanNearQuery - inOrder parameter
>
> Attachment didn't work - test below:
>
> import org.apache.lucene.analysis.standard.StandardAnalyzer;
> import org.apache.lucene.document.Document;
> import org.apache.lucene.document.Field; import
> org.apache.lucene.index.IndexWriter;
> import org.apache.lucene.index.Term;
> import org.apache.lucene.search.IndexSearcher;
> import org.apache.lucene.search.TopDocsCollector;
> import org.apache.lucene.search.TopScoreDocCollector;
> import org.apache.lucene.search.spans.SpanNearQuery;
> import org.apache.lucene.search.spans.SpanQuery;
> import org.apache.lucene.search.spans.SpanTermQuery;
> import org.apache.lucene.store.RAMDirectory;
> import org.apache.lucene.util.Version;
> import org.junit.Assert;
> import org.junit.Test;
>
> public class TestSpanNearQueryInOrder {
>
> @Test
> public void testSpanNearQueryInOrder() { RAMDirectory directory = new
> RAMDirectory(); IndexWriter writer = new IndexWriter(directory, new
> StandardAnalyzer(Version.LUCENE_29), true,
> IndexWriter.MaxFieldLength.UNLIMITED);
> TopDocsCollector collector = TopScoreDocCollector.create(3, false);
>
> Document doc = new Document();
>
> // DOC1
> doc.add(new Field("text","dddd aaaa bbbb cccc", Field.Store.YES,
> Field.Index.ANALYZED));
>
> writer.addDocument(doc);
> doc = new Document();
>
> // DOC2
> doc.add(new Field("text","dddd aaaa aaaa cccc"));
>
> writer.addDocument(doc);
> doc = new Document();
>
> // DOC3
> doc.add(new Field("text","dddd aaaa yyyy aaaa xxxx cccc"));
>
> writer.addDocument(doc);
> writer.optimize();
> writer.close();
>
> searcher = new IndexSearcher(directory, false);
>
> SpanQuery[] clauses = new SpanQuery[2]; clauses[0] = new
> SpanTermQuery(new Term("text", "aaaa")); clauses[1] = new
> SpanTermQuery(new Term("text", "aaaa"));
>
> // Don't care about order, so setting inOrder = false SpanNearQuery q
> = new SpanNearQuery(clauses, 1, false); searcher.search(q, collector);
>
> // This assert fails - 3 docs are returned. Expecting only DOC2 and
> DOC3
> Assert.assertEquals("Check 2 results", 2, collector.getTotalHits());
>
> collector = new TopScoreDocCollector.create(3, false); clauses = new
> SpanQuery[2]; clauses[0] = new SpanTermQuery(new Term("text", "aaaa"));
> clauses[1] = new SpanTermQuery(new Term("text", "aaaa"));
>
> // Don't care about order, so setting inOrder = false q = new
> SpanNearQuery(clauses, 0, false); searcher.search(q, collector);
>
> // This assert fails - 3 docs are returned. Expecting only DOC2
> Assert.assertEquals("Check 1 result", 1, collector.getTotalHits()); }
>
> }
>
> ________________________________
>
> From: Gregory Tarr [mailto:Gregory.tarr@detica.com]
> Sent: 09 May 2011 12:29
> To: java-user@lucene.apache.org
> Subject: SpanNearQuery - inOrder parameter
>
>
>
> I attach a junit test which shows strange behaviour of the inOrder
> parameter on the SpanNearQuery constructor, using Lucene 2.9.4.
>
> My understanding of this parameter is that true forces the order and
> false doesn't care about the order.
>
> Using true always works. However using false works fine when the terms
> in the query are distinct, but if they are equivalent, e.g. searching
> for "john john", I do not get the expected results. The workaround seems
> to be to always use true for queries with repeated terms.
>
> Any help?
>
> Thanks
>
> Greg
>
> <<TestSpanNearQueryInOrder.java>>
>
>
> Please consider the environment before printing this email.
>
> This message should be regarded as confidential. If you have received
> this email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard
> copy by an authorised signatory. The contents of this email may relate
> to dealings with other companies within the Detica Limited group of
> companies.
>
> Detica Limited is registered in England under No: 1337451.
>
> Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP,
> England.
>
>
> Please consider the environment before printing this email.
>
> This message should be regarded as confidential. If you have received
> this email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard
> copy by an authorised signatory. The contents of this email may relate
> to dealings with other companies within the Detica Limited group of
> companies.
>
> Detica Limited is registered in England under No: 1337451.
>
> Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP,
> England.
>
> Please consider the environment before printing this email.
>
> This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.
>
> Detica Limited is registered in England under No: 1337451.
>
> Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: SpanNearQuery - inOrder parameter
Posted by Gregory Tarr <Gr...@detica.com>.
Anyone able to help me with the problem below?
Thanks
Greg
-----Original Message-----
From: Gregory Tarr [mailto:Gregory.tarr@detica.com]
Sent: 09 May 2011 12:33
To: java-user@lucene.apache.org
Subject: RE: SpanNearQuery - inOrder parameter
Attachment didn't work - test below:
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field; import
org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.TopDocsCollector;
import org.apache.lucene.search.TopScoreDocCollector;
import org.apache.lucene.search.spans.SpanNearQuery;
import org.apache.lucene.search.spans.SpanQuery;
import org.apache.lucene.search.spans.SpanTermQuery;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.Version;
import org.junit.Assert;
import org.junit.Test;
public class TestSpanNearQueryInOrder {
@Test
public void testSpanNearQueryInOrder() { RAMDirectory directory = new
RAMDirectory(); IndexWriter writer = new IndexWriter(directory, new
StandardAnalyzer(Version.LUCENE_29), true,
IndexWriter.MaxFieldLength.UNLIMITED);
TopDocsCollector collector = TopScoreDocCollector.create(3, false);
Document doc = new Document();
// DOC1
doc.add(new Field("text","dddd aaaa bbbb cccc", Field.Store.YES,
Field.Index.ANALYZED));
writer.addDocument(doc);
doc = new Document();
// DOC2
doc.add(new Field("text","dddd aaaa aaaa cccc"));
writer.addDocument(doc);
doc = new Document();
// DOC3
doc.add(new Field("text","dddd aaaa yyyy aaaa xxxx cccc"));
writer.addDocument(doc);
writer.optimize();
writer.close();
searcher = new IndexSearcher(directory, false);
SpanQuery[] clauses = new SpanQuery[2]; clauses[0] = new
SpanTermQuery(new Term("text", "aaaa")); clauses[1] = new
SpanTermQuery(new Term("text", "aaaa"));
// Don't care about order, so setting inOrder = false SpanNearQuery q
= new SpanNearQuery(clauses, 1, false); searcher.search(q, collector);
// This assert fails - 3 docs are returned. Expecting only DOC2 and
DOC3
Assert.assertEquals("Check 2 results", 2, collector.getTotalHits());
collector = new TopScoreDocCollector.create(3, false); clauses = new
SpanQuery[2]; clauses[0] = new SpanTermQuery(new Term("text", "aaaa"));
clauses[1] = new SpanTermQuery(new Term("text", "aaaa"));
// Don't care about order, so setting inOrder = false q = new
SpanNearQuery(clauses, 0, false); searcher.search(q, collector);
// This assert fails - 3 docs are returned. Expecting only DOC2
Assert.assertEquals("Check 1 result", 1, collector.getTotalHits()); }
}
________________________________
From: Gregory Tarr [mailto:Gregory.tarr@detica.com]
Sent: 09 May 2011 12:29
To: java-user@lucene.apache.org
Subject: SpanNearQuery - inOrder parameter
I attach a junit test which shows strange behaviour of the inOrder
parameter on the SpanNearQuery constructor, using Lucene 2.9.4.
My understanding of this parameter is that true forces the order and
false doesn't care about the order.
Using true always works. However using false works fine when the terms
in the query are distinct, but if they are equivalent, e.g. searching
for "john john", I do not get the expected results. The workaround seems
to be to always use true for queries with repeated terms.
Any help?
Thanks
Greg
<<TestSpanNearQueryInOrder.java>>
Please consider the environment before printing this email.
This message should be regarded as confidential. If you have received
this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard
copy by an authorised signatory. The contents of this email may relate
to dealings with other companies within the Detica Limited group of
companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP,
England.
Please consider the environment before printing this email.
This message should be regarded as confidential. If you have received
this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard
copy by an authorised signatory. The contents of this email may relate
to dealings with other companies within the Detica Limited group of
companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP,
England.
Please consider the environment before printing this email.
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: SpanNearQuery - inOrder parameter
Posted by Gregory Tarr <Gr...@detica.com>.
Chris, and others
Thanks for your reply. In effect what you are saying is that
SpanNearQuery works as expected, and I should set inOrder=true to obtain
the behaviour I require, even though I don't care about the order?
Thanks
Greg
-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
Sent: 11 May 2011 00:32
To: java-user@lucene.apache.org
Subject: RE: SpanNearQuery - inOrder parameter
: I attach a junit test which shows strange behaviour of the inOrder
: parameter on the SpanNearQuery constructor, using Lucene 2.9.4.
:
: My understanding of this parameter is that true forces the order and
: false doesn't care about the order.
:
: Using true always works. However using false works fine when the terms
: in the query are distinct, but if they are equivalent, e.g. searching
: for "john john", I do not get the expected results. The workaround
seems
: to be to always use true for queries with repeated terms.
I don't think the situation of "overlapping spans" has changed much
since this thread...
http://search.lucidimagination.com/search/document/ee23395e5a93c525/non_
overlapping_span_queries#868b3a3ec6431afc
the crux of hte issue (as i recall) is that there is really no
conecptual reason to why a query for "'john' near 'john', in any order,
with slop of Z" shouldn't match a doc that contains only one instance of
"john" ... the first SpanTermQuery says "i found a match at position X"
the second SpanTermQuery says "i found a match at position Y" and the
SpanNearQuery says "the differnece between X and Y is less then Z"
therefore i have a match. (The SpanNearQuery can't fail just because X
and Y are the same -- they might be two distinct term instances, with
differnet payloads perhaps, that just happen to have the same position).
However: if true==inOrder case works because the SpanNearQuery enforces
that "X must be less then Y" so the same term can't ever match twice.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Please consider the environment before printing this email.
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: SpanNearQuery - inOrder parameter
Posted by Doron Cohen <cd...@gmail.com>.
Hi Greg,
On Thu, May 19, 2011 at 12:26 PM, Gregory Tarr <Gr...@detica.com>wrote:
> We let our users decide whether they want to force the order or not, so
> in effect they pass in "inOrder".
>
> I would have to detect a repeated term and change the parameter as a
> result of that in order to workround this - I'd rather not do that though.
>
I see, ok, thanks for reporting this and for describing the use case, let's
see what others think/comment about this in JIRA... Please feel more than
welcome to investigate further and submit a patch to solve this! :)
Doron
RE: SpanNearQuery - inOrder parameter
Posted by Gregory Tarr <Gr...@detica.com>.
Doron
We let our users decide whether they want to force the order or not, so
in effect they pass in "inOrder".
I would have to detect a repeated term and change the parameter as a
result of that in order to workround this - I'd rather not do that
though.
Thanks
Greg
-----Original Message-----
From: Doron Cohen [mailto:cdoronc@gmail.com]
Sent: 19 May 2011 10:22
To: java-user@lucene.apache.org
Subject: Re: SpanNearQuery - inOrder parameter
Hi Greg,
I created http://issues.apache.org/jira/browse/LUCENE-3120 for this
problem, and attached there a more general test that exposes this
problem, based on your test case.
I am not sure yet that this is indeed a problem to be fixed with regard
to span queries (see more there in JIRA) but at least I would like the
test case to be in JIRA.
Could you provide some context on why is this a problem for you?
That is, what is preventing you from specifying in-order?
Doron
On Tue, May 17, 2011 at 12:56 PM, Gregory Tarr
<Gr...@detica.com>wrote:
> Anyone else able to reply to this?
>
> Thanks
>
> Greg
>
> -----Original Message-----
> From: Gregory Tarr
> Sent: 13 May 2011 15:46
> To: 'java-user@lucene.apache.org'
> Subject: RE: SpanNearQuery - inOrder parameter
>
> Chris, and others
>
> Thanks for your reply. In effect what you are saying is that
> SpanNearQuery works as expected, and I should set inOrder=true to
> obtain the behaviour I require, even though I don't care about the
order?
>
> Thanks
>
> Greg
>
> -----Original Message-----
> From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
> Sent: 11 May 2011 00:32
> To: java-user@lucene.apache.org
> Subject: RE: SpanNearQuery - inOrder parameter
>
>
>
> : I attach a junit test which shows strange behaviour of the inOrder
> : parameter on the SpanNearQuery constructor, using Lucene 2.9.4.
> :
> : My understanding of this parameter is that true forces the order and
> : false doesn't care about the order.
> :
> : Using true always works. However using false works fine when the
> terms
> : in the query are distinct, but if they are equivalent, e.g.
> searching
> : for "john john", I do not get the expected results. The workaround
> seems
> : to be to always use true for queries with repeated terms.
>
> I don't think the situation of "overlapping spans" has changed much
> since this thread...
>
> http://search.lucidimagination.com/search/document/ee23395e5a93c525/no
> n_ overlapping_span_queries#868b3a3ec6431afc
>
> the crux of hte issue (as i recall) is that there is really no
> conecptual reason to why a query for "'john' near 'john', in any
> order, with slop of Z" shouldn't match a doc that contains only one
> instance of "john" ... the first SpanTermQuery says "i found a match
at position X"
> the second SpanTermQuery says "i found a match at position Y" and the
> SpanNearQuery says "the differnece between X and Y is less then Z"
> therefore i have a match. (The SpanNearQuery can't fail just because
> X and Y are the same -- they might be two distinct term instances,
> with differnet payloads perhaps, that just happen to have the same
position).
>
> However: if true==inOrder case works because the SpanNearQuery
> enforces that "X must be less then Y" so the same term can't ever
match twice.
>
>
>
> -Hoss
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> Please consider the environment before printing this email.
>
> This message should be regarded as confidential. If you have received
> this email in error please notify the sender and destroy it
immediately.
> Statements of intent shall only become binding when confirmed in hard
> copy by an authorised signatory. The contents of this email may
> relate to dealings with other companies within the Detica Limited
group of companies.
>
> Detica Limited is registered in England under No: 1337451.
>
> Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP,
> England.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Please consider the environment before printing this email.
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: SpanNearQuery - inOrder parameter
Posted by Doron Cohen <cd...@gmail.com>.
Hi Greg,
I created http://issues.apache.org/jira/browse/LUCENE-3120 for this problem,
and attached there a more general test that exposes this problem, based on
your test case.
I am not sure yet that this is indeed a problem to be fixed with regard to
span queries (see more there in JIRA) but at least I would like the test
case to be in JIRA.
Could you provide some context on why is this a problem for you?
That is, what is preventing you from specifying in-order?
Doron
On Tue, May 17, 2011 at 12:56 PM, Gregory Tarr <Gr...@detica.com>wrote:
> Anyone else able to reply to this?
>
> Thanks
>
> Greg
>
> -----Original Message-----
> From: Gregory Tarr
> Sent: 13 May 2011 15:46
> To: 'java-user@lucene.apache.org'
> Subject: RE: SpanNearQuery - inOrder parameter
>
> Chris, and others
>
> Thanks for your reply. In effect what you are saying is that
> SpanNearQuery works as expected, and I should set inOrder=true to obtain
> the behaviour I require, even though I don't care about the order?
>
> Thanks
>
> Greg
>
> -----Original Message-----
> From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
> Sent: 11 May 2011 00:32
> To: java-user@lucene.apache.org
> Subject: RE: SpanNearQuery - inOrder parameter
>
>
>
> : I attach a junit test which shows strange behaviour of the inOrder
> : parameter on the SpanNearQuery constructor, using Lucene 2.9.4.
> :
> : My understanding of this parameter is that true forces the order and
> : false doesn't care about the order.
> :
> : Using true always works. However using false works fine when the terms
> : in the query are distinct, but if they are equivalent, e.g. searching
> : for "john john", I do not get the expected results. The workaround
> seems
> : to be to always use true for queries with repeated terms.
>
> I don't think the situation of "overlapping spans" has changed much
> since this thread...
>
> http://search.lucidimagination.com/search/document/ee23395e5a93c525/non_
> overlapping_span_queries#868b3a3ec6431afc
>
> the crux of hte issue (as i recall) is that there is really no
> conecptual reason to why a query for "'john' near 'john', in any order,
> with slop of Z" shouldn't match a doc that contains only one instance of
> "john" ... the first SpanTermQuery says "i found a match at position X"
> the second SpanTermQuery says "i found a match at position Y" and the
> SpanNearQuery says "the differnece between X and Y is less then Z"
> therefore i have a match. (The SpanNearQuery can't fail just because X
> and Y are the same -- they might be two distinct term instances, with
> differnet payloads perhaps, that just happen to have the same position).
>
> However: if true==inOrder case works because the SpanNearQuery enforces
> that "X must be less then Y" so the same term can't ever match twice.
>
>
>
> -Hoss
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> Please consider the environment before printing this email.
>
> This message should be regarded as confidential. If you have received this
> email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard copy
> by an authorised signatory. The contents of this email may relate to
> dealings with other companies within the Detica Limited group of companies.
>
> Detica Limited is registered in England under No: 1337451.
>
> Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP,
> England.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: SpanNearQuery - inOrder parameter
Posted by Gregory Tarr <Gr...@detica.com>.
Anyone else able to reply to this?
Thanks
Greg
-----Original Message-----
From: Gregory Tarr
Sent: 13 May 2011 15:46
To: 'java-user@lucene.apache.org'
Subject: RE: SpanNearQuery - inOrder parameter
Chris, and others
Thanks for your reply. In effect what you are saying is that
SpanNearQuery works as expected, and I should set inOrder=true to obtain
the behaviour I require, even though I don't care about the order?
Thanks
Greg
-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
Sent: 11 May 2011 00:32
To: java-user@lucene.apache.org
Subject: RE: SpanNearQuery - inOrder parameter
: I attach a junit test which shows strange behaviour of the inOrder
: parameter on the SpanNearQuery constructor, using Lucene 2.9.4.
:
: My understanding of this parameter is that true forces the order and
: false doesn't care about the order.
:
: Using true always works. However using false works fine when the terms
: in the query are distinct, but if they are equivalent, e.g. searching
: for "john john", I do not get the expected results. The workaround
seems
: to be to always use true for queries with repeated terms.
I don't think the situation of "overlapping spans" has changed much
since this thread...
http://search.lucidimagination.com/search/document/ee23395e5a93c525/non_
overlapping_span_queries#868b3a3ec6431afc
the crux of hte issue (as i recall) is that there is really no
conecptual reason to why a query for "'john' near 'john', in any order,
with slop of Z" shouldn't match a doc that contains only one instance of
"john" ... the first SpanTermQuery says "i found a match at position X"
the second SpanTermQuery says "i found a match at position Y" and the
SpanNearQuery says "the differnece between X and Y is less then Z"
therefore i have a match. (The SpanNearQuery can't fail just because X
and Y are the same -- they might be two distinct term instances, with
differnet payloads perhaps, that just happen to have the same position).
However: if true==inOrder case works because the SpanNearQuery enforces
that "X must be less then Y" so the same term can't ever match twice.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Please consider the environment before printing this email.
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: SpanNearQuery - inOrder parameter
Posted by Chris Hostetter <ho...@fucit.org>.
: I attach a junit test which shows strange behaviour of the inOrder
: parameter on the SpanNearQuery constructor, using Lucene 2.9.4.
:
: My understanding of this parameter is that true forces the order and
: false doesn't care about the order.
:
: Using true always works. However using false works fine when the terms
: in the query are distinct, but if they are equivalent, e.g. searching
: for "john john", I do not get the expected results. The workaround seems
: to be to always use true for queries with repeated terms.
I don't think the situation of "overlapping spans" has changed much since
this thread...
http://search.lucidimagination.com/search/document/ee23395e5a93c525/non_overlapping_span_queries#868b3a3ec6431afc
the crux of hte issue (as i recall) is that there is really no conecptual
reason to why a query for "'john' near 'john', in any order, with slop of
Z" shouldn't match a doc that contains only one instance of "john" ... the
first SpanTermQuery says "i found a match at position X" the second
SpanTermQuery says "i found a match at position Y" and the SpanNearQuery
says "the differnece between X and Y is less then Z" therefore i have a
match. (The SpanNearQuery can't fail just because X and Y are the same --
they might be two distinct term instances, with differnet payloads
perhaps, that just happen to have the same position).
However: if true==inOrder case works because the SpanNearQuery enforces
that "X must be less then Y" so the same term can't ever match twice.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: SpanNearQuery - inOrder parameter
Posted by Gregory Tarr <Gr...@detica.com>.
Attachment didn't work - test below:
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.TopDocsCollector;
import org.apache.lucene.search.TopScoreDocCollector;
import org.apache.lucene.search.spans.SpanNearQuery;
import org.apache.lucene.search.spans.SpanQuery;
import org.apache.lucene.search.spans.SpanTermQuery;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.Version;
import org.junit.Assert;
import org.junit.Test;
public class TestSpanNearQueryInOrder {
@Test
public void testSpanNearQueryInOrder() {
RAMDirectory directory = new RAMDirectory();
IndexWriter writer = new IndexWriter(directory, new
StandardAnalyzer(Version.LUCENE_29), true,
IndexWriter.MaxFieldLength.UNLIMITED);
TopDocsCollector collector = TopScoreDocCollector.create(3, false);
Document doc = new Document();
// DOC1
doc.add(new Field("text","dddd aaaa bbbb cccc", Field.Store.YES,
Field.Index.ANALYZED));
writer.addDocument(doc);
doc = new Document();
// DOC2
doc.add(new Field("text","dddd aaaa aaaa cccc"));
writer.addDocument(doc);
doc = new Document();
// DOC3
doc.add(new Field("text","dddd aaaa yyyy aaaa xxxx cccc"));
writer.addDocument(doc);
writer.optimize();
writer.close();
searcher = new IndexSearcher(directory, false);
SpanQuery[] clauses = new SpanQuery[2];
clauses[0] = new SpanTermQuery(new Term("text", "aaaa"));
clauses[1] = new SpanTermQuery(new Term("text", "aaaa"));
// Don't care about order, so setting inOrder = false
SpanNearQuery q = new SpanNearQuery(clauses, 1, false);
searcher.search(q, collector);
// This assert fails - 3 docs are returned. Expecting only DOC2 and
DOC3
Assert.assertEquals("Check 2 results", 2, collector.getTotalHits());
collector = new TopScoreDocCollector.create(3, false);
clauses = new SpanQuery[2];
clauses[0] = new SpanTermQuery(new Term("text", "aaaa"));
clauses[1] = new SpanTermQuery(new Term("text", "aaaa"));
// Don't care about order, so setting inOrder = false
q = new SpanNearQuery(clauses, 0, false);
searcher.search(q, collector);
// This assert fails - 3 docs are returned. Expecting only DOC2
Assert.assertEquals("Check 1 result", 1, collector.getTotalHits());
}
}
________________________________
From: Gregory Tarr [mailto:Gregory.tarr@detica.com]
Sent: 09 May 2011 12:29
To: java-user@lucene.apache.org
Subject: SpanNearQuery - inOrder parameter
I attach a junit test which shows strange behaviour of the inOrder
parameter on the SpanNearQuery constructor, using Lucene 2.9.4.
My understanding of this parameter is that true forces the order and
false doesn't care about the order.
Using true always works. However using false works fine when the terms
in the query are distinct, but if they are equivalent, e.g. searching
for "john john", I do not get the expected results. The workaround seems
to be to always use true for queries with repeated terms.
Any help?
Thanks
Greg
<<TestSpanNearQueryInOrder.java>>
Please consider the environment before printing this email.
This message should be regarded as confidential. If you have received
this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard
copy by an authorised signatory. The contents of this email may relate
to dealings with other companies within the Detica Limited group of
companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP,
England.
Please consider the environment before printing this email.
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Limited group of companies.
Detica Limited is registered in England under No: 1337451.
Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.