You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Greg K <gr...@gmail.com> on 2005/11/16 21:19:33 UTC
Filtering on a SpanQuery without losing spans
Hi All,
I'm using a bunch of SpanNearQueries combined in a SpanOrQuery to do a set
of searches matches a phrase with a prefix search at the end. I.e.
"phrase with prefix s*" kind of thing that matches "phrase with prefix
search" and "phrase with prefix string".
It works great and lets me quickly produce excerpts with the spans that I
get via getSpans.
Now, however, I'd like to be able restrict the search to certain documents
in the index, so I don't have to stream through a couple of thousand spans
to produce the 10 excerpts on a subset of the documents.
I've tried added a term to the SpanNearQueries that targets a keyword field
containing the document ids, but SNQ (understandably) can't handle that.
There isn't a SpanBooleanQuery and if I wrap the SpanOrQuery in a
BooleanQuery I'll lose access to the getSpans() method - same if I use a
Filter - can't get spans out of a Hits object, right?
Am I missing something? Is there a straightforward way to do what I'm trying
to do that I'm missing?
Thanks,
Greg
Re: Filtering on a SpanQuery without losing spans
Posted by Paul Elschot <pa...@xs4all.nl>.
On Thursday 17 November 2005 00:33, Greg K wrote:
> Yow! - that's very helpful. Thanks much!
>
> Question about an apparent assumption in your example - is it true that
> spans always stream in increasing document order? Is that a guarantee or
> just a fact about the currently implementation?
Spans are always processed in increasing document order, as
are TermQueries.
The result of a BooleanQuery in 1.4.3 may not be that order, but
in the trunk it is.
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Filtering on a SpanQuery without losing spans
Posted by Greg K <gr...@gmail.com>.
Yow! - that's very helpful. Thanks much!
Question about an apparent assumption in your example - is it true that
spans always stream in increasing document order? Is that a guarantee or
just a fact about the currently implementation?
Greg
On 11/16/05, Doug Cutting <cu...@apache.org> wrote:
>
> Greg K wrote:
> > Now, however, I'd like to be able restrict the search to certain
> documents
> > in the index, so I don't have to stream through a couple of thousand
> spans
> > to produce the 10 excerpts on a subset of the documents.
> >
> > I've tried added a term to the SpanNearQueries that targets a keyword
> field
> > containing the document ids, but SNQ (understandably) can't handle that.
> > There isn't a SpanBooleanQuery and if I wrap the SpanOrQuery in a
> > BooleanQuery I'll lose access to the getSpans() method - same if I use a
> > Filter - can't get spans out of a Hits object, right?
> >
> > Am I missing something? Is there a straightforward way to do what I'm
> trying
> > to do that I'm missing?
>
> I think what you need is a SpanAndQuery, with a constructor like:
>
> SpanAndQuery(SpanQuery spanQ, Query docsQ);
>
> This would return the subset of spans matching spanQ whose documents
> match docQ. This should not be too hard to implement. Its Spans
> implementation might look something like:
>
> private Spans spans;
> private Scorer docs;
> private boolean more;
>
> public SpanAndSpans(SpanQuery spanQ, Query docQ,
> IndexReader reader, Searcher searcher) {
> spans = spanQ.getSpans(reader);
> docs = docQ.weight(searcher).scorer(reader);
> more = docs.next();
> }
>
> public boolean next() throws IOException {
> if (!more) return false;
> more = spans.next();
> while (more) {
> if (spans.doc() > docs.doc()) {
> more = docs.skipTo(spans.doc());
> } else if (spans.doc() < docs.doc()) {
> more = spans.skipTo(docs.doc ());
> } else if (spans.doc() == docs.doc()) {
> return true;
> }
> }
> return false;
> }
>
> public boolean skipTo(int doc) throws IOException {
> if (!more) return false;
> more = docs.skipTo(doc);
> return next();
> }
>
> This is untested and incomplete, but does it look like the right
> direction?
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Filtering on a SpanQuery without losing spans
Posted by Doug Cutting <cu...@apache.org>.
Greg K wrote:
> Now, however, I'd like to be able restrict the search to certain documents
> in the index, so I don't have to stream through a couple of thousand spans
> to produce the 10 excerpts on a subset of the documents.
>
> I've tried added a term to the SpanNearQueries that targets a keyword field
> containing the document ids, but SNQ (understandably) can't handle that.
> There isn't a SpanBooleanQuery and if I wrap the SpanOrQuery in a
> BooleanQuery I'll lose access to the getSpans() method - same if I use a
> Filter - can't get spans out of a Hits object, right?
>
> Am I missing something? Is there a straightforward way to do what I'm trying
> to do that I'm missing?
I think what you need is a SpanAndQuery, with a constructor like:
SpanAndQuery(SpanQuery spanQ, Query docsQ);
This would return the subset of spans matching spanQ whose documents
match docQ. This should not be too hard to implement. Its Spans
implementation might look something like:
private Spans spans;
private Scorer docs;
private boolean more;
public SpanAndSpans(SpanQuery spanQ, Query docQ,
IndexReader reader, Searcher searcher) {
spans = spanQ.getSpans(reader);
docs = docQ.weight(searcher).scorer(reader);
more = docs.next();
}
public boolean next() throws IOException {
if (!more) return false;
more = spans.next();
while (more) {
if (spans.doc() > docs.doc()) {
more = docs.skipTo(spans.doc());
} else if (spans.doc() < docs.doc()) {
more = spans.skipTo(docs.doc());
} else if (spans.doc() == docs.doc()) {
return true;
}
}
return false;
}
public boolean skipTo(int doc) throws IOException {
if (!more) return false;
more = docs.skipTo(doc);
return next();
}
This is untested and incomplete, but does it look like the right direction?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Filtering on a SpanQuery without losing spans
Posted by Paul Elschot <pa...@xs4all.nl>.
Greg,
On Wednesday 16 November 2005 21:19, Greg K wrote:
> Hi All,
>
> I'm using a bunch of SpanNearQueries combined in a SpanOrQuery to do a set
> of searches matches a phrase with a prefix search at the end. I.e.
>
> "phrase with prefix s*" kind of thing that matches "phrase with prefix
> search" and "phrase with prefix string".
>
> It works great and lets me quickly produce excerpts with the spans that I
> get via getSpans.
>
> Now, however, I'd like to be able restrict the search to certain documents
> in the index, so I don't have to stream through a couple of thousand spans
> to produce the 10 excerpts on a subset of the documents.
>
> I've tried added a term to the SpanNearQueries that targets a keyword field
> containing the document ids, but SNQ (understandably) can't handle that.
> There isn't a SpanBooleanQuery and if I wrap the SpanOrQuery in a
> BooleanQuery I'll lose access to the getSpans() method - same if I use a
> Filter - can't get spans out of a Hits object, right?
>
...
Once you know the document numbers that you want the getSpans() to work
on, you can make a Filter for those doc nrs (set the corresponding bits
in the BitSet for the Filter) and use this FilteredQuery:
http://issues.apache.org/jira/browse/LUCENE-330
to wrap your SpanQuery.
This FilteredQuery requires skipTo(), and the scorer for
SNQ implements that, so it should work.
The FilteredQuery in 1.4.3 will score all documents regardless of whether
they are filtered out or not.
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org