You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Andrew Cunningham <cu...@csiro.au> on 2005/01/06 02:17:45 UTC
Span Query Performance
Hi all,
I'm currently doing a query similar to the following:
for w in wordset:
query = w near (word1 V word2 V word3 ... V word1422);
perform query
and I am doing this through SpanQuery.getSpans(), iterating through the
spans and counting
the matches, which can result in 4782282 matches (essentially I am only
after the match count).
The query works but the performance can be somewhat slow; so I am wondering:
a) Would the query potentially run faster if I used
Searcher.search(query) with a custom similarity,
or do both methods essentially use the same mechanics
b) Does using a RAMDirectory improve query performance any significant
amount.
c) Is there a faster method to what I am doing I should consider?
Thanks,
Andrew
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Span Query Performance
Posted by Paul Elschot <pa...@xs4all.nl>.
Sorry for the duplicate on lucene-dev, it should have gone to lucene-user
directly:
A bit more:
On Thursday 06 January 2005 10:22, Paul Elschot wrote:
> On Thursday 06 January 2005 02:17, Andrew Cunningham wrote:
> > Hi all,
> >
> > I'm currently doing a query similar to the following:
> >
> > for w in wordset:
> > query = w near (word1 V word2 V word3 ... V word1422);
> > perform query
> >
> > and I am doing this through SpanQuery.getSpans(), iterating through the
> > spans and counting
> > the matches, which can result in 4782282 matches (essentially I am only
> > after the match count).
> > The query works but the performance can be somewhat slow; so I am
wondering:
> >
...
> > c) Is there a faster method to what I am doing I should consider?
>
> Preindexing all word combinations that you're interested in.
>
In case you know all the words in advance, you could also index a
helper word at the same position as each of those words.
This requires a custom analyzer that inserts the helper word in the
token stream with a zero position increment.
The query then simplifies to:
query = w near helperword
which would probably speed things up significantly.
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Span Query Performance
Posted by Paul Elschot <pa...@xs4all.nl>.
A bit more:
On Thursday 06 January 2005 10:22, Paul Elschot wrote:
> On Thursday 06 January 2005 02:17, Andrew Cunningham wrote:
> > Hi all,
> >
> > I'm currently doing a query similar to the following:
> >
> > for w in wordset:
> > query = w near (word1 V word2 V word3 ... V word1422);
> > perform query
> >
> > and I am doing this through SpanQuery.getSpans(), iterating through the
> > spans and counting
> > the matches, which can result in 4782282 matches (essentially I am only
> > after the match count).
> > The query works but the performance can be somewhat slow; so I am
wondering:
> >
...
> > c) Is there a faster method to what I am doing I should consider?
>
> Preindexing all word combinations that you're interested in.
>
In case you know all the words in advance, you could also index a
helper word at the same position as each of those words.
This requires a custom analyzer that inserts the helper word in the
token stream with a zero position increment.
The query then simplifies to:
query = w near helperword
which would probably speed things up significantly.
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Span Query Performance
Posted by Paul Elschot <pa...@xs4all.nl>.
On Thursday 06 January 2005 02:17, Andrew Cunningham wrote:
> Hi all,
>
> I'm currently doing a query similar to the following:
>
> for w in wordset:
> query = w near (word1 V word2 V word3 ... V word1422);
> perform query
>
> and I am doing this through SpanQuery.getSpans(), iterating through the
> spans and counting
> the matches, which can result in 4782282 matches (essentially I am only
> after the match count).
> The query works but the performance can be somewhat slow; so I am wondering:
>
> a) Would the query potentially run faster if I used
> Searcher.search(query) with a custom similarity,
> or do both methods essentially use the same mechanics
It would be somewhat slower, because it loops over the getSpans()
and computes document scores and constructs a Hits from the scores.
> b) Does using a RAMDirectory improve query performance any significant
> amount.
That depends on your operating system, the size of the index, the amount
of RAM you can use, the file buffering efficiency, other loads on the
computer ...
> c) Is there a faster method to what I am doing I should consider?
Preindexing all word combinations that you're interested in.
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org