You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Konstantin Shaposhnikov <k8...@tut.by> on 2004/02/25 17:20:14 UTC
Performance of fuzzy query
Hi all.
I am using lucene-1.3-final and have performance problems with fuzzy
queries.
If I understand right to perform fuzzy query lucene enumerate all terms
in the index and construct BooleanQuery which consists of simple
TermQueries.
The main problem is that this process is performing several times during
search significantly decreasing performance.
Let me explain.
For example my search returns 1000 documents. In my application I need
to get all this documents from index for later processing, but lucene
rereads every 100 documents all terms in the index, because by default
we get only 100 documents from index (see Hits class) and when I access
101st document the search process is performed again and absolutely
unnecessary operation of creating FilteredTermEnum is performed.
Unfortunately I can't say to Hits class to get all my 1000 document
initially, because value 100 (actually 50 in the code) is hard coded. So
I think that this value should be configurable in Searcher.
Actually I have performed some research and found that if I get all my
1000 documents at the first stage the speed of my fuzzy query increased
by 4 times!
Also please could you give me some advices on increasing performance of
fuzzy search.
One approach that I already use in our application is custom fuzzy query
that compares word prefix (f.e. the first 3 symbols) of terms and only
if they equals than it tries to compare the rest of terms using the same
algorithm that is used in FuzzyQuery.
Best regards,
Konstantin
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Performance of fuzzy query
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Feb 25, 2004, at 11:20 AM, Konstantin Shaposhnikov wrote:
> For example my search returns 1000 documents. In my application I need
> to get all this documents from index for later processing
This last sentence is key. If you need all documents from a search,
then you should be using a HitCollector instead. Look at the variants
of the search methods and use the HitCollector one to get all documents
in one shot.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Performance of fuzzy query
Posted by Konstantin Shaposhnikov <k8...@tut.by>.
On 10:12 Wed 25 Feb , Doug Cutting wrote:
> Konstantin Shaposhnikov wrote:
> >For example my search returns 1000 documents. In my application I need
> >to get all this documents from index for later processing, but lucene
> >rereads every 100 documents all terms in the index, because by default
> >we get only 100 documents from index (see Hits class) and when I access
> >101st document the search process is performed again and absolutely
> >unnecessary operation of creating FilteredTermEnum is performed.
>
> If you need all of the hits, use HitCollector instead of Hits.
Thank you for reply. I really need to read lucene
documentation once again :)
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Performance of fuzzy query
Posted by Doug Cutting <cu...@apache.org>.
Konstantin Shaposhnikov wrote:
> For example my search returns 1000 documents. In my application I need
> to get all this documents from index for later processing, but lucene
> rereads every 100 documents all terms in the index, because by default
> we get only 100 documents from index (see Hits class) and when I access
> 101st document the search process is performed again and absolutely
> unnecessary operation of creating FilteredTermEnum is performed.
If you need all of the hits, use HitCollector instead of Hits.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org