You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Shelly_Singh <Sh...@infosys.com> on 2010/08/27 05:34:50 UTC
Bettering search performance
Hi,
I have a lucene index of 100 million documents. But the document size is very small - 5 fields with 1 or 2 terms each. Only 1 field is analyzed and others are just simply indexed. The index is optimized to 2 segments and the total index size is 7GB.
I open a searcher with a termsInfoDivisor value as '10'. I have warmed my searcher by executing 10 random termQueries (on the field that is analyzed). After this, when I do a search on the same field, I get a response time of over 2 seconds.
I am executing with a RAM of 1.5G. Is it possible to bring the search time down to few milliseconds in this configuration.
Any tips would help.
Thanks and Regards,
Shelly Singh
Center For KNowledge Driven Information Systems, Infosys
Email: shelly_singh@infosys.com
Phone: (M) 91 992 369 7200, (VoIP)2022978622
Re: Bettering search performance
Posted by Erick Erickson <er...@gmail.com>.
I'm assuming that the queries you're timing also do not sort. Warmup queries
that don't sort also don't help speed up the first queries that do sort.
But assuming that sorting isn't the problem, you'll need to do a bit of
performance
monitoring. You haven't mentioned what op system you're using, so suggesting
a particular quick-n-dirty tool is kinda hard....
I suspect though that you're swapping. 1.5G or a 7G index jumps out as the
first place to look...
HTH
Erick
On Fri, Aug 27, 2010 at 2:46 AM, Shelly_Singh <Sh...@infosys.com>wrote:
> The queries that I have recorded performance for matches documents ranging
> from 340 to 3000. I am not processing any documents yet; I mean time of 2
> seconds does not include time for processing /collect scoreDocs...
>
> Thanks and Regards,
>
> Shelly Singh
> Center For KNowledge Driven Information Systems, Infosys
> Email: shelly_singh@infosys.com
> Phone: (M) 91 992 369 7200, (VoIP)2022978622
>
>
> -----Original Message-----
> From: Toke Eskildsen [mailto:te@statsbiblioteket.dk]
> Sent: Friday, August 27, 2010 2:27 PM
> To: java-user@lucene.apache.org
> Subject: Re: Bettering search performance
>
> On Fri, 2010-08-27 at 05:34 +0200, Shelly_Singh wrote:
> > I have a lucene index of 100 million documents. [...] total index size
> is 7GB.
>
> [...]
>
> > I get a response time of over 2 seconds.
>
> How many documents match such a query and how many of those documents do
> you process (i.e. extract a term for)?
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Bettering search performance
Posted by Shelly_Singh <Sh...@infosys.com>.
The queries that I have recorded performance for matches documents ranging from 340 to 3000. I am not processing any documents yet; I mean time of 2 seconds does not include time for processing /collect scoreDocs...
Thanks and Regards,
Shelly Singh
Center For KNowledge Driven Information Systems, Infosys
Email: shelly_singh@infosys.com
Phone: (M) 91 992 369 7200, (VoIP)2022978622
-----Original Message-----
From: Toke Eskildsen [mailto:te@statsbiblioteket.dk]
Sent: Friday, August 27, 2010 2:27 PM
To: java-user@lucene.apache.org
Subject: Re: Bettering search performance
On Fri, 2010-08-27 at 05:34 +0200, Shelly_Singh wrote:
> I have a lucene index of 100 million documents. [...] total index size is 7GB.
[...]
> I get a response time of over 2 seconds.
How many documents match such a query and how many of those documents do
you process (i.e. extract a term for)?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Bettering search performance
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Fri, 2010-08-27 at 05:34 +0200, Shelly_Singh wrote:
> I have a lucene index of 100 million documents. [...] total index size is 7GB.
[...]
> I get a response time of over 2 seconds.
How many documents match such a query and how many of those documents do
you process (i.e. extract a term for)?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org