You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by phiras <ph...@gmail.com> on 2008/02/03 18:24:17 UTC
high lookup time on single lucene index
Hi,
We have a single Lucene index consist of 4.5 million document with 20
million different term.
The lookup time is too high , (more than 3 seconds) for a none-simple
boolean query, this high lookup time is not acceptable.
We did a simple test to figure out the relation between ( documents count &
lookup time ) and the results was amazing.
note : queries and documents are generated randomly
Doc count / lookup time for 1000 multi term query ( in milliseconds )
5000 / 1683
10000 / 2898
15000 / 23667
20000 / 25002
25000 / 31282
30000 / 43849
35000 / 53526
40000 / 69321
45000 / 73319
50000 / 80006
My questions now are :
1- What's the search complexity in Lucene (driven by what ?).
2- How can we enhance search performance ?
Thank you.
--------------------------------------------------
Firas Kassem,
d1g corp.
--
View this message in context: http://www.nabble.com/high-lookup-time-on-single-lucene-index-tp15255763p15255763.html
Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: high lookup time on single lucene index
Posted by phiras <ph...@gmail.com>.
In fact we are following lucene performance hints (
http://wiki.apache.org/lucene-java/BasicsOfPerformance )
Our queries are not that complicated , we are expanding user query, let's
say between 2 to 5 new keyword for each keyword in the user query.
in the test we did , queries are not more than 8 'OR-ed' keyword.
Grant Ingersoll-6 wrote:
>
> Have a look at http://wiki.apache.org/lucene-java/BasicsOfPerformance
>
> Your times sound pretty high, but it depends on the complexity of the
> query, etc. Can you share information about your queries, etc.? Are
> you opening the IndexSearcher on every query?
>
> -Grant
>
> On Feb 3, 2008, at 12:24 PM, phiras wrote:
>
>>
>> Hi,
>> We have a single Lucene index consist of 4.5 million document with 20
>> million different term.
>> The lookup time is too high , (more than 3 seconds) for a none-simple
>> boolean query, this high lookup time is not acceptable.
>>
>> We did a simple test to figure out the relation between ( documents
>> count &
>> lookup time ) and the results was amazing.
>> note : queries and documents are generated randomly
>>
>> Doc count / lookup time for 1000 multi term query ( in
>> milliseconds )
>> 5000 / 1683
>> 10000 / 2898
>> 15000 / 23667
>> 20000 / 25002
>> 25000 / 31282
>> 30000 / 43849
>> 35000 / 53526
>> 40000 / 69321
>> 45000 / 73319
>> 50000 / 80006
>>
>> My questions now are :
>> 1- What's the search complexity in Lucene (driven by what ?).
>> 2- How can we enhance search performance ?
>>
>> Thank you.
>>
>> --------------------------------------------------
>> Firas Kassem,
>> d1g corp.
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/high-lookup-time-on-single-lucene-index-tp15255763p15255763.html
>> Sent from the Lucene - Java Developer mailing list archive at
>> Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
> http://www.lucenebootcamp.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
--
View this message in context: http://www.nabble.com/high-lookup-time-on-single-lucene-index-tp15255763p15256173.html
Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: high lookup time on single lucene index
Posted by Grant Ingersoll <gs...@apache.org>.
Have a look at http://wiki.apache.org/lucene-java/BasicsOfPerformance
Your times sound pretty high, but it depends on the complexity of the
query, etc. Can you share information about your queries, etc.? Are
you opening the IndexSearcher on every query?
-Grant
On Feb 3, 2008, at 12:24 PM, phiras wrote:
>
> Hi,
> We have a single Lucene index consist of 4.5 million document with 20
> million different term.
> The lookup time is too high , (more than 3 seconds) for a none-simple
> boolean query, this high lookup time is not acceptable.
>
> We did a simple test to figure out the relation between ( documents
> count &
> lookup time ) and the results was amazing.
> note : queries and documents are generated randomly
>
> Doc count / lookup time for 1000 multi term query ( in
> milliseconds )
> 5000 / 1683
> 10000 / 2898
> 15000 / 23667
> 20000 / 25002
> 25000 / 31282
> 30000 / 43849
> 35000 / 53526
> 40000 / 69321
> 45000 / 73319
> 50000 / 80006
>
> My questions now are :
> 1- What's the search complexity in Lucene (driven by what ?).
> 2- How can we enhance search performance ?
>
> Thank you.
>
> --------------------------------------------------
> Firas Kassem,
> d1g corp.
>
>
>
> --
> View this message in context: http://www.nabble.com/high-lookup-time-on-single-lucene-index-tp15255763p15255763.html
> Sent from the Lucene - Java Developer mailing list archive at
> Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org