You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mou <mo...@gmail.com> on 2013/02/01 05:34:51 UTC
Re: long QTime for big index
Thank you again.
Unfortunately the index files will not fit in the RAM.I have to try using
document cache. I am also moving my index to SSD again, we took our index
off when fusion IO cards failed twice during indexing and index was
corrupted.Now with the bios upgrade and new driver, it is supposed to be
more reliable.
Also I am going to look into the client app to verify that it is making
proper query requests.
Surprisingly when I used a much lower value than default for
defaultconnectionperhost and maxconnectionperhost in solrmeter , it performs
very well, the same queries return in less than one sec . I am not sure yet,
need to run solrmeter with different heap size , with cache and without
cache etc.
--
View this message in context: http://lucene.472066.n3.nabble.com/long-QTime-for-big-index-tp4037635p4037870.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: long QTime for big index
Posted by Mou <mo...@gmail.com>.
We have two boxes, they are really nice servers, 32 core cpu, 192 G
memory with both RAID arrays and fusion IOs. But each of them running
two instances of Solr, one for indexing and the other for
searching.Search index is on fusion IO card.
Each instance has 11 cores and a small core for making indexing almost realtime.
We have around 300 Million documents and 250G on disk. They are all
metadata . Search queries are very diverse and they do not repeat very
frequently , 40 -60 qps. Before we had two cores each 125 G on disk
and solr was taking long time to get results from those two cores. CPU
use was 90%.
We never had problem with indexing. 50% of all our docs gets updated
every day, so very high indexing rate.
On Thu, Feb 14, 2013 at 4:20 PM, alxsss [via Lucene]
<ml...@n3.nabble.com> wrote:
> Hi,
>
> It is curious to know how many linux boxes do you have and how many cores in
> each of them. It was my understanding that solr puts in the memory all
> documents found for a keyword, not the whole index. So, why it must be
> faster with more cores, when number of selected documents from many separate
> cores are the same as from one core?
>
> Thanks.
> Alex.
>
>
>
>
>
>
>
> -----Original Message-----
> From: Mou <[hidden email]>
> To: solr-user <[hidden email]>
> Sent: Thu, Feb 14, 2013 2:35 pm
> Subject: Re: long QTime for big index
>
>
> Just to close this discussion , we solved the problem by splitting the
> index.
> It turned out that distributed search with 12 cores are faster than
> searching two cores.
>
> All queries ,tomcat configuration, jvm configuration remain same. Now
> queries are served in milliseconds.
>
>
> On Thu, Jan 31, 2013 at 9:34 PM, Mou [via Lucene]
> <[hidden email]> wrote:
>
>> Thank you again.
>>
>> Unfortunately the index files will not fit in the RAM.I have to try using
>> document cache. I am also moving my index to SSD again, we took our index
>> off when fusion IO cards failed twice during indexing and index was
>> corrupted.Now with the bios upgrade and new driver, it is supposed to be
>> more reliable.
>>
>> Also I am going to look into the client app to verify that it is making
>> proper query requests.
>>
>> Surprisingly when I used a much lower value than default for
>> defaultconnectionperhost and maxconnectionperhost in solrmeter , it
>> performs
>> very well, the same queries return in less than one sec . I am not sure
>> yet,
>> need to run solrmeter with different heap size , with cache and without
>> cache etc.
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://lucene.472066.n3.nabble.com/long-QTime-for-big-index-tp4037635p4037870.html
>> To unsubscribe from long QTime for big index, click here.
>> NAML
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/long-QTime-for-big-index-tp4037635p4040535.html
>
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://lucene.472066.n3.nabble.com/long-QTime-for-big-index-tp4037635p4040545.html
> To unsubscribe from long QTime for big index, click here.
> NAML
--
View this message in context: http://lucene.472066.n3.nabble.com/long-QTime-for-big-index-tp4037635p4040549.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: long QTime for big index
Posted by al...@aim.com.
Hi,
It is curious to know how many linux boxes do you have and how many cores in each of them. It was my understanding that solr puts in the memory all documents found for a keyword, not the whole index. So, why it must be faster with more cores, when number of selected documents from many separate cores are the same as from one core?
Thanks.
Alex.
-----Original Message-----
From: Mou <mo...@gmail.com>
To: solr-user <so...@lucene.apache.org>
Sent: Thu, Feb 14, 2013 2:35 pm
Subject: Re: long QTime for big index
Just to close this discussion , we solved the problem by splitting the index.
It turned out that distributed search with 12 cores are faster than
searching two cores.
All queries ,tomcat configuration, jvm configuration remain same. Now
queries are served in milliseconds.
On Thu, Jan 31, 2013 at 9:34 PM, Mou [via Lucene]
<ml...@n3.nabble.com> wrote:
> Thank you again.
>
> Unfortunately the index files will not fit in the RAM.I have to try using
> document cache. I am also moving my index to SSD again, we took our index
> off when fusion IO cards failed twice during indexing and index was
> corrupted.Now with the bios upgrade and new driver, it is supposed to be
> more reliable.
>
> Also I am going to look into the client app to verify that it is making
> proper query requests.
>
> Surprisingly when I used a much lower value than default for
> defaultconnectionperhost and maxconnectionperhost in solrmeter , it performs
> very well, the same queries return in less than one sec . I am not sure yet,
> need to run solrmeter with different heap size , with cache and without
> cache etc.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://lucene.472066.n3.nabble.com/long-QTime-for-big-index-tp4037635p4037870.html
> To unsubscribe from long QTime for big index, click here.
> NAML
--
View this message in context: http://lucene.472066.n3.nabble.com/long-QTime-for-big-index-tp4037635p4040535.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: long QTime for big index
Posted by Mou <mo...@gmail.com>.
Just to close this discussion , we solved the problem by splitting the index.
It turned out that distributed search with 12 cores are faster than
searching two cores.
All queries ,tomcat configuration, jvm configuration remain same. Now
queries are served in milliseconds.
On Thu, Jan 31, 2013 at 9:34 PM, Mou [via Lucene]
<ml...@n3.nabble.com> wrote:
> Thank you again.
>
> Unfortunately the index files will not fit in the RAM.I have to try using
> document cache. I am also moving my index to SSD again, we took our index
> off when fusion IO cards failed twice during indexing and index was
> corrupted.Now with the bios upgrade and new driver, it is supposed to be
> more reliable.
>
> Also I am going to look into the client app to verify that it is making
> proper query requests.
>
> Surprisingly when I used a much lower value than default for
> defaultconnectionperhost and maxconnectionperhost in solrmeter , it performs
> very well, the same queries return in less than one sec . I am not sure yet,
> need to run solrmeter with different heap size , with cache and without
> cache etc.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://lucene.472066.n3.nabble.com/long-QTime-for-big-index-tp4037635p4037870.html
> To unsubscribe from long QTime for big index, click here.
> NAML
--
View this message in context: http://lucene.472066.n3.nabble.com/long-QTime-for-big-index-tp4037635p4040535.html
Sent from the Solr - User mailing list archive at Nabble.com.