You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Preetam Rao <bl...@gmail.com> on 2008/07/09 09:00:31 UTC
estimating memory needed for solr instances...
Hi,
Since we plan to share the same box among multiple solr instances on a 16gb
RAM multi core box, Need to estimate how much memory we need for our
application.
The index size is on disk 2.4G with close to 3 million documents. The plan
is to use dismax query with some fqs.
Since we do not sort the results, the sort will be by score which eliminates
the option "userFiterFprSortedQuerries".
Thus assuming all q's will use query result cache and all fqs will use
filter caches the below is what i am thinking.
I would like to know how to relate the index size on disk to its memory size
?
Would it be safe to assume gven the disk size of 2.4g, that we can have ram
size for whole index plus 1g for any other overhead plus the cache size
which comes to 150MB (calculation below). Thus making it around 4g.
cache size calculation -
--------------------------------
query result cache - size = 50K;
since we paginate the results and each page has 10 items and assuming each
user will at the max see 3 pages, per query
we will set queryResultWindowSize to 30. Assuming this, for 50k querries we
will use up 50000* 30 bits = 187K asuming results are stored in bitset.
we use few common fqs, lets say 200. Assuming each returns around 30k
documents, it adds to 200 * 30000 bits = 750K.
If we use document cache of size 20K, assuming each document size is around
5k at the max, it will take up 20000 * 5= 100MB.
Thus we can increase the cache more drastically and still it will use up
only 150MB or less.
Is this reasoning on cache's correct ?
Thanks
Preetam
Re: estimating memory needed for solr instances...
Posted by Preetam Rao <bl...@gmail.com>.
Thanks for the responses, Ian, Jacob.
While I could not locate the previous thread, this is what I understand..
While we can fine tune the cache parameters and other stuff which we can
directly control, with respect to index files the key is to give enough RAM
and let the the OS do its best with respect to keeping the index file in
memory,
----------
Preetam
On Thu, Jul 10, 2008 at 7:12 AM, Ian Connor <ia...@gmail.com> wrote:
> I would guess so also to a point. After you run out of RAM, indexing
> also takes a hit. I have noticed on a 2Gb machine when the index gets
> over 2Gb, my indexing rate when down from 100/s to 40/s. After
> reaching 4Gb it was down to 10/s. I am trying now with a 8Gb machine
> to see how far I get through my data before slowing down.
>
> On Wed, Jul 9, 2008 at 7:56 PM, Jacob Singh <ja...@gmail.com> wrote:
> > My total guess is that indexing is CPU bound, and searching is RAM bound.
> >
> > Best,
> > Jacob
> > Ian Connor wrote:
> >> There was a thread a while ago, that suggested just need to factor in
> >> the index's total size (Mike Klaas I think was the author). It was
> >> suggested having the RAM is enough and the OS will cache the files as
> >> needed to give you the performance boost needed.
> >>
> >> If I misread the thread, please chime in - but it seems having enough
> >> RAM is the key to performance.
> >>
> >> On Wed, Jul 9, 2008 at 3:00 AM, Preetam Rao <bl...@gmail.com>
> wrote:
> >>> Hi,
> >>>
> >>> Since we plan to share the same box among multiple solr instances on a
> 16gb
> >>> RAM multi core box, Need to estimate how much memory we need for our
> >>> application.
> >>>
> >>> The index size is on disk 2.4G with close to 3 million documents. The
> plan
> >>> is to use dismax query with some fqs.
> >>> Since we do not sort the results, the sort will be by score which
> eliminates
> >>> the option "userFiterFprSortedQuerries".
> >>> Thus assuming all q's will use query result cache and all fqs will use
> >>> filter caches the below is what i am thinking.
> >>>
> >>> I would like to know how to relate the index size on disk to its memory
> size
> >>> ?
> >>> Would it be safe to assume gven the disk size of 2.4g, that we can have
> ram
> >>> size for whole index plus 1g for any other overhead plus the cache size
> >>> which comes to 150MB (calculation below). Thus making it around 4g.
> >>>
> >>> cache size calculation -
> >>> --------------------------------
> >>> query result cache - size = 50K;
> >>> since we paginate the results and each page has 10 items and assuming
> each
> >>> user will at the max see 3 pages, per query
> >>> we will set queryResultWindowSize to 30. Assuming this, for 50k
> querries we
> >>> will use up 50000* 30 bits = 187K asuming results are stored in bitset.
> >>>
> >>> we use few common fqs, lets say 200. Assuming each returns around 30k
> >>> documents, it adds to 200 * 30000 bits = 750K.
> >>>
> >>> If we use document cache of size 20K, assuming each document size is
> around
> >>> 5k at the max, it will take up 20000 * 5= 100MB.
> >>>
> >>> Thus we can increase the cache more drastically and still it will use
> up
> >>> only 150MB or less.
> >>>
> >>> Is this reasoning on cache's correct ?
> >>>
> >>> Thanks
> >>> Preetam
> >>>
> >>
> >>
> >>
> >
> >
>
Re: estimating memory needed for solr instances...
Posted by Ian Connor <ia...@gmail.com>.
I would guess so also to a point. After you run out of RAM, indexing
also takes a hit. I have noticed on a 2Gb machine when the index gets
over 2Gb, my indexing rate when down from 100/s to 40/s. After
reaching 4Gb it was down to 10/s. I am trying now with a 8Gb machine
to see how far I get through my data before slowing down.
On Wed, Jul 9, 2008 at 7:56 PM, Jacob Singh <ja...@gmail.com> wrote:
> My total guess is that indexing is CPU bound, and searching is RAM bound.
>
> Best,
> Jacob
> Ian Connor wrote:
>> There was a thread a while ago, that suggested just need to factor in
>> the index's total size (Mike Klaas I think was the author). It was
>> suggested having the RAM is enough and the OS will cache the files as
>> needed to give you the performance boost needed.
>>
>> If I misread the thread, please chime in - but it seems having enough
>> RAM is the key to performance.
>>
>> On Wed, Jul 9, 2008 at 3:00 AM, Preetam Rao <bl...@gmail.com> wrote:
>>> Hi,
>>>
>>> Since we plan to share the same box among multiple solr instances on a 16gb
>>> RAM multi core box, Need to estimate how much memory we need for our
>>> application.
>>>
>>> The index size is on disk 2.4G with close to 3 million documents. The plan
>>> is to use dismax query with some fqs.
>>> Since we do not sort the results, the sort will be by score which eliminates
>>> the option "userFiterFprSortedQuerries".
>>> Thus assuming all q's will use query result cache and all fqs will use
>>> filter caches the below is what i am thinking.
>>>
>>> I would like to know how to relate the index size on disk to its memory size
>>> ?
>>> Would it be safe to assume gven the disk size of 2.4g, that we can have ram
>>> size for whole index plus 1g for any other overhead plus the cache size
>>> which comes to 150MB (calculation below). Thus making it around 4g.
>>>
>>> cache size calculation -
>>> --------------------------------
>>> query result cache - size = 50K;
>>> since we paginate the results and each page has 10 items and assuming each
>>> user will at the max see 3 pages, per query
>>> we will set queryResultWindowSize to 30. Assuming this, for 50k querries we
>>> will use up 50000* 30 bits = 187K asuming results are stored in bitset.
>>>
>>> we use few common fqs, lets say 200. Assuming each returns around 30k
>>> documents, it adds to 200 * 30000 bits = 750K.
>>>
>>> If we use document cache of size 20K, assuming each document size is around
>>> 5k at the max, it will take up 20000 * 5= 100MB.
>>>
>>> Thus we can increase the cache more drastically and still it will use up
>>> only 150MB or less.
>>>
>>> Is this reasoning on cache's correct ?
>>>
>>> Thanks
>>> Preetam
>>>
>>
>>
>>
>
>
Re: estimating memory needed for solr instances...
Posted by Jacob Singh <ja...@gmail.com>.
My total guess is that indexing is CPU bound, and searching is RAM bound.
Best,
Jacob
Ian Connor wrote:
> There was a thread a while ago, that suggested just need to factor in
> the index's total size (Mike Klaas I think was the author). It was
> suggested having the RAM is enough and the OS will cache the files as
> needed to give you the performance boost needed.
>
> If I misread the thread, please chime in - but it seems having enough
> RAM is the key to performance.
>
> On Wed, Jul 9, 2008 at 3:00 AM, Preetam Rao <bl...@gmail.com> wrote:
>> Hi,
>>
>> Since we plan to share the same box among multiple solr instances on a 16gb
>> RAM multi core box, Need to estimate how much memory we need for our
>> application.
>>
>> The index size is on disk 2.4G with close to 3 million documents. The plan
>> is to use dismax query with some fqs.
>> Since we do not sort the results, the sort will be by score which eliminates
>> the option "userFiterFprSortedQuerries".
>> Thus assuming all q's will use query result cache and all fqs will use
>> filter caches the below is what i am thinking.
>>
>> I would like to know how to relate the index size on disk to its memory size
>> ?
>> Would it be safe to assume gven the disk size of 2.4g, that we can have ram
>> size for whole index plus 1g for any other overhead plus the cache size
>> which comes to 150MB (calculation below). Thus making it around 4g.
>>
>> cache size calculation -
>> --------------------------------
>> query result cache - size = 50K;
>> since we paginate the results and each page has 10 items and assuming each
>> user will at the max see 3 pages, per query
>> we will set queryResultWindowSize to 30. Assuming this, for 50k querries we
>> will use up 50000* 30 bits = 187K asuming results are stored in bitset.
>>
>> we use few common fqs, lets say 200. Assuming each returns around 30k
>> documents, it adds to 200 * 30000 bits = 750K.
>>
>> If we use document cache of size 20K, assuming each document size is around
>> 5k at the max, it will take up 20000 * 5= 100MB.
>>
>> Thus we can increase the cache more drastically and still it will use up
>> only 150MB or less.
>>
>> Is this reasoning on cache's correct ?
>>
>> Thanks
>> Preetam
>>
>
>
>
Re: estimating memory needed for solr instances...
Posted by Ian Connor <ia...@gmail.com>.
There was a thread a while ago, that suggested just need to factor in
the index's total size (Mike Klaas I think was the author). It was
suggested having the RAM is enough and the OS will cache the files as
needed to give you the performance boost needed.
If I misread the thread, please chime in - but it seems having enough
RAM is the key to performance.
On Wed, Jul 9, 2008 at 3:00 AM, Preetam Rao <bl...@gmail.com> wrote:
> Hi,
>
> Since we plan to share the same box among multiple solr instances on a 16gb
> RAM multi core box, Need to estimate how much memory we need for our
> application.
>
> The index size is on disk 2.4G with close to 3 million documents. The plan
> is to use dismax query with some fqs.
> Since we do not sort the results, the sort will be by score which eliminates
> the option "userFiterFprSortedQuerries".
> Thus assuming all q's will use query result cache and all fqs will use
> filter caches the below is what i am thinking.
>
> I would like to know how to relate the index size on disk to its memory size
> ?
> Would it be safe to assume gven the disk size of 2.4g, that we can have ram
> size for whole index plus 1g for any other overhead plus the cache size
> which comes to 150MB (calculation below). Thus making it around 4g.
>
> cache size calculation -
> --------------------------------
> query result cache - size = 50K;
> since we paginate the results and each page has 10 items and assuming each
> user will at the max see 3 pages, per query
> we will set queryResultWindowSize to 30. Assuming this, for 50k querries we
> will use up 50000* 30 bits = 187K asuming results are stored in bitset.
>
> we use few common fqs, lets say 200. Assuming each returns around 30k
> documents, it adds to 200 * 30000 bits = 750K.
>
> If we use document cache of size 20K, assuming each document size is around
> 5k at the max, it will take up 20000 * 5= 100MB.
>
> Thus we can increase the cache more drastically and still it will use up
> only 150MB or less.
>
> Is this reasoning on cache's correct ?
>
> Thanks
> Preetam
>
--
Regards,
Ian Connor
82 Fellsway W #2
Somerville, MA 02145
Direct Line: +1 (978) 6333372
Call Center Phone: +1 (714) 239 3875 (24 hrs)
Mobile Phone: +1 (312) 218 3209
Fax: +1(770) 818 5697
Suisse Phone: +41 (0) 22 548 1664
Skype: ian.connor
Re: estimating memory needed for solr instances...
Posted by Preetam Rao <bl...@gmail.com>.
Oops. Sorry for the typo. I will be careful next time.
Thanks a lot for digging out the old thread :-)
It was helpful.
Should we remove the option "useFilterForSortedQuery" altogether if its not
being used anymore ?
-------
Preetam
On Fri, Jul 11, 2008 at 2:10 AM, Chris Harris <ry...@gmail.com> wrote:
> I didn't know what option was being referred to here, but I eventually
> figured it out. If anyone else was confused, the option is called
> useFilterForSortedQuery, you can set it via solrconfig.xml, and, at
> least according to Yonik in late 2006, you probably won't want to
> enable it even if you *do* sort by something other than score:
>
>
> http://www.nabble.com/try-setting-useFilterForSortedQuery-to-false-td7822871.html#a7822871
>
> Cheers,
> Chris
>
> On Wed, Jul 9, 2008 at 12:00 AM, Preetam Rao <bl...@gmail.com>
> wrote:
> >
> > Since we do not sort the results, the sort will be by score which
> eliminates
> > the option "userFiterFprSortedQuerries".
>
Re: estimating memory needed for solr instances...
Posted by Chris Harris <ry...@gmail.com>.
I didn't know what option was being referred to here, but I eventually
figured it out. If anyone else was confused, the option is called
useFilterForSortedQuery, you can set it via solrconfig.xml, and, at
least according to Yonik in late 2006, you probably won't want to
enable it even if you *do* sort by something other than score:
http://www.nabble.com/try-setting-useFilterForSortedQuery-to-false-td7822871.html#a7822871
Cheers,
Chris
On Wed, Jul 9, 2008 at 12:00 AM, Preetam Rao <bl...@gmail.com> wrote:
>
> Since we do not sort the results, the sort will be by score which eliminates
> the option "userFiterFprSortedQuerries".