parameters to decide solr memory consumption

[Sachin Aggarwal <di...@gmail.com>]

hello,

need help regarding how  solr stores the indexes i was reading a article
that says solr also stores the indexes in same format as explained in
appendix B of  lucene in action   is it true

and what parameters do i need to focus on while estimating the memory used
by my use case

as i have table like (userid, username, usertime, userlocation, userphn,
timestamp, address)
what i believe in my case cardinality of some fields like gender and
location userphnmodel will be very less will that influence

any links to read further will b appreciated.

-- 

Thanks & Regards

Sachin Aggarwal
7760502772

Re: parameters to decide solr memory consumption

[Erick Erickson <er...@gmail.com>]

No, that's 255 bytes/record. Also, any time you store a field, the
raw data is preserved in the *.fdt and *.fdx files. If you're thinking
about RAM requirements, you must subtract the amount of data
in those files from the total, as a start. This might help:

http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/fileformats.html

Best
Erick

On Thu, Jun 21, 2012 at 1:48 AM, Sachin Aggarwal
<di...@gmail.com> wrote:
> thanks for help
>
>
> hey
> I tried some exercise
> I m storing schema (uuid,key, userlocation)
> uuid and key are unique and user location have cardinality as 150
> uuid and key are stored and indexed while userlocation is indexed not
> stored.
> still the index directory size is 51 MB just for 200,000 records don't u
> think its not optimal....
> what if i go for billions of records.
>
> --
>
> Thanks & Regards
>
> Sachin Aggarwal
> 7760502772

Re: parameters to decide solr memory consumption

[Sachin Aggarwal <di...@gmail.com>]

thanks for help


hey
I tried some exercise
I m storing schema (uuid,key, userlocation)
uuid and key are unique and user location have cardinality as 150
uuid and key are stored and indexed while userlocation is indexed not
stored.
still the index directory size is 51 MB just for 200,000 records don't u
think its not optimal....
what if i go for billions of records.

-- 

Thanks & Regards

Sachin Aggarwal
7760502772

Re: parameters to decide solr memory consumption

[Erick Erickson <er...@gmail.com>]

This is really difficult to answer because there are so many variables;
the number of unique terms, whether you store fields or not (which is
really unrelated to memory consumption during searching), etc, etc,
etc. So even trying the index and just looking at the index directory
won't tell you much about memory consumption.

And memory use has been dramatically improved in the 4.x code line, so
anything we can say is actually wrong.

Not to mention that your particular use of caches (filterCache, queryResultCache
etc) will change during runtime.

I'm afraid you'll just have to try it and see.

Yes, LIA is accurate...

Best
Erick

On Tue, Jun 19, 2012 at 8:28 AM, Sachin Aggarwal
<di...@gmail.com> wrote:
> hello,
>
> need help regarding how  solr stores the indexes i was reading a article
> that says solr also stores the indexes in same format as explained in
> appendix B of  lucene in action   is it true
>
> and what parameters do i need to focus on while estimating the memory used
> by my use case
>
> as i have table like (userid, username, usertime, userlocation, userphn,
> timestamp, address)
> what i believe in my case cardinality of some fields like gender and
> location userphnmodel will be very less will that influence
>
> any links to read further will b appreciated.
>
> --
>
> Thanks & Regards
>
> Sachin Aggarwal
> 7760502772