out of memory when trying to sort by id in a 1.5 billion index

[adfel70 <ad...@gmail.com>]

hi
I have 11 machines in my cluster.
each machine 128GB memory, 2 solr jvm's with 12gb heap each.
cluster has 7 shard, 3 replicas.
1.5 billion docs total.
most user queries are pretty simple for now, sorting by date fields and
another field the has around 1000 unique values.

I have a usecase for using cursorpage and when tried to check this, I got
outOfMemory just for sorting by id.
I read in old posts that I should add heap memory, and I can do that, but I
would rather not .
All other usecases I have are using stable 8gb heap .
Any other way to handle this in solr 4.8?



--
View this message in context: http://lucene.472066.n3.nabble.com/out-of-memory-when-trying-to-sort-by-id-in-a-1-5-billion-index-tp4168156.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: out of memory when trying to sort by id in a 1.5 billion index

[Chris Hostetter <ho...@fucit.org>]

: For sorting DocValues are the best option I think.

yep, definitely a good idea.

: > I have a usecase for using cursorpage and when tried to check this, I got
: > outOfMemory just for sorting by id.

what does the field/fieldType for your uniqueKey field look like?

If you aren't using DocValues, then the amount of RAM needed is going to 
vary widely depending on the datatype used by the FieldCache.

-Hoss
http://www.lucidworks.com/

Re: out of memory when trying to sort by id in a 1.5 billion index

[Yago Riveiro <ya...@gmail.com>]

For sorting DocValues are the best option I think.

—
/Yago Riveiro

On Fri, Nov 7, 2014 at 12:45 PM, adfel70 <ad...@gmail.com> wrote:

> hi
> I have 11 machines in my cluster.
> each machine 128GB memory, 2 solr jvm's with 12gb heap each.
> cluster has 7 shard, 3 replicas.
> 1.5 billion docs total.
> most user queries are pretty simple for now, sorting by date fields and
> another field the has around 1000 unique values.
> I have a usecase for using cursorpage and when tried to check this, I got
> outOfMemory just for sorting by id.
> I read in old posts that I should add heap memory, and I can do that, but I
> would rather not .
> All other usecases I have are using stable 8gb heap .
> Any other way to handle this in solr 4.8?
> --
> View this message in context: http://lucene.472066.n3.nabble.com/out-of-memory-when-trying-to-sort-by-id-in-a-1-5-billion-index-tp4168156.html
> Sent from the Solr - User mailing list archive at Nabble.com.