You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Per Steffensen <st...@designware.dk> on 2013/03/21 09:13:47 UTC
Sort-field for ALL docs in FieldCache for sort queries -> OOM on
lots of docs
Hi
We have a lot of docs in Solr. Each particular Solr-node handles a lot
of docs distributed among several replica. When you issue a sort query,
it seems to me that, the value of the sort-field of ALL docs under the
Solr-node is added to the FieldCache. This leads to OOM-exceptions at
some point when you have enough docs under the Solr-node - relative to
its Xmx of course. Are there any "tricks" to get around this issue, so
that a sort-query will never trigger a OOM, no matter how many docs are
handled by a particular Solr-node. Of course you need to be ready to
accept the penalty of more disk-IO as soon as the entire thing does not
fit in memory, but I would rather accept that than accept OOM's.
Regards, Per Steffensen
Re: Sort-field for ALL docs in FieldCache for sort queries -> OOM
on lots of docs
Posted by Shawn Heisey <so...@elyograg.org>.
On 3/22/2013 8:54 AM, Per Steffensen wrote:
> Me too. I will find out soon - I hope! But re-indexing is kinda a
> problem for us, but we will figure out.
> Any "guide to re-index all you stuff" anywhere, so I do it the easiest
> way? Guess maybe there are some nice tricks about steaming data directly
> from one Solr running the old index into a new Solr running the new
> index, and then discard the old index afterwards?
There is no guide to reindexing, because there are so many ways to
index. The basic procedure is to repeat whatever you did the first
time, possibly deleting the entire index first. Because Lucene and Solr
indexes often require changes to deal with changing requirements, the
full index procedure should be automated and repeatable.
The dataimport handler has a SolrEntityProcessor that can index from
another Solr instance. All fields must be stored for this to work,
because it just retrieves documents and ignores the search index. Many
people (including myself) do not store all fields, in an attempt to keep
the index size down.
Thanks,
Shawn
Re: Sort-field for ALL docs in FieldCache for sort queries -> OOM
on lots of docs
Posted by Per Steffensen <st...@designware.dk>.
On 3/21/13 10:50 PM, Shawn Heisey wrote:
> On 3/21/2013 4:05 AM, Per Steffensen wrote:
>> Can anyone else elaborate? How to "activate" it? How to make sure, for
>> sorting, that sort-field-value for all docs are not read into memory for
>> sorting - leading to OOM when you have a lot of docs? Can this feature
>> be activated on top of an existing 4.0 index, or do you have to re-index
>> everything?
>
> There is one requirement that may not be obvious - every document must
> have a value in the field, so you must either make the field either
> required or give it a default value in the schema. Solr 4.2 will
> refuse to start the core if this requirement is not met.
That is not problem for us. The field exist on every document.
> The example schema hints that the value might need to be
> single-valued. I have not tested this. Sorting is already
> problematic on multi-valued fields, so I assume that this won't be the
> case for you.
That is not a problem for us either. The field is single-valued.
>
> To use docValues, add docValues="true" and then either set
> required="true" or default="<somevalue>" on the field definition in
> schema.xml, restart Solr or reload the core, and reindex. Your index
> will get bigger.
So the answer to "...or do you have to re-index everything?" is yes!?
>
> If the touted behavior of handling the sort mechanism in OS disk cache
> memory (or just reading the disk if there's not enough memory) rather
> than heap is correct, then it should solve your issues. I hope it does!
Me too. I will find out soon - I hope! But re-indexing is kinda a
problem for us, but we will figure out.
Any "guide to re-index all you stuff" anywhere, so I do it the easiest
way? Guess maybe there are some nice tricks about steaming data directly
from one Solr running the old index into a new Solr running the new
index, and then discard the old index afterwards?
>
> Thanks,
> Shawn
>
>
Thanks a lot, Shawn!
Regards, Per Steffensen
Re: Sort-field for ALL docs in FieldCache for sort queries -> OOM
on lots of docs
Posted by Shawn Heisey <so...@elyograg.org>.
On 3/21/2013 4:05 AM, Per Steffensen wrote:
> Can anyone else elaborate? How to "activate" it? How to make sure, for
> sorting, that sort-field-value for all docs are not read into memory for
> sorting - leading to OOM when you have a lot of docs? Can this feature
> be activated on top of an existing 4.0 index, or do you have to re-index
> everything?
There is one requirement that may not be obvious - every document must
have a value in the field, so you must either make the field either
required or give it a default value in the schema. Solr 4.2 will refuse
to start the core if this requirement is not met. The example schema
hints that the value might need to be single-valued. I have not tested
this. Sorting is already problematic on multi-valued fields, so I
assume that this won't be the case for you.
To use docValues, add docValues="true" and then either set
required="true" or default="<somevalue>" on the field definition in
schema.xml, restart Solr or reload the core, and reindex. Your index
will get bigger.
If the touted behavior of handling the sort mechanism in OS disk cache
memory (or just reading the disk if there's not enough memory) rather
than heap is correct, then it should solve your issues. I hope it does!
Thanks,
Shawn
Re: Sort-field for ALL docs in FieldCache for sort queries -> OOM
on lots of docs
Posted by Per Steffensen <st...@designware.dk>.
On 3/21/13 10:52 AM, Toke Eskildsen wrote:
> On Thu, 2013-03-21 at 09:57 +0100, Per Steffensen wrote:
>> Thanks Toke! Can you please elaborate a little bit? How to use it? What
>> it is supposed to do for you?
> Sorry, no, I only know about it on the abstract level. The release notes
> for Solr 4.2 says
>
> * DocValues have been integrated into Solr. DocValues can be loaded up a
> lot faster than the field cache and can also use different compression
> algorithms as well as in RAM or on Disk representations. Faceting,
> sorting, and function queries all get to benefit. How about the OS
> handling faceting and sorting caches off heap? No more tuning 60
> gigabyte heaps? How about a snappy new per segment DocValues faceting
> method? Improved numeric faceting? Sweet.
>
> Spending 5 minutes searching on how to activate the new powers did not
> get me much; my Google-fu is clearly not strong enough. The example
> schema shows that docValues="true" is a valid attribute for "StrField,
> UUIDField and all Trie*Fields", but I do not know if they are used
> automatically by sort or if they should be requested explicitly.
>
> Regards,
> Toke Eskildsen
>
>
Thanks again, Toke!
Can anyone else elaborate? How to "activate" it? How to make sure, for
sorting, that sort-field-value for all docs are not read into memory for
sorting - leading to OOM when you have a lot of docs? Can this feature
be activated on top of an existing 4.0 index, or do you have to re-index
everything?
Thanks a lot for any feedback!
Regards, Per Steffensen
Re: Sort-field for ALL docs in FieldCache for sort queries -> OOM
on lots of docs
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Thu, 2013-03-21 at 09:57 +0100, Per Steffensen wrote:
> Thanks Toke! Can you please elaborate a little bit? How to use it? What
> it is supposed to do for you?
Sorry, no, I only know about it on the abstract level. The release notes
for Solr 4.2 says
* DocValues have been integrated into Solr. DocValues can be loaded up a
lot faster than the field cache and can also use different compression
algorithms as well as in RAM or on Disk representations. Faceting,
sorting, and function queries all get to benefit. How about the OS
handling faceting and sorting caches off heap? No more tuning 60
gigabyte heaps? How about a snappy new per segment DocValues faceting
method? Improved numeric faceting? Sweet.
Spending 5 minutes searching on how to activate the new powers did not
get me much; my Google-fu is clearly not strong enough. The example
schema shows that docValues="true" is a valid attribute for "StrField,
UUIDField and all Trie*Fields", but I do not know if they are used
automatically by sort or if they should be requested explicitly.
Regards,
Toke Eskildsen
Re: Sort-field for ALL docs in FieldCache for sort queries -> OOM
on lots of docs
Posted by Per Steffensen <st...@designware.dk>.
On 3/21/13 9:48 AM, Toke Eskildsen wrote:
> On Thu, 2013-03-21 at 09:13 +0100, Per Steffensen wrote:
>> We have a lot of docs in Solr. Each particular Solr-node handles a lot
>> of docs distributed among several replica. When you issue a sort query,
>> it seems to me that, the value of the sort-field of ALL docs under the
>> Solr-node is added to the FieldCache. [...]
> I haven't used it yet, but DocValues in Solr 4.2 seems to be the answer.
>
> - Toke Eskildsen
>
>
Thanks Toke! Can you please elaborate a little bit? How to use it? What
it is supposed to do for you?
Regards, Per Steffensen
Re: Sort-field for ALL docs in FieldCache for sort queries -> OOM
on lots of docs
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Thu, 2013-03-21 at 09:13 +0100, Per Steffensen wrote:
> We have a lot of docs in Solr. Each particular Solr-node handles a lot
> of docs distributed among several replica. When you issue a sort query,
> it seems to me that, the value of the sort-field of ALL docs under the
> Solr-node is added to the FieldCache. [...]
I haven't used it yet, but DocValues in Solr 4.2 seems to be the answer.
- Toke Eskildsen