You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by John Nielsen <jn...@mcb.dk> on 2013/03/24 09:19:31 UTC
Solr using a ridiculous amount of memory
Hello all,
We are running a solr cluster which is now running solr-4.2.
The index is about 35GB on disk with each register between 15k and 30k.
(This is simply the size of a full xml reply of one register. I'm not sure
how to measure it otherwise.)
Our memory requirements are running amok. We have less than a quarter of
our customers running now and even though we have allocated 25GB to the JVM
already, we are still seeing daily OOM crashes. We used to just allocate
more memory to the JVM, but with the way solr is scaling, we would need
well over 100GB of memory on each node to finish the project, and thats
just not going to happen. I need to lower the memory requirements somehow.
I can see from the memory dumps we've done that the field cache is by far
the biggest sinner. Of special interest to me is the recent introduction of
DocValues which supposedly mitigates this issue by using memory outside the
JVM. I just can't, because of lack of documentation, seem to make it work.
We do a lot of facetting. One client facets on about 50.000 docs of approx
30k each on 5 fields. I understand that this is VERY memory intensive.
Schema with DocValues attempt at solving problem:
http://pastebin.com/Ne23NnW4
Config: http://pastebin.com/x1qykyXW
The cache is pretty well tuned. Any lower and i get evictions.
Come hell or high water, my JVM memory requirements must come down. Simply
moving some memory load outside of the JVM would be awesome! Making it not
use the field cache for anything would also (probably) work for me. I
thought about killing off my other caches, but from the dumps, they just
don't seem to use that much memory.
I am at my wits end. Any help would be sorely appreciated.
--
Med venlig hilsen / Best regards
*John Nielsen*
Programmer
*MCB A/S*
Enghaven 15
DK-7500 Holstebro
Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk
Re: Solr using a ridiculous amount of memory
Posted by Jack Krupansky <ja...@basetechnology.com>.
Yeah, this is yet another "anti-pattern" we need to be discouraging - large
multivalued fields. They indicate that the data model is not well balanced
and aligned with the strengths of Solr and Lucene.
-- Jack Krupansky
-----Original Message-----
From: adityab
Sent: Sunday, June 16, 2013 9:36 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr using a ridiculous amount of memory
It was interesting to read this post. I had similar issue on Solr v4.2.1.
The
nature of our document is that it has huge multiValued fields and we were
able to knock off out server in about 30muns
We then found a bug "Lucene-4995" which was causing all the problem.
Applying the patch has helped a lot.
Not sure related but you might want to check that out.
Thanks.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-using-a-ridiculous-amount-of-memory-tp4050840p4070803.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr using a ridiculous amount of memory
Posted by adityab <ad...@yahoo.com>.
It was interesting to read this post. I had similar issue on Solr v4.2.1. The
nature of our document is that it has huge multiValued fields and we were
able to knock off out server in about 30muns
We then found a bug "Lucene-4995" which was causing all the problem.
Applying the patch has helped a lot.
Not sure related but you might want to check that out.
Thanks.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-using-a-ridiculous-amount-of-memory-tp4050840p4070803.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr using a ridiculous amount of memory
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Fri, 2013-06-14 at 14:55 +0200, John Nielsen wrote:
> Sorry for not getting back to the list sooner.
Time not important, only feedback important (apologies to Fifth
Element).
> After some major refactoring, our 15 cores have now turned into ~500 cores
> and our memory consumption has dropped dramaticly. Running 200 webshops now
> actually uses less memory as our 24 test shops did before.
That's great to hear. One core/shop also sounds like a cleaner setup.
> I looked at the wiki, but could not find any reference to this unintuitive
> way of using memory. Did I miss it somewhere?
I am not aware of a wikified explanation, but a section on "Why does
Solr use so much memory?" with some suggestions for changes to setup
would seem appropriate. You are not the first to have these kinds of
problems.
Thank you for closing the issue,
Toke Eskildsen
Re: Solr using a ridiculous amount of memory
Posted by Erick Erickson <er...@gmail.com>.
John:
If you'd like to add your experience to the Wiki, create
an ID and let us know what it is and we'll add you to the
contributors list. Unfortunately we had problems with
spam pages to we added this step.
Make sure you include your logon in the request.
Thanks,
Erick
On Fri, Jun 14, 2013 at 8:55 AM, John Nielsen <jn...@mcb.dk> wrote:
> Sorry for not getting back to the list sooner. It seems like I finally
> solved the memory problems by following Toke's instruction of splitting the
> cores up into smaller chunks.
>
> After some major refactoring, our 15 cores have now turned into ~500 cores
> and our memory consumption has dropped dramaticly. Running 200 webshops now
> actually uses less memory as our 24 test shops did before.
>
> Thank you to everyone who helped, and especially to Toke.
>
> I looked at the wiki, but could not find any reference to this unintuitive
> way of using memory. Did I miss it somewhere?
>
>
>
> On Fri, Apr 19, 2013 at 1:30 PM, Erick Erickson <er...@gmail.com>wrote:
>
>> Hmmm. There has been quite a bit of work lately to support a couple of
>> things that might be of interest (4.3, which Simon cut today, probably
>> available to all mid next week at the latest). Basically, you can
>> choose to pre-define all the cores in solr.xml (so-called "old style")
>> _or_ use the new-style solr.xml which uses "auto-discover" mode to
>> walk the indicated directory and find all the cores (indicated by the
>> presence of a 'core.properties' file). Don't know if this would make
>> your particular case easier, and I should warn you that this is
>> relatively new code (although there are some reasonable unit tests).
>>
>> You also have the option to only load the cores when they are
>> referenced, and only keep N cores open at a time (loadOnStartup and
>> transient properties).
>>
>> See: http://wiki.apache.org/solr/CoreAdmin#Configuration and
>> http://wiki.apache.org/solr/Solr.xml%204.3%20and%20beyond
>>
>> Note, the docs are somewhat sketchy, so if you try to go down this
>> route let us know anything that should be improved (or you can be
>> added to the list of wiki page contributors and help out!)
>>
>> Best
>> Erick
>>
>> On Thu, Apr 18, 2013 at 8:31 AM, John Nielsen <jn...@mcb.dk> wrote:
>> >> You are missing an essential part: Both the facet and the sort
>> >> structures needs to hold one reference for each document
>> >> _in_the_full_index_, even when the document does not have any values in
>> >> the fields.
>> >>
>> >
>> > Wow, thank you for this awesome explanation! This is where the penny
>> > dropped for me.
>> >
>> > I will definetely move to a multi-core setup. It will take some time and
>> a
>> > lot of re-coding. As soon as I know the result, I will let you know!
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Med venlig hilsen / Best regards
>> >
>> > *John Nielsen*
>> > Programmer
>> >
>> >
>> >
>> > *MCB A/S*
>> > Enghaven 15
>> > DK-7500 Holstebro
>> >
>> > Kundeservice: +45 9610 2824
>> > post@mcb.dk
>> > www.mcb.dk
>>
>
>
>
> --
> Med venlig hilsen / Best regards
>
> *John Nielsen*
> Programmer
>
>
>
> *MCB A/S*
> Enghaven 15
> DK-7500 Holstebro
>
> Kundeservice: +45 9610 2824
> post@mcb.dk
> www.mcb.dk
Re: Solr using a ridiculous amount of memory
Posted by John Nielsen <jn...@mcb.dk>.
Sorry for not getting back to the list sooner. It seems like I finally
solved the memory problems by following Toke's instruction of splitting the
cores up into smaller chunks.
After some major refactoring, our 15 cores have now turned into ~500 cores
and our memory consumption has dropped dramaticly. Running 200 webshops now
actually uses less memory as our 24 test shops did before.
Thank you to everyone who helped, and especially to Toke.
I looked at the wiki, but could not find any reference to this unintuitive
way of using memory. Did I miss it somewhere?
On Fri, Apr 19, 2013 at 1:30 PM, Erick Erickson <er...@gmail.com>wrote:
> Hmmm. There has been quite a bit of work lately to support a couple of
> things that might be of interest (4.3, which Simon cut today, probably
> available to all mid next week at the latest). Basically, you can
> choose to pre-define all the cores in solr.xml (so-called "old style")
> _or_ use the new-style solr.xml which uses "auto-discover" mode to
> walk the indicated directory and find all the cores (indicated by the
> presence of a 'core.properties' file). Don't know if this would make
> your particular case easier, and I should warn you that this is
> relatively new code (although there are some reasonable unit tests).
>
> You also have the option to only load the cores when they are
> referenced, and only keep N cores open at a time (loadOnStartup and
> transient properties).
>
> See: http://wiki.apache.org/solr/CoreAdmin#Configuration and
> http://wiki.apache.org/solr/Solr.xml%204.3%20and%20beyond
>
> Note, the docs are somewhat sketchy, so if you try to go down this
> route let us know anything that should be improved (or you can be
> added to the list of wiki page contributors and help out!)
>
> Best
> Erick
>
> On Thu, Apr 18, 2013 at 8:31 AM, John Nielsen <jn...@mcb.dk> wrote:
> >> You are missing an essential part: Both the facet and the sort
> >> structures needs to hold one reference for each document
> >> _in_the_full_index_, even when the document does not have any values in
> >> the fields.
> >>
> >
> > Wow, thank you for this awesome explanation! This is where the penny
> > dropped for me.
> >
> > I will definetely move to a multi-core setup. It will take some time and
> a
> > lot of re-coding. As soon as I know the result, I will let you know!
> >
> >
> >
> >
> >
> >
> > --
> > Med venlig hilsen / Best regards
> >
> > *John Nielsen*
> > Programmer
> >
> >
> >
> > *MCB A/S*
> > Enghaven 15
> > DK-7500 Holstebro
> >
> > Kundeservice: +45 9610 2824
> > post@mcb.dk
> > www.mcb.dk
>
--
Med venlig hilsen / Best regards
*John Nielsen*
Programmer
*MCB A/S*
Enghaven 15
DK-7500 Holstebro
Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk
Re: Solr using a ridiculous amount of memory
Posted by Erick Erickson <er...@gmail.com>.
Hmmm. There has been quite a bit of work lately to support a couple of
things that might be of interest (4.3, which Simon cut today, probably
available to all mid next week at the latest). Basically, you can
choose to pre-define all the cores in solr.xml (so-called "old style")
_or_ use the new-style solr.xml which uses "auto-discover" mode to
walk the indicated directory and find all the cores (indicated by the
presence of a 'core.properties' file). Don't know if this would make
your particular case easier, and I should warn you that this is
relatively new code (although there are some reasonable unit tests).
You also have the option to only load the cores when they are
referenced, and only keep N cores open at a time (loadOnStartup and
transient properties).
See: http://wiki.apache.org/solr/CoreAdmin#Configuration and
http://wiki.apache.org/solr/Solr.xml%204.3%20and%20beyond
Note, the docs are somewhat sketchy, so if you try to go down this
route let us know anything that should be improved (or you can be
added to the list of wiki page contributors and help out!)
Best
Erick
On Thu, Apr 18, 2013 at 8:31 AM, John Nielsen <jn...@mcb.dk> wrote:
>> You are missing an essential part: Both the facet and the sort
>> structures needs to hold one reference for each document
>> _in_the_full_index_, even when the document does not have any values in
>> the fields.
>>
>
> Wow, thank you for this awesome explanation! This is where the penny
> dropped for me.
>
> I will definetely move to a multi-core setup. It will take some time and a
> lot of re-coding. As soon as I know the result, I will let you know!
>
>
>
>
>
>
> --
> Med venlig hilsen / Best regards
>
> *John Nielsen*
> Programmer
>
>
>
> *MCB A/S*
> Enghaven 15
> DK-7500 Holstebro
>
> Kundeservice: +45 9610 2824
> post@mcb.dk
> www.mcb.dk
Re: Solr using a ridiculous amount of memory
Posted by John Nielsen <jn...@mcb.dk>.
> You are missing an essential part: Both the facet and the sort
> structures needs to hold one reference for each document
> _in_the_full_index_, even when the document does not have any values in
> the fields.
>
Wow, thank you for this awesome explanation! This is where the penny
dropped for me.
I will definetely move to a multi-core setup. It will take some time and a
lot of re-coding. As soon as I know the result, I will let you know!
--
Med venlig hilsen / Best regards
*John Nielsen*
Programmer
*MCB A/S*
Enghaven 15
DK-7500 Holstebro
Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk
Re: Solr using a ridiculous amount of memory
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Thu, 2013-04-18 at 11:59 +0200, John Nielsen wrote:
> Yes, thats right. No search from any given client ever returns
> anything from another client.
Great. That makes the 1 core/client solution feasible.
[No sort & facet warmup is performed]
[Suggestion 1: Reduce the number of sort fields by mapping]
[Suggestion 3: 1 core/customer]
> If I understand the fieldCache mechanism correctly (which i can see
> that I don't), the data used for faceting and sorting is saved in the
> fieldCache using a key comprised of the fields used for said
> faceting/sorting. That data only contains the data which is actually
> used for the operation. This is what the fq queries are for.
>
You are missing an essential part: Both the facet and the sort
structures needs to hold one reference for each document
_in_the_full_index_, even when the document does not have any values in
the fields.
It might help to visualize the structures as arrays of values with docID
as index: String[] myValues = new String[1400000] takes up 1.4M * 32 bit
(or more for a 64 bit machine) = 5.6MB, even when it is empty.
Note: Neither String-objects, nor Java references are used for the real
facet- and sort-structures, but the principle is quite the same.
> So if i generate a core for each client, I would have a client
> specific fieldCache containing the data from that client. Wouldn't I
> just split up the same data into several cores?
The same terms, yes, but not the same references.
Let's say your customer has 10K documents in the index and that there
are 100 unique values, each 10 bytes long, in each group .
As each group holds its own separate structure, we use the old formula
to get the memory overhead:
#documents*log2(#unique_terms*average_term_length) +
#unique_terms*average_term_length
>
1.4M*log2(100*(10*8)) + 100*(10*8) bit = 1.2MB + 1KB.
Note how the values themselves are just 1KB, while the nearly empty
reference list takes 1.2MB.
Compare this to a dedicated core with just the 10K documents:
10K*log2(100*(10*8)) + 100*(10*8) bit = 8.5KB + 1KB.
The terms take up exactly the same space, but the heap requirement for
the references is reduced by 99%.
Now, 25GB for 180 clients means 140MB/client with your current setup.
I do not know the memory overhead of running a core, but since Solr can
run fine with 32MB for small indexes, it should be smaller than that.
You will of course have to experiment and to measure.
- Toke Eskildsen, State and University Library, Denmark
Re: Solr using a ridiculous amount of memory
Posted by John Nielsen <jn...@mcb.dk>.
>
> > http://172.22.51.111:8000/solr/default1_Danish/search
>
> [...]
>
> > &fq=site_guid%3a(10217)
>
> This constraints to hits to a specific customer, right? Any search will
> only be in a single customer's data?
>
Yes, thats right. No search from any given client ever returns anything
from another client.
[Toke: Are you warming all the sort- and facet-fields?]
>
> > I'm sorry, I don't know. I have the field value cache commented out in
> > my config, so... Whatever is default?
>
> (a bit shaky here) I would say not warming. You could check simply by
> starting solr and looking at the caches before you issue any searches.
>
The field cache shows 0 entries at startup. On the running server, forcing
a commit (and thus opening a new searcher) does not change the number of
entries.
> > The problem is that each item can have several sort orders. The sort
> > order to use is defined by a group number which is known ahead of
> > time. The group number is included in the sort order field name. To
> > solve it in the same way i solved the facet problem, I would need to
> > be able to sort on a multi-valued field, and unless I'm wrong, I don't
> > think that it's possible.
>
> That is correct.
>
> Three suggestions off the bat:
>
> 1) Reduce the number of sort fields by mapping names.
> Count the maximum number of unique sort fields for any given customer.
> That will be the total number of sort fields in the index. For each
> group number for a customer, map that number to one of the index-wide
> sort fields.
> This only works if the maximum number of unique fields is low (let's say
> a single field takes 50MB, so 20 fields should be okay).
>
I just checked our DB. Our worst case scenario client has over a thousand
groups for sorting. Granted, it may be, probably is, an error with the
data. It is an interesting idea though and I will look into this posibility.
> 3) Switch to a layout where each customer has a dedicated core.
> The basic overhead is a lot larger than for a shared index, but it would
> make your setup largely immune to the adverse effect of many documents
> coupled with many facet- and sort-fields.
>
Now this is where my brain melts down.
If I understand the fieldCache mechanism correctly (which i can see that I
don't), the data used for faceting and sorting is saved in the fieldCache
using a key comprised of the fields used for said faceting/sorting. That
data only contains the data which is actually used for the operation. This
is what the fq queries are for.
So if i generate a core for each client, I would have a client specific
fieldCache containing the data from that client. Wouldn't I just split up
the same data into several cores?
I'm afraid I don't understand how this would help.
--
Med venlig hilsen / Best regards
*John Nielsen*
Programmer
*MCB A/S*
Enghaven 15
DK-7500 Holstebro
Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk
Re: Solr using a ridiculous amount of memory
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Thu, 2013-04-18 at 08:34 +0200, John Nielsen wrote:
>
[Toke: Can you find the facet fields in any of the other caches?]
> Yes, here it is, in the field cache:
> http://screencast.com/t/mAwEnA21yL
>
Ah yes, mystery solved, my mistake.
> http://172.22.51.111:8000/solr/default1_Danish/search
[...]
> &fq=site_guid%3a(10217)
This constraints to hits to a specific customer, right? Any search will
only be in a single customer's data?
>
[Toke: Are you warming all the sort- and facet-fields?]
> I'm sorry, I don't know. I have the field value cache commented out in
> my config, so... Whatever is default?
(a bit shaky here) I would say not warming. You could check simply by
starting solr and looking at the caches before you issue any searches.
This fits the description of your searchers gradually eating memory
until your JVM OOMs. Each time a new field is faceted or sorted upon, it
it added to the cache. As your index is relatively small and the number
of values in the single fields is small, the initialization time for a
field is so short that it is not a performance problem. Memory wise is
is death by a thousand cuts.
If you did explicit warming of all the possible fields for sorting and
faceting, your would allocate it all up front and would be sure that
there would be enough memory available. But it would take much longer
than your current setup. You might want to try it out (no need to fiddle
with Solr setup, just make a script and fire wgets as this has the same
effect).
> The problem is that each item can have several sort orders. The sort
> order to use is defined by a group number which is known ahead of
> time. The group number is included in the sort order field name. To
> solve it in the same way i solved the facet problem, I would need to
> be able to sort on a multi-valued field, and unless I'm wrong, I don't
> think that it's possible.
That is correct.
Three suggestions off the bat:
1) Reduce the number of sort fields by mapping names.
Count the maximum number of unique sort fields for any given customer.
That will be the total number of sort fields in the index. For each
group number for a customer, map that number to one of the index-wide
sort fields.
This only works if the maximum number of unique fields is low (let's say
a single field takes 50MB, so 20 fields should be okay).
2) Create a custom sorter for Solr.
Create a field with all the sort values, prefixed by group ID. Create a
structure (or reuse the one from Lucene) with a doc->terms map with all
the terms in-memory. When sorting, extract the relevant compare-string
for a document by iterating all the terms for the document and selecting
the one with the right prefix.
Memory wise this scales linear to the number of terms instead of the
number of fields, but it would require quite some coding.
3) Switch to a layout where each customer has a dedicated core.
The basic overhead is a lot larger than for a shared index, but it would
make your setup largely immune to the adverse effect of many documents
coupled with many facet- and sort-fields.
- Toke Eskildsen, State and University Library, Denmark
Re: Solr using a ridiculous amount of memory
Posted by John Nielsen <jn...@mcb.dk>.
> That was strange. As you are using a multi-valued field with the new
setup, they should appear there.
Yes, the new field we use for faceting is a multi valued field.
> Can you find the facet fields in any of the other caches?
Yes, here it is, in the field cache:
http://screencast.com/t/mAwEnA21yL
> I hope you are not calling the facets with facet.method=enum? Could you
paste a typical facet-enabled search request?
Here is a typical example (I added newlines for readability):
http://172.22.51.111:8000/solr/default1_Danish/search
?defType=edismax
&q=*%3a*
&facet.field=%7b!ex%3dtagitemvariantoptions_int_mv_7+key%3ditemvariantoptions_int_mv_7%7ditemvariantoptions_int_mv
&facet.field=%7b!ex%3dtagitemvariantoptions_int_mv_9+key%3ditemvariantoptions_int_mv_9%7ditemvariantoptions_int_mv
&facet.field=%7b!ex%3dtagitemvariantoptions_int_mv_8+key%3ditemvariantoptions_int_mv_8%7ditemvariantoptions_int_mv
&facet.field=%7b!ex%3dtagitemvariantoptions_int_mv_2+key%3ditemvariantoptions_int_mv_2%7ditemvariantoptions_int_mv
&fq=site_guid%3a(10217)
&fq=item_type%3a(PRODUCT)
&fq=language_guid%3a(1)
&fq=item_group_1522_combination%3a(*)
&fq=is_searchable%3a(True)
&sort=item_group_1522_name_int+asc, variant_of_item_guid+asc
&querytype=Technical
&fl=feed_item_serialized
&facet=true
&group=true
&group.facet=true
&group.ngroups=true
&group.field=groupby_variant_of_item_guid
&group.sort=name+asc
&rows=0
> Are you warming all the sort- and facet-fields?
I'm sorry, I don't know. I have the field value cache commented out in my
config, so... Whatever is default?
Removing the custom sort fields is unfortunately quite a bit more difficult
than my other facet modification.
The problem is that each item can have several sort orders. The sort order
to use is defined by a group number which is known ahead of time. The group
number is included in the sort order field name. To solve it in the same
way i solved the facet problem, I would need to be able to sort on a
multi-valued field, and unless I'm wrong, I don't think that it's possible.
I am quite stomped on how to fix this.
On Wed, Apr 17, 2013 at 3:06 PM, Toke Eskildsen <te...@statsbiblioteket.dk>wrote:
> John Nielsen [jn@mcb.dk]:
> > I never seriously looked at my fieldValueCache. It never seemed to get
> used:
>
> > http://screencast.com/t/YtKw7UQfU
>
> That was strange. As you are using a multi-valued field with the new
> setup, they should appear there. Can you find the facet fields in any of
> the other caches?
>
> ...I hope you are not calling the facets with facet.method=enum? Could you
> paste a typical facet-enabled search request?
>
> > Yep. We still do a lot of sorting on dynamic field names, so the field
> cache
> > has a lot of entries. (9.411 entries as we speak. This is considerably
> lower
> > than before.). You mentioned in an earlier mail that faceting on a field
> > shared between all facet queries would bring down the memory needed.
> > Does the same thing go for sorting?
>
> More or less. Sorting stores the raw string representations (utf-8) in
> memory so the number of unique values has more to say than it does for
> faceting. Just as with faceting, a list of pointers from documents to
> values (1 value/document as we are sorting) is maintained, so the overhead
> is something like
>
> #documents*log2(#unique_terms*average_term_length) +
> #unique_terms*average_term_length
> (where average_term_length is in bits)
>
> Caveat: This is with the index-wide sorting structure. I am fairly
> confident that this is what Solr uses, but I have not looked at it lately
> so it is possible that some memory-saving segment-based trickery has been
> implemented.
>
> > Does those 9411 entries duplicate data between them?
>
> Sorry, I do not know. SOLR-1111 discusses the problems with the field
> cache and duplication of data, but I cannot infer if it is has been solved
> or not. I am not familiar with the stat breakdown of the fieldCache, but it
> _seems_ to me that there are 2 or 3 entries for each segment for each sort
> field. Guesstimating further, let's say you have 30 segments in your index.
> Going with the guesswork, that would bring the number of sort fields to
> 9411/3/30 ~= 100. Looks like you use a custom sort field for each client?
>
> Extrapolating from 1.4M documents and 180 clients, let's say that there
> are 1.4M/180/5 unique terms for each sort-field and that their average
> length is 10. We thus have
> 1.4M*log2(1500*10*8) + 1500*10*8 bit ~= 23MB
> per sort field or about 4GB for all the 180 fields.
>
> With this few unique values, the doc->value structure is by far the
> biggest, just as with facets. As opposed to the faceting structure, this is
> fairly close to the actual memory usage. Switching to a single sort field
> would reduce the memory usage from 4GB to about 55MB.
>
> > I do commit a bit more often than i should. I get these in my log file
> from
> > time to time: PERFORMANCE WARNING: Overlapping onDeckSearchers=2
>
> So 1 active searcher and 2 warming searchers. Ignoring that one of the
> warming searchers is highly likely to finish well ahead of the other one,
> that means that your heap must hold 3 times the structures for a single
> searcher. With the old heap size of 25GB that left "only" 8GB for a full
> dataset. Subtract the 4GB for sorting and a similar amount for faceting and
> you have your OOM.
>
> Tweaking your ingest to avoid 3 overlapping searchers will lower your
> memory requirements by 1/3. Fixing the facet & sorting logic will bring it
> down to laptop size.
>
> > The control panel says that the warm up time of the last searcher is
> 5574. Is that seconds or milliseconds?
> > http://screencast.com/t/d9oIbGLCFQwl
>
> milliseconds, I am fairly sure. It is much faster than I anticipated. Are
> you warming all the sort- and facet-fields?
>
> > Waiting for a full GC would take a long time.
>
> Until you have fixed the core memory issue, you might consider doing an
> explicit GC every night to clean up and hope that it does not occur
> automatically at daytime (or whenever your clients uses it).
>
> > Unfortunately I don't know of a way to provoke a full GC on command.
>
> VisualVM, which is delivered with the Oracle JDK (look somewhere in the
> bin folder), is your friend. Just start it on the server and click on the
> relevant process.
>
> Regards,
> Toke Eskildsen
--
Med venlig hilsen / Best regards
*John Nielsen*
Programmer
*MCB A/S*
Enghaven 15
DK-7500 Holstebro
Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk
RE: Solr using a ridiculous amount of memory
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
Whopps. I made some mistakes in the previous post.
Toke Eskildsen [te@statsbiblioteket.dk]:
> Extrapolating from 1.4M documents and 180 clients, let's say that
> there are 1.4M/180/5 unique terms for each sort-field and that their
> average length is 10. We thus have
> 1.4M*log2(1500*10*8) + 1500*10*8 bit ~= 23MB
> per sort field or about 4GB for all the 180 fields.
That would be 10 bytes and thus 80 bits. The results were correct though.
> So 1 active searcher and 2 warming searchers. Ignoring that one of
> the warming searchers is highly likely to finish well ahead of the other
> one, that means that your heap must hold 3 times the structures for
> a single searcher.
This should be taken with a grain of salt as it depends on whether or not there is any re-use of segments. There might be for sorting.
Apologies for any confusion,
Toke Eskildsen
RE: Solr using a ridiculous amount of memory
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
John Nielsen [jn@mcb.dk]:
> I never seriously looked at my fieldValueCache. It never seemed to get used:
> http://screencast.com/t/YtKw7UQfU
That was strange. As you are using a multi-valued field with the new setup, they should appear there. Can you find the facet fields in any of the other caches?
...I hope you are not calling the facets with facet.method=enum? Could you paste a typical facet-enabled search request?
> Yep. We still do a lot of sorting on dynamic field names, so the field cache
> has a lot of entries. (9.411 entries as we speak. This is considerably lower
> than before.). You mentioned in an earlier mail that faceting on a field
> shared between all facet queries would bring down the memory needed.
> Does the same thing go for sorting?
More or less. Sorting stores the raw string representations (utf-8) in memory so the number of unique values has more to say than it does for faceting. Just as with faceting, a list of pointers from documents to values (1 value/document as we are sorting) is maintained, so the overhead is something like
#documents*log2(#unique_terms*average_term_length) + #unique_terms*average_term_length
(where average_term_length is in bits)
Caveat: This is with the index-wide sorting structure. I am fairly confident that this is what Solr uses, but I have not looked at it lately so it is possible that some memory-saving segment-based trickery has been implemented.
> Does those 9411 entries duplicate data between them?
Sorry, I do not know. SOLR-1111 discusses the problems with the field cache and duplication of data, but I cannot infer if it is has been solved or not. I am not familiar with the stat breakdown of the fieldCache, but it _seems_ to me that there are 2 or 3 entries for each segment for each sort field. Guesstimating further, let's say you have 30 segments in your index. Going with the guesswork, that would bring the number of sort fields to 9411/3/30 ~= 100. Looks like you use a custom sort field for each client?
Extrapolating from 1.4M documents and 180 clients, let's say that there are 1.4M/180/5 unique terms for each sort-field and that their average length is 10. We thus have
1.4M*log2(1500*10*8) + 1500*10*8 bit ~= 23MB
per sort field or about 4GB for all the 180 fields.
With this few unique values, the doc->value structure is by far the biggest, just as with facets. As opposed to the faceting structure, this is fairly close to the actual memory usage. Switching to a single sort field would reduce the memory usage from 4GB to about 55MB.
> I do commit a bit more often than i should. I get these in my log file from
> time to time: PERFORMANCE WARNING: Overlapping onDeckSearchers=2
So 1 active searcher and 2 warming searchers. Ignoring that one of the warming searchers is highly likely to finish well ahead of the other one, that means that your heap must hold 3 times the structures for a single searcher. With the old heap size of 25GB that left "only" 8GB for a full dataset. Subtract the 4GB for sorting and a similar amount for faceting and you have your OOM.
Tweaking your ingest to avoid 3 overlapping searchers will lower your memory requirements by 1/3. Fixing the facet & sorting logic will bring it down to laptop size.
> The control panel says that the warm up time of the last searcher is 5574. Is that seconds or milliseconds?
> http://screencast.com/t/d9oIbGLCFQwl
milliseconds, I am fairly sure. It is much faster than I anticipated. Are you warming all the sort- and facet-fields?
> Waiting for a full GC would take a long time.
Until you have fixed the core memory issue, you might consider doing an explicit GC every night to clean up and hope that it does not occur automatically at daytime (or whenever your clients uses it).
> Unfortunately I don't know of a way to provoke a full GC on command.
VisualVM, which is delivered with the Oracle JDK (look somewhere in the bin folder), is your friend. Just start it on the server and click on the relevant process.
Regards,
Toke Eskildsen
Re: Solr using a ridiculous amount of memory
Posted by John Nielsen <jn...@mcb.dk>.
> I am surprised about the lack of "UnInverted" from your logs as it is
logged on INFO level.
Nope, no trace of it. No mention either in Logging -> Level from the admin
interface.
> It should also be available from the admin interface under
collection/Plugin / Stats/CACHE/fieldValueCache.
I never seriously looked at my fieldValueCache. It never seemed to get used:
http://screencast.com/t/YtKw7UQfU
> You stated that you were unable to make a 4GB JVM OOM when you just
performed faceting (I guesstimate that it will also run fine with just ½GB
or at least with 1GB, based on the
> numbers above) and you have observed that the field cache eats the
memory.
Yep. We still do a lot of sorting on dynamic field names, so the field
cache has a lot of entries. (9.411 entries as we speak. This is
considerably lower than before.). You mentioned in an earlier mail that
faceting on a field shared between all facet queries would bring down the
memory needed. Does the same thing go for sorting? Does those 9411 entries
duplicate data between them? If this is where all the memory is going, I
have a lot of coding to do.
> Guessing wildly: Do you issue a high frequency small updates with
frequent commits? If you pause the indexing, does memory use fall back to
the single GB level
I do commit a bit more often than i should. I get these in my log file from
time to time: PERFORMANCE WARNING: Overlapping onDeckSearchers=2 The way I
understand this is that two searchers are being warmed at the same time and
that one will be discarded when it finishes its auto warming procedure. If
the math above is correct, I would need tens of searchers auto
warming in parallel to cause my problem. If I misunderstand how this works,
do let me know.
My indexer has a cleanup routine that deletes replay logs and other things
when it has nothing to do. This includes running a commit on the solr
server to make sure nothing is ever in a state where something is not
written to disk anywhere. In theory it can commit once every 60 seconds,
though i doubt that ever happenes. The less work the indexer has, the more
often it commits. (yes i know, its on my todo list)
Other than that, my autocommit settings look like this:
<autoCommit> <maxTime>60000</maxTime> <maxDocs>6000</maxDocs> <openSearcher>
false</openSearcher> </autoCommit>
The control panel says that the warm up time of the last searcher is 5574.
Is that seconds or milliseconds?
http://screencast.com/t/d9oIbGLCFQwl
I would prefer to not turn off the indexer unless the numbers above
suggests that I really should try this. Waiting for a full GC would take a
long time. Unfortunately I don't know of a way to provoke a full GC on
command.
On Wed, Apr 17, 2013 at 11:48 AM, Toke Eskildsen <te...@statsbiblioteket.dk>wrote:
> John Nielsen [jn@mcb.dk] wrote:
> > I managed to get this done. The facet queries now facets on a multivalue
> field as opposed to the dynamic field names.
>
> > Unfortunately it doesn't seem to have done much difference, if any at
> all.
>
> I am sorry to hear that.
>
> > documents = ~1.400.000
> > references 11.200.000 (we facet on two multivalue fields with each 4
> values
> > on average, so 1.400.000 * 2 * 4 = 11.200.000
> > unique values = 1.132.344 (total number of variant options across all
> clients.
> > This is what we facet on)
>
> > 1.400.000 * log2(11.200.000) + 1.400.000 * log2(1132344) = ~14MB per
> field (we have 4 fields)?
>
> > I must be calculating this wrong.
>
> No, that sounds about right. In reality you need to multiply with 3 or 4,
> so let's round to 50MB/field: 1.4M documents with 2 fields with 5M
> references/field each is not very much and should not take a lot of memory.
> In comparison, we facet on 12M documents with 166M references and do some
> other stuff (in Lucene with a different faceting implementation, but at
> this level it is equivalent to Solr's in terms of memory). Our heap is 3GB.
>
> I am surprised about the lack of "UnInverted" from your logs as it is
> logged on INFO level. It should also be available from the admin interface
> under collection/Plugin / Stats/CACHE/fieldValueCache. But I am guessing
> you got your numbers from that and that the list only contains the few
> facets you mentioned previously? It might be wise to sanity check by
> summing the memSizes though; they ought to take up far below 1GB.
>
> From your description, your index is small and your faceting requirements
> modest. A SSD-equipped laptop should be adequate as server. So we are back
> to "math does not check out".
>
>
> You stated that you were unable to make a 4GB JVM OOM when you just
> performed faceting (I guesstimate that it will also run fine with just ½GB
> or at least with 1GB, based on the numbers above) and you have observed
> that the field cache eats the memory. This does indicate that the old
> caches are somehow not freed when the index is updated. That is strange as
> Solr should take care of that automatically.
>
> Guessing wildly: Do you issue a high frequency small updates with frequent
> commits? If you pause the indexing, does memory use fall back to the single
> GB level (You probably need to trigger a full GC to check that)? If that is
> the case, it might be a warmup problem with old warmups still running when
> new commits are triggered.
>
> Regards,
> Toke Eskildsen, State and University Library, Denmark
--
Med venlig hilsen / Best regards
*John Nielsen*
Programmer
*MCB A/S*
Enghaven 15
DK-7500 Holstebro
Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk
RE: Solr using a ridiculous amount of memory
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
John Nielsen [jn@mcb.dk] wrote:
> I managed to get this done. The facet queries now facets on a multivalue field as opposed to the dynamic field names.
> Unfortunately it doesn't seem to have done much difference, if any at all.
I am sorry to hear that.
> documents = ~1.400.000
> references 11.200.000 (we facet on two multivalue fields with each 4 values
> on average, so 1.400.000 * 2 * 4 = 11.200.000
> unique values = 1.132.344 (total number of variant options across all clients.
> This is what we facet on)
> 1.400.000 * log2(11.200.000) + 1.400.000 * log2(1132344) = ~14MB per field (we have 4 fields)?
> I must be calculating this wrong.
No, that sounds about right. In reality you need to multiply with 3 or 4, so let's round to 50MB/field: 1.4M documents with 2 fields with 5M references/field each is not very much and should not take a lot of memory. In comparison, we facet on 12M documents with 166M references and do some other stuff (in Lucene with a different faceting implementation, but at this level it is equivalent to Solr's in terms of memory). Our heap is 3GB.
I am surprised about the lack of "UnInverted" from your logs as it is logged on INFO level. It should also be available from the admin interface under collection/Plugin / Stats/CACHE/fieldValueCache. But I am guessing you got your numbers from that and that the list only contains the few facets you mentioned previously? It might be wise to sanity check by summing the memSizes though; they ought to take up far below 1GB.
>From your description, your index is small and your faceting requirements modest. A SSD-equipped laptop should be adequate as server. So we are back to "math does not check out".
You stated that you were unable to make a 4GB JVM OOM when you just performed faceting (I guesstimate that it will also run fine with just ½GB or at least with 1GB, based on the numbers above) and you have observed that the field cache eats the memory. This does indicate that the old caches are somehow not freed when the index is updated. That is strange as Solr should take care of that automatically.
Guessing wildly: Do you issue a high frequency small updates with frequent commits? If you pause the indexing, does memory use fall back to the single GB level (You probably need to trigger a full GC to check that)? If that is the case, it might be a warmup problem with old warmups still running when new commits are triggered.
Regards,
Toke Eskildsen, State and University Library, Denmark
Re: Solr using a ridiculous amount of memory
Posted by John Nielsen <jn...@mcb.dk>.
I managed to get this done. The facet queries now facets on a multivalue
field as opposed to the dynamic field names.
Unfortunately it doesn't seem to have done much difference, if any at all.
Some more information that might help:
The JVM memory seem to be eaten up slowly. I dont think that there is one
single query that causes the problem. My test case (dumping 180 clients on
top of solr) takes hours before it causes an OOM. Often a full day. The
memory usage wobbles up and down, so the GC is at least partially doing its
job. It still works its way up to 100% eventually. When that happens it
either OOM's or it stops the world and brings the memory consumption to
10-15 gigs.
I did try to facet on all products across all clients (about 1.4 mil docs)
and i could not make it OOM on a server with a 4 gig jvm. This was on a
dedicated test server with my test being the only traffic.
I am beginning to think that this may be related to traffic volume and not
just on the type of query that I do.
I tried to calculate the memory requirement example you gave me above based
on the change that got rid of the dynamic fields.
documents = ~1.400.000
references 11.200.000 (we facet on two multivalue fields with each 4
values on average, so 1.400.000 * 2 * 4 = 11.200.000
unique values = 1.132.344 (total number of variant options across all
clients. This is what we facet on)
1.400.000 * log2(11.200.000) + 1.400.000 * log2(1132344) = ~14MB per field
(we have 4 fields)?
I must be calculating this wrong.
On Mon, Apr 15, 2013 at 2:10 PM, John Nielsen <jn...@mcb.dk> wrote:
> I did a search. I have no occurrence of "UnInverted" in the solr logs.
>
> > Another explanation for the large amount of memory presents itself if
> > you use a single index: If each of your clients facet on at least one
> > fields specific to the client ("client123_persons" or something like
> > that), then your memory usage goes through the roof.
>
> This is exactly how we facet right now! I will definetely rewrite the
> relevant parts of our product to test this out before moving further down
> the docValues path.
>
> I will let you know as soon as I know one way or the other.
>
>
> On Mon, Apr 15, 2013 at 1:38 PM, Toke Eskildsen <te...@statsbiblioteket.dk>wrote:
>
>> On Mon, 2013-04-15 at 10:25 +0200, John Nielsen wrote:
>>
>> > The FieldCache is the big culprit. We do a huge amount of faceting so
>> > it seems right.
>>
>> Yes, you wrote that earlier. The mystery is that the math does not check
>> out with the description you have given us.
>>
>> > Unfortunately I am super swamped at work so I have precious little
>> > time to work on this, which is what explains my silence.
>>
>> No problem, we've all been there.
>> >
>> [Band aid: More memory]
>>
>> > The extra memory helped a lot, but it still OOM with about 180 clients
>> > using it.
>>
>> You stated earlier that you has a "solr cluster" and your total(?) index
>> size was 35GB, with each "register" being between "15k" and "30k". I am
>> using the quotes to signify that it is unclear what you mean. Is your
>> cluster multiple machines (I'm guessing no), multiple Solr's, cores,
>> shards or maybe just a single instance prepared for later distribution?
>> Is a register a core, shard or a simply logical part (one client's data)
>> of the index?
>>
>> If each client has their own core or shard, that would mean that each
>> client uses more than 25GB/180 bytes ~= 142MB of heap to access 35GB/180
>> ~= 200MB of index. That sounds quite high and you would need a very
>> heavy facet to reach that.
>>
>> If you could grep "UnInverted" from the Solr log file and paste the
>> entries here, that would help to clarify things.
>>
>>
>> Another explanation for the large amount of memory presents itself if
>> you use a single index: If each of your clients facet on at least one
>> fields specific to the client ("client123_persons" or something like
>> that), then your memory usage goes through the roof.
>>
>> Assuming an index with 10M documents, each with 5 references to a modest
>> 10K unique values in a facet field, the simplified formula
>> #documents*log2(#references) + #references*log2(#unique_values) bit
>> tells us that this takes at least 110MB with field cache based faceting.
>>
>> 180 clients @ 110MB ~= 20GB. As that is a theoretical low, we can at
>> least double that. This fits neatly with your new heap of 64GB.
>>
>>
>> If my guessing is correct, you can solve your memory problems very
>> easily by sharing _all_ the facet fields between your clients.
>> This should bring your memory usage down to a few GB.
>>
>> You are probably already restricting their searches to their own data by
>> filtering, so this should not influence the returned facet values and
>> counts, as compared to separate fields.
>>
>> This is very similar to the thread "Facets with 5000 facet fields" BTW.
>>
>> > Today I finally managed to set up a test core so I can begin to play
>> > around with docValues.
>>
>> If you are using a single index with the individual-facet-fields for
>> each client approach, the DocValues will also have scaling issues, as
>> the amount of values (of which the majority will be null) will be
>> #clients*#documents*#facet_fields
>> This means that the adding a new client will be progressively more
>> expensive.
>>
>> On the other hand, if you use a lot of small shards, DocValues should
>> work for you.
>>
>> Regards,
>> Toke Eskildsen
>>
>>
>>
>
>
> --
> Med venlig hilsen / Best regards
>
> *John Nielsen*
> Programmer
>
>
>
> *MCB A/S*
> Enghaven 15
> DK-7500 Holstebro
>
> Kundeservice: +45 9610 2824
> post@mcb.dk
> www.mcb.dk
>
--
Med venlig hilsen / Best regards
*John Nielsen*
Programmer
*MCB A/S*
Enghaven 15
DK-7500 Holstebro
Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk
Re: Solr using a ridiculous amount of memory
Posted by John Nielsen <jn...@mcb.dk>.
I did a search. I have no occurrence of "UnInverted" in the solr logs.
> Another explanation for the large amount of memory presents itself if
> you use a single index: If each of your clients facet on at least one
> fields specific to the client ("client123_persons" or something like
> that), then your memory usage goes through the roof.
This is exactly how we facet right now! I will definetely rewrite the
relevant parts of our product to test this out before moving further down
the docValues path.
I will let you know as soon as I know one way or the other.
On Mon, Apr 15, 2013 at 1:38 PM, Toke Eskildsen <te...@statsbiblioteket.dk>wrote:
> On Mon, 2013-04-15 at 10:25 +0200, John Nielsen wrote:
>
> > The FieldCache is the big culprit. We do a huge amount of faceting so
> > it seems right.
>
> Yes, you wrote that earlier. The mystery is that the math does not check
> out with the description you have given us.
>
> > Unfortunately I am super swamped at work so I have precious little
> > time to work on this, which is what explains my silence.
>
> No problem, we've all been there.
> >
> [Band aid: More memory]
>
> > The extra memory helped a lot, but it still OOM with about 180 clients
> > using it.
>
> You stated earlier that you has a "solr cluster" and your total(?) index
> size was 35GB, with each "register" being between "15k" and "30k". I am
> using the quotes to signify that it is unclear what you mean. Is your
> cluster multiple machines (I'm guessing no), multiple Solr's, cores,
> shards or maybe just a single instance prepared for later distribution?
> Is a register a core, shard or a simply logical part (one client's data)
> of the index?
>
> If each client has their own core or shard, that would mean that each
> client uses more than 25GB/180 bytes ~= 142MB of heap to access 35GB/180
> ~= 200MB of index. That sounds quite high and you would need a very
> heavy facet to reach that.
>
> If you could grep "UnInverted" from the Solr log file and paste the
> entries here, that would help to clarify things.
>
>
> Another explanation for the large amount of memory presents itself if
> you use a single index: If each of your clients facet on at least one
> fields specific to the client ("client123_persons" or something like
> that), then your memory usage goes through the roof.
>
> Assuming an index with 10M documents, each with 5 references to a modest
> 10K unique values in a facet field, the simplified formula
> #documents*log2(#references) + #references*log2(#unique_values) bit
> tells us that this takes at least 110MB with field cache based faceting.
>
> 180 clients @ 110MB ~= 20GB. As that is a theoretical low, we can at
> least double that. This fits neatly with your new heap of 64GB.
>
>
> If my guessing is correct, you can solve your memory problems very
> easily by sharing _all_ the facet fields between your clients.
> This should bring your memory usage down to a few GB.
>
> You are probably already restricting their searches to their own data by
> filtering, so this should not influence the returned facet values and
> counts, as compared to separate fields.
>
> This is very similar to the thread "Facets with 5000 facet fields" BTW.
>
> > Today I finally managed to set up a test core so I can begin to play
> > around with docValues.
>
> If you are using a single index with the individual-facet-fields for
> each client approach, the DocValues will also have scaling issues, as
> the amount of values (of which the majority will be null) will be
> #clients*#documents*#facet_fields
> This means that the adding a new client will be progressively more
> expensive.
>
> On the other hand, if you use a lot of small shards, DocValues should
> work for you.
>
> Regards,
> Toke Eskildsen
>
>
>
--
Med venlig hilsen / Best regards
*John Nielsen*
Programmer
*MCB A/S*
Enghaven 15
DK-7500 Holstebro
Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk
Re: Solr using a ridiculous amount of memory
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Mon, 2013-04-15 at 10:25 +0200, John Nielsen wrote:
> The FieldCache is the big culprit. We do a huge amount of faceting so
> it seems right.
Yes, you wrote that earlier. The mystery is that the math does not check
out with the description you have given us.
> Unfortunately I am super swamped at work so I have precious little
> time to work on this, which is what explains my silence.
No problem, we've all been there.
>
[Band aid: More memory]
> The extra memory helped a lot, but it still OOM with about 180 clients
> using it.
You stated earlier that you has a "solr cluster" and your total(?) index
size was 35GB, with each "register" being between "15k" and "30k". I am
using the quotes to signify that it is unclear what you mean. Is your
cluster multiple machines (I'm guessing no), multiple Solr's, cores,
shards or maybe just a single instance prepared for later distribution?
Is a register a core, shard or a simply logical part (one client's data)
of the index?
If each client has their own core or shard, that would mean that each
client uses more than 25GB/180 bytes ~= 142MB of heap to access 35GB/180
~= 200MB of index. That sounds quite high and you would need a very
heavy facet to reach that.
If you could grep "UnInverted" from the Solr log file and paste the
entries here, that would help to clarify things.
Another explanation for the large amount of memory presents itself if
you use a single index: If each of your clients facet on at least one
fields specific to the client ("client123_persons" or something like
that), then your memory usage goes through the roof.
Assuming an index with 10M documents, each with 5 references to a modest
10K unique values in a facet field, the simplified formula
#documents*log2(#references) + #references*log2(#unique_values) bit
tells us that this takes at least 110MB with field cache based faceting.
180 clients @ 110MB ~= 20GB. As that is a theoretical low, we can at
least double that. This fits neatly with your new heap of 64GB.
If my guessing is correct, you can solve your memory problems very
easily by sharing _all_ the facet fields between your clients.
This should bring your memory usage down to a few GB.
You are probably already restricting their searches to their own data by
filtering, so this should not influence the returned facet values and
counts, as compared to separate fields.
This is very similar to the thread "Facets with 5000 facet fields" BTW.
> Today I finally managed to set up a test core so I can begin to play
> around with docValues.
If you are using a single index with the individual-facet-fields for
each client approach, the DocValues will also have scaling issues, as
the amount of values (of which the majority will be null) will be
#clients*#documents*#facet_fields
This means that the adding a new client will be progressively more
expensive.
On the other hand, if you use a lot of small shards, DocValues should
work for you.
Regards,
Toke Eskildsen
Re: Solr using a ridiculous amount of memory
Posted by John Nielsen <jn...@mcb.dk>.
Yes and no,
The FieldCache is the big culprit. We do a huge amount of faceting so it
seems right. Unfortunately I am super swamped at work so I have precious
little time to work on this, which is what explains my silence.
Out of desperation, I added another 32G of memory to each server and
increased the JVM size to 64G from 25G. The servers are running with 96G
memory right now (this is the max amount supported by the hardware) which
leaves solr somewhat starved for memory. I am aware of the performance
implications of doing this but I have little choice.
The extra memory helped a lot, but it still OOM with about 180 clients
using it. Unfortunately I need to support at least double that. After
upgrading the RAM, I ran for almost two weeks with the same workload that
used to OOM a couple of times a day, so it doesn't look like a leak.
Today I finally managed to set up a test core so I can begin to play around
with docValues.
I actually have a couple of questions regarding docValues:
1) If I facet on multible fields and only some of those fields are using
docValues, will I still get the memory saving benefit of docValues? (one of
the facet fields use null values and will require a lot of work in our
product to fix)
2) If i just use docValues on one small core with very limited traffic at
first for testing purposes, how can I test that it is actually using the
disk for caching?
I really appreciate all the help I have received on this list so far. I do
feel confident that I will be able to solve this issue eventually.
On Mon, Apr 15, 2013 at 9:00 AM, Toke Eskildsen <te...@statsbiblioteket.dk>wrote:
> On Sun, 2013-03-24 at 09:19 +0100, John Nielsen wrote:
> > Our memory requirements are running amok. We have less than a quarter of
> > our customers running now and even though we have allocated 25GB to the
> JVM
> > already, we are still seeing daily OOM crashes.
>
> Out of curiosity: Did you manage to pinpoint the memory eater in your
> setup?
>
> - Toke Eskildsen
>
>
--
Med venlig hilsen / Best regards
*John Nielsen*
Programmer
*MCB A/S*
Enghaven 15
DK-7500 Holstebro
Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk
Re: Solr using a ridiculous amount of memory
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Sun, 2013-03-24 at 09:19 +0100, John Nielsen wrote:
> Our memory requirements are running amok. We have less than a quarter of
> our customers running now and even though we have allocated 25GB to the JVM
> already, we are still seeing daily OOM crashes.
Out of curiosity: Did you manage to pinpoint the memory eater in your
setup?
- Toke Eskildsen
RE: Solr using a ridiculous amount of memory
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
Toke Eskildsen [te@statsbiblioteket.dk]:
> If your whole index has 10M documents, which each has 100 values
> for each field, with each field having 50M unique values, then the
> memory requirement would be more than
> 10M*log2(100*10M) + 100*10M*log2(50M) bit ~= 340MB/field ~=
> 1.6GB for faceting on all fields.
Whoops. Missed a 0 when calculating. The case above would actually take more than 15GB, probably also more than the 25GB you have allocated.
Anyway, I see now in your solrconfig that your main facet fields are "cat", "manu_exact", "content_type" and "author_s", with the 5th being maybe "price", "popularity" or "manufacturedate_dt"?
cat seems like category (relatively few references, few uniques), content_type probably has a single value/item and again few uniques. No memory problem there, unless you have a lot of documents (100M-range). That leaves manu_exact and author_s. If those are freetext fields with item descriptions or similar, that might explain the OOM.
Could you describe the facet fields in more detail and provide us with the total document count?
Quick sanity check: If you are using a Linux server, could you please verify that your virtual memory is set to unlimited with 'ulimit -v'?
Regards,
Toke Eskildsen
RE: Solr using a ridiculous amount of memory
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
From: John Nielsen [jn@mcb.dk]:
> The index is about 35GB on disk with each register between 15k and 30k.
> (This is simply the size of a full xml reply of one register. I'm not sure
> how to measure it otherwise.)
> Our memory requirements are running amok. We have less than a quarter of
> our customers running now and even though we have allocated 25GB to the JVM
> already, we are still seeing daily OOM crashes.
That does sound a bit peculiar. I do not understand what you mean by "register" though. How many documents does your index holds?
> I can see from the memory dumps we've done that the field cache is by far
> the biggest sinner.
Do you sort on a lot of different fields?
> We do a lot of facetting. One client facets on about 50.000 docs of approx
> 30k each on 5 fields. I understand that this is VERY memory intensive.
To get a rough approximation of memory usage, we need the total number of documents, the average number of values for each of the 5 fields for a document and the number of unique values in each of the 5 fields. The rule of thumb I use for lower ceiling is
#documents*log2(#references) + #references*log2(#unique_values) bit
If your whole index has 10M documents, which each has 100 values for each field, with each field having 50M unique values, then the memory requirement would be more than 10M*log2(100*10M) + 100*10M*log2(50M) bit ~= 340MB/field ~= 1.6GB for faceting on all fields. Even when we multiply that with 4 to get a more real-world memory requirement, it is far from the 25GB that you are allocating. Either you have an interestingly high number somewhere in the equation or something's off.
Regards,
Toke Eskildsen
Re: Solr using a ridiculous amount of memory
Posted by Upayavira <uv...@odoko.co.uk>.
Might be obvious, but just in case - remember that you'll need to
re-index your content once you've added docValues to your schema, in
order to get the on-disk files to be created.
Upayavira
On Mon, Mar 25, 2013, at 03:16 PM, John Nielsen wrote:
> I apologize for the slow reply. Today has been killer. I will reply to
> everyone as soon as I get the time.
>
> I am having difficulties understanding how docValues work.
>
> Should I only add docValues to the fields that I actually use for sorting
> and faceting or on all fields?
>
> Will the docValues magic apply to the fields i activate docValues on or
> on
> the entire document when sorting/faceting on a field that has docValues
> activated?
>
> I'm not even sure which question to ask. I am struggling to understand
> this
> on a conceptual level.
>
>
> On Sun, Mar 24, 2013 at 7:11 PM, Robert Muir <rc...@gmail.com> wrote:
>
> > On Sun, Mar 24, 2013 at 4:19 AM, John Nielsen <jn...@mcb.dk> wrote:
> >
> > > Schema with DocValues attempt at solving problem:
> > > http://pastebin.com/Ne23NnW4
> > > Config: http://pastebin.com/x1qykyXW
> > >
> >
> > This schema isn't using docvalues, due to a typo in your config.
> > it should not be DocValues="true" but docValues="true".
> >
> > Are you not getting an error? Solr needs to throw exception if you
> > provide invalid attributes to the field. Nothing is more frustrating
> > than having a typo or something in your configuration and solr just
> > ignores this, reports no error, and "doesnt work the way you want".
> > I'll look into this (I already intend to add these checks to analysis
> > factories for the same reason).
> >
> > Separately, if you really want the terms data and so on to remain on
> > disk, it is not enough to "just enable docvalues" for the field. The
> > default implementation uses the heap. So if you want that, you need to
> > set docValuesFormat="Disk" on the fieldtype. This will keep the
> > majority of the data on disk, and only some key datastructures in heap
> > memory. This might have significant performance impact depending upon
> > what you are doing so you need to test that.
> >
>
>
>
> --
> Med venlig hilsen / Best regards
>
> *John Nielsen*
> Programmer
>
>
>
> *MCB A/S*
> Enghaven 15
> DK-7500 Holstebro
>
> Kundeservice: +45 9610 2824
> post@mcb.dk
> www.mcb.dk
Re: Solr using a ridiculous amount of memory
Posted by John Nielsen <jn...@mcb.dk>.
I apologize for the slow reply. Today has been killer. I will reply to
everyone as soon as I get the time.
I am having difficulties understanding how docValues work.
Should I only add docValues to the fields that I actually use for sorting
and faceting or on all fields?
Will the docValues magic apply to the fields i activate docValues on or on
the entire document when sorting/faceting on a field that has docValues
activated?
I'm not even sure which question to ask. I am struggling to understand this
on a conceptual level.
On Sun, Mar 24, 2013 at 7:11 PM, Robert Muir <rc...@gmail.com> wrote:
> On Sun, Mar 24, 2013 at 4:19 AM, John Nielsen <jn...@mcb.dk> wrote:
>
> > Schema with DocValues attempt at solving problem:
> > http://pastebin.com/Ne23NnW4
> > Config: http://pastebin.com/x1qykyXW
> >
>
> This schema isn't using docvalues, due to a typo in your config.
> it should not be DocValues="true" but docValues="true".
>
> Are you not getting an error? Solr needs to throw exception if you
> provide invalid attributes to the field. Nothing is more frustrating
> than having a typo or something in your configuration and solr just
> ignores this, reports no error, and "doesnt work the way you want".
> I'll look into this (I already intend to add these checks to analysis
> factories for the same reason).
>
> Separately, if you really want the terms data and so on to remain on
> disk, it is not enough to "just enable docvalues" for the field. The
> default implementation uses the heap. So if you want that, you need to
> set docValuesFormat="Disk" on the fieldtype. This will keep the
> majority of the data on disk, and only some key datastructures in heap
> memory. This might have significant performance impact depending upon
> what you are doing so you need to test that.
>
--
Med venlig hilsen / Best regards
*John Nielsen*
Programmer
*MCB A/S*
Enghaven 15
DK-7500 Holstebro
Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk
Re: Solr using a ridiculous amount of memory
Posted by Robert Muir <rc...@gmail.com>.
On Sun, Mar 24, 2013 at 4:19 AM, John Nielsen <jn...@mcb.dk> wrote:
> Schema with DocValues attempt at solving problem:
> http://pastebin.com/Ne23NnW4
> Config: http://pastebin.com/x1qykyXW
>
This schema isn't using docvalues, due to a typo in your config.
it should not be DocValues="true" but docValues="true".
Are you not getting an error? Solr needs to throw exception if you
provide invalid attributes to the field. Nothing is more frustrating
than having a typo or something in your configuration and solr just
ignores this, reports no error, and "doesnt work the way you want".
I'll look into this (I already intend to add these checks to analysis
factories for the same reason).
Separately, if you really want the terms data and so on to remain on
disk, it is not enough to "just enable docvalues" for the field. The
default implementation uses the heap. So if you want that, you need to
set docValuesFormat="Disk" on the fieldtype. This will keep the
majority of the data on disk, and only some key datastructures in heap
memory. This might have significant performance impact depending upon
what you are doing so you need to test that.
Re: Solr using a ridiculous amount of memory
Posted by Jack Krupansky <ja...@basetechnology.com>.
A step I meant to include was that after you "warm" Solr with a
representative collection of queries that references all of the fields,
facets, sorting, etc. that your daily load will reference, check the Java
heap size at that point, and then set your Java heap limit to a moderate
level higher, like 256M, restart, and then see what happens.
The theory is that if you have too much available heap, Java will gradually
fill it all with garbage (no leaks implied, but maybe some leaks as well),
and then a Java GC will be an expensive hit, and sometimes a rapid flow of
incoming requests at that point can cause Java to freak out and even hit OOM
even though a more graceful garbage collection would eventually free up tons
of garbage.
So, by only allowing for a moderate amount of garbage, more frequent GCs
will be less intensive and less likely to cause weird situations.
The other part of the theory is that it is usually better to leave tons of
memory to the OS for efficiently caching files, rather than force Java to
manage large amounts of memory, which it typically does not do so well.
-- Jack Krupansky
-----Original Message-----
From: Jack Krupansky
Sent: Sunday, March 24, 2013 2:00 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr using a ridiculous amount of memory
Just to get started, do you hit OOM quickly with a few expensive queries, or
is it after a number of hours and lots of queries?
Does Java heap usage seem to be growing linearly as queries come in, or are
there big spikes?
How complex/rich are your queries (e.g., how many terms, wildcards, faceted
fields, sorting, etc.)?
As a baseline experiment, start a Solr server, see how much Java heap is
used/available. Then do a couple of typical queries, and check the heap size
again. Then do a couple more similar but different (to avoid query cache
matches), and check the heap again. Maybe do that a few times to get a
handle on the baseline memory required and whether there might be a leak of
some sort. Do enough queries to hits all of the fields, facets, sorting,
etc. that are likely to be encountered in one of your typical days that hits
OOM - just not the volume of queries. The goal is to determine if there is
something inherently memory intensive in your index/queries, or something
relating to a leak based on total query volume.
-- Jack Krupansky
-----Original Message-----
From: John Nielsen
Sent: Sunday, March 24, 2013 4:19 AM
To: solr-user@lucene.apache.org
Subject: Solr using a ridiculous amount of memory
Hello all,
We are running a solr cluster which is now running solr-4.2.
The index is about 35GB on disk with each register between 15k and 30k.
(This is simply the size of a full xml reply of one register. I'm not sure
how to measure it otherwise.)
Our memory requirements are running amok. We have less than a quarter of
our customers running now and even though we have allocated 25GB to the JVM
already, we are still seeing daily OOM crashes. We used to just allocate
more memory to the JVM, but with the way solr is scaling, we would need
well over 100GB of memory on each node to finish the project, and thats
just not going to happen. I need to lower the memory requirements somehow.
I can see from the memory dumps we've done that the field cache is by far
the biggest sinner. Of special interest to me is the recent introduction of
DocValues which supposedly mitigates this issue by using memory outside the
JVM. I just can't, because of lack of documentation, seem to make it work.
We do a lot of facetting. One client facets on about 50.000 docs of approx
30k each on 5 fields. I understand that this is VERY memory intensive.
Schema with DocValues attempt at solving problem:
http://pastebin.com/Ne23NnW4
Config: http://pastebin.com/x1qykyXW
The cache is pretty well tuned. Any lower and i get evictions.
Come hell or high water, my JVM memory requirements must come down. Simply
moving some memory load outside of the JVM would be awesome! Making it not
use the field cache for anything would also (probably) work for me. I
thought about killing off my other caches, but from the dumps, they just
don't seem to use that much memory.
I am at my wits end. Any help would be sorely appreciated.
--
Med venlig hilsen / Best regards
*John Nielsen*
Programmer
*MCB A/S*
Enghaven 15
DK-7500 Holstebro
Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk
Re: Solr using a ridiculous amount of memory
Posted by Jack Krupansky <ja...@basetechnology.com>.
Just to get started, do you hit OOM quickly with a few expensive queries, or
is it after a number of hours and lots of queries?
Does Java heap usage seem to be growing linearly as queries come in, or are
there big spikes?
How complex/rich are your queries (e.g., how many terms, wildcards, faceted
fields, sorting, etc.)?
As a baseline experiment, start a Solr server, see how much Java heap is
used/available. Then do a couple of typical queries, and check the heap size
again. Then do a couple more similar but different (to avoid query cache
matches), and check the heap again. Maybe do that a few times to get a
handle on the baseline memory required and whether there might be a leak of
some sort. Do enough queries to hits all of the fields, facets, sorting,
etc. that are likely to be encountered in one of your typical days that hits
OOM - just not the volume of queries. The goal is to determine if there is
something inherently memory intensive in your index/queries, or something
relating to a leak based on total query volume.
-- Jack Krupansky
-----Original Message-----
From: John Nielsen
Sent: Sunday, March 24, 2013 4:19 AM
To: solr-user@lucene.apache.org
Subject: Solr using a ridiculous amount of memory
Hello all,
We are running a solr cluster which is now running solr-4.2.
The index is about 35GB on disk with each register between 15k and 30k.
(This is simply the size of a full xml reply of one register. I'm not sure
how to measure it otherwise.)
Our memory requirements are running amok. We have less than a quarter of
our customers running now and even though we have allocated 25GB to the JVM
already, we are still seeing daily OOM crashes. We used to just allocate
more memory to the JVM, but with the way solr is scaling, we would need
well over 100GB of memory on each node to finish the project, and thats
just not going to happen. I need to lower the memory requirements somehow.
I can see from the memory dumps we've done that the field cache is by far
the biggest sinner. Of special interest to me is the recent introduction of
DocValues which supposedly mitigates this issue by using memory outside the
JVM. I just can't, because of lack of documentation, seem to make it work.
We do a lot of facetting. One client facets on about 50.000 docs of approx
30k each on 5 fields. I understand that this is VERY memory intensive.
Schema with DocValues attempt at solving problem:
http://pastebin.com/Ne23NnW4
Config: http://pastebin.com/x1qykyXW
The cache is pretty well tuned. Any lower and i get evictions.
Come hell or high water, my JVM memory requirements must come down. Simply
moving some memory load outside of the JVM would be awesome! Making it not
use the field cache for anything would also (probably) work for me. I
thought about killing off my other caches, but from the dumps, they just
don't seem to use that much memory.
I am at my wits end. Any help would be sorely appreciated.
--
Med venlig hilsen / Best regards
*John Nielsen*
Programmer
*MCB A/S*
Enghaven 15
DK-7500 Holstebro
Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk