You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Walter Underwood <wu...@wunderwood.org> on 2021/03/29 18:33:53 UTC
Running out of 16GB heap during indexing
We have a cluster that is running out of heap during indexing. Our older cluster does not have this problem. Both clusters have 8 shards and 3 replica sets, for a total of 24 hosts. Both use the same G1GC settings. Both have the same indexing feed of 57 million documents.
Old cluster:
Solr 6.6.2
11 GB heap
Oracle Corporation OpenJDK 64-Bit Server VM 1.8.0_131 25.131-b11
New cluster:
Solr 8.7.0
16 GB heap
Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.10 11.0.10+9-LTS
Cache sizes on the new cluster are pretty standard, filter and query result caches are 512, document cache is 2048. Documents aren’t huge, maybe average size a few kbytes. They are homework problems.
Any ideas? The 6.6.2 cluster has been running for a few years with no indexing problems.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
Re: Running out of 16GB heap during indexing
Posted by Jan Høydahl <ja...@cominvent.com>.
SOLR-14013 is only an issue in 7.7.1->8.3.x, so 6.x and newer 8.x should be fine.
Have you analyzed the OOM to see which part of the indexing process that was killing the heap?
Jan
> 29. mar. 2021 kl. 22:18 skrev Walter Underwood <wu...@wunderwood.org>:
>
> Does this fix need changes on both the client and server side of SolrJ? Because of some older clusters, we are using SolrJ 4.4.10. We can upgrade to 6.6.2, but that predates this fix.
>
> https://issues.apache.org/jira/browse/SOLR-14013
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>> On Mar 29, 2021, at 11:33 AM, Walter Underwood <wu...@wunderwood.org> wrote:
>>
>> We have a cluster that is running out of heap during indexing. Our older cluster does not have this problem. Both clusters have 8 shards and 3 replica sets, for a total of 24 hosts. Both use the same G1GC settings. Both have the same indexing feed of 57 million documents.
>>
>> Old cluster:
>> Solr 6.6.2
>> 11 GB heap
>> Oracle Corporation OpenJDK 64-Bit Server VM 1.8.0_131 25.131-b11
>>
>> New cluster:
>> Solr 8.7.0
>> 16 GB heap
>> Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.10 11.0.10+9-LTS
>>
>> Cache sizes on the new cluster are pretty standard, filter and query result caches are 512, document cache is 2048. Documents aren’t huge, maybe average size a few kbytes. They are homework problems.
>>
>> Any ideas? The 6.6.2 cluster has been running for a few years with no indexing problems.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>
Re: Running out of 16GB heap during indexing
Posted by Walter Underwood <wu...@wunderwood.org>.
Any suggestions?
Looks like we are ditching SolrJ anyway, but it would be nice to have a bit more info.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Mar 29, 2021, at 1:18 PM, Walter Underwood <wu...@wunderwood.org> wrote:
>
> Does this fix need changes on both the client and server side of SolrJ? Because of some older clusters, we are using SolrJ 4.4.10. We can upgrade to 6.6.2, but that predates this fix.
>
> https://issues.apache.org/jira/browse/SOLR-14013
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>> On Mar 29, 2021, at 11:33 AM, Walter Underwood <wu...@wunderwood.org> wrote:
>>
>> We have a cluster that is running out of heap during indexing. Our older cluster does not have this problem. Both clusters have 8 shards and 3 replica sets, for a total of 24 hosts. Both use the same G1GC settings. Both have the same indexing feed of 57 million documents.
>>
>> Old cluster:
>> Solr 6.6.2
>> 11 GB heap
>> Oracle Corporation OpenJDK 64-Bit Server VM 1.8.0_131 25.131-b11
>>
>> New cluster:
>> Solr 8.7.0
>> 16 GB heap
>> Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.10 11.0.10+9-LTS
>>
>> Cache sizes on the new cluster are pretty standard, filter and query result caches are 512, document cache is 2048. Documents aren’t huge, maybe average size a few kbytes. They are homework problems.
>>
>> Any ideas? The 6.6.2 cluster has been running for a few years with no indexing problems.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>
Re: Running out of 16GB heap during indexing
Posted by Walter Underwood <wu...@wunderwood.org>.
Does this fix need changes on both the client and server side of SolrJ? Because of some older clusters, we are using SolrJ 4.4.10. We can upgrade to 6.6.2, but that predates this fix.
https://issues.apache.org/jira/browse/SOLR-14013
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Mar 29, 2021, at 11:33 AM, Walter Underwood <wu...@wunderwood.org> wrote:
>
> We have a cluster that is running out of heap during indexing. Our older cluster does not have this problem. Both clusters have 8 shards and 3 replica sets, for a total of 24 hosts. Both use the same G1GC settings. Both have the same indexing feed of 57 million documents.
>
> Old cluster:
> Solr 6.6.2
> 11 GB heap
> Oracle Corporation OpenJDK 64-Bit Server VM 1.8.0_131 25.131-b11
>
> New cluster:
> Solr 8.7.0
> 16 GB heap
> Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.10 11.0.10+9-LTS
>
> Cache sizes on the new cluster are pretty standard, filter and query result caches are 512, document cache is 2048. Documents aren’t huge, maybe average size a few kbytes. They are homework problems.
>
> Any ideas? The 6.6.2 cluster has been running for a few years with no indexing problems.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>