You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Hendrik Haddorp <he...@gmx.net> on 2018/03/18 16:22:30 UTC
collection reload leads to OutOfMemoryError
Hi,
I did a simple test on a three node cluster using Solr 7.2.1. The JVMs
(Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0_162
25.162-b12) have about 6.5GB heap and 1.5GB metaspace. In my test I have
1000 collections with only 1000 simple documents each. I'm then
triggering collections reloads via SolrJ using a fixed number of
threads, as this has shown memory issues in the past. Even with two
threads the nodes eventually die with an OOM Error as they are running
out of metaspace. I found the following Jiras that might be about the
same issue:
https://issues.apache.org/jira/browse/SOLR-10506
https://issues.apache.org/jira/browse/SOLR-9117
https://issues.apache.org/jira/browse/SOLR-6678
The first two are flagged as fixed in 7.0.
Any ideas, beside not doing reloads?
regards,
Hendrik
Re: collection reload leads to OutOfMemoryError
Posted by Erick Erickson <er...@gmail.com>.
Hendrik:
There are a limited number of threads that load in parallel when
starting up, depends on the configuration. The defaults are 3 threads
in stand-alone and 8 in Cloud (see: NodeConfig.java)
public static final int DEFAULT_CORE_LOAD_THREADS = 3;
public static final int DEFAULT_CORE_LOAD_THREADS_IN_CLOUD = 8;
Not sure whether that's germane, but thought it worth mentioning just in case.
Best,
Erick
On Sun, Mar 18, 2018 at 1:11 PM, Hendrik Haddorp
<he...@gmx.net> wrote:
> I increased the metaspace size to 2GB. This way I could do multiple rounds
> of reloading all collections already. The GC logs do show now an almost
> stable metaspace size. So maybe I did just set the limits too low. Still a
> bit odd that reloading the collections results in a higher memory usage.
> Shouldn't all collections be loaded during the startup?
>
> On 18.03.2018 17:22, Hendrik Haddorp wrote:
>>
>> Hi,
>>
>> I did a simple test on a three node cluster using Solr 7.2.1. The JVMs
>> (Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0_162 25.162-b12)
>> have about 6.5GB heap and 1.5GB metaspace. In my test I have 1000
>> collections with only 1000 simple documents each. I'm then triggering
>> collections reloads via SolrJ using a fixed number of threads, as this has
>> shown memory issues in the past. Even with two threads the nodes eventually
>> die with an OOM Error as they are running out of metaspace. I found the
>> following Jiras that might be about the same issue:
>> https://issues.apache.org/jira/browse/SOLR-10506
>> https://issues.apache.org/jira/browse/SOLR-9117
>> https://issues.apache.org/jira/browse/SOLR-6678
>>
>> The first two are flagged as fixed in 7.0.
>>
>> Any ideas, beside not doing reloads?
>>
>> regards,
>> Hendrik
>
>
Re: collection reload leads to OutOfMemoryError
Posted by Hendrik Haddorp <he...@gmx.net>.
I increased the metaspace size to 2GB. This way I could do multiple
rounds of reloading all collections already. The GC logs do show now an
almost stable metaspace size. So maybe I did just set the limits too
low. Still a bit odd that reloading the collections results in a higher
memory usage. Shouldn't all collections be loaded during the startup?
On 18.03.2018 17:22, Hendrik Haddorp wrote:
> Hi,
>
> I did a simple test on a three node cluster using Solr 7.2.1. The JVMs
> (Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0_162
> 25.162-b12) have about 6.5GB heap and 1.5GB metaspace. In my test I
> have 1000 collections with only 1000 simple documents each. I'm then
> triggering collections reloads via SolrJ using a fixed number of
> threads, as this has shown memory issues in the past. Even with two
> threads the nodes eventually die with an OOM Error as they are running
> out of metaspace. I found the following Jiras that might be about the
> same issue:
> https://issues.apache.org/jira/browse/SOLR-10506
> https://issues.apache.org/jira/browse/SOLR-9117
> https://issues.apache.org/jira/browse/SOLR-6678
>
> The first two are flagged as fixed in 7.0.
>
> Any ideas, beside not doing reloads?
>
> regards,
> Hendrik