You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Altamirano, Emmanuel" <Em...@transunion.com.INVALID> on 2020/12/11 03:55:23 UTC
SolrCloud crashing due to memory error - 'Cannot allocate memory'
(errno=12)
Hello,
We have a SolrCloud(8.6) with 3 servers with the same characteristics and configuration. We assigned 32GB for heap memory each, and after some short period of time sending 40 concurrent requests to the SolrCloud using a load balancer, we are getting the following error that shutdown each Solr Server and Zookeeper:
OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 536870912 (errno = 12).
OpenJDK 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007edd4d9da000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
20201201 10:43:29.495 [ERROR] {qtp2051853139-23369} [c:express s:shard1 r:core_node6 x:express_shard1_replica_n4] [org.apache.solr.handler.RequestHandlerBase, 148] | org.apache.solr.common.SolrException: Cannot talk to ZooKeeper - Updates are disabled.
at org.apache.solr.update.processor.DistributedZkUpdateProcessor.zkCheck(DistributedZkUpdateProcessor.java:1245)
at org.apache.solr.update.processor.DistributedZkUpdateProcessor.setupRequest(DistributedZkUpdateProcessor.java:582)
at org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:239)
[cid:image004.jpg@01D6CF3F.27574B90]
We have a one collection with one shard, almost 400 million documents (~334GB).
$ sysctl vm.nr_hugepages
vm.nr_hugepages = 32768
$ sysctl vm.max_map_count
vm.max_map_count = 131072
/etc/security/limits.conf
* - core unlimited
* - data unlimited
* - priority unlimited
* - fsize unlimited
* - sigpending 513928
* - memlock unlimited
* - nofile 131072
* - msgqueue 819200
* - rtprio 0
* - stack 8192
* - cpu unlimited
* - rss unlimited #virtual memory unlimited
* - locks unlimited
* soft nproc 65536
* hard nproc 65536
* - nofile 131072
/etc/sysctl.conf
vm.nr_hugepages = 32768
vm.max_map_count = 131072
Could you please provide me some advice to fix this error?
Thanks,
Emmanuel Altamirano
Re: SolrCloud crashing due to memory error - 'Cannot allocate memory'
(errno=12)
Posted by Walter Underwood <wu...@wunderwood.org>.
How much RAM do you have on those machines? That message says you ran out.
32 GB is a HUGE heap. Unless you have a specific need for that, run with a 8 GB
heap and see how that works.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Dec 10, 2020, at 7:55 PM, Altamirano, Emmanuel <Em...@transunion.com.INVALID> wrote:
>
> Hello,
>
> We have a SolrCloud(8.6) with 3 servers with the same characteristics and configuration. We assigned32GB for heap memory each, and after some short period of time sending 40 concurrent requests to the SolrCloud using a load balancer, we are getting the following error that shutdown each Solr Server and Zookeeper:
>
> OpenJDK 64-Bit Server VM warning: Failed to reserve large pages memory req_addr: 0x0000000000000000 bytes: 536870912 (errno = 12).
> OpenJDK 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
> OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007edd4d9da000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
>
>
> 20201201 10:43:29.495 [ERROR] {qtp2051853139-23369} [c:express s:shard1 r:core_node6 x:express_shard1_replica_n4] [org.apache.solr.handler.RequestHandlerBase, 148] | org.apache.solr.common.SolrException: Cannot talk to ZooKeeper - Updates are disabled.
> at org.apache.solr.update.processor.DistributedZkUpdateProcessor.zkCheck(DistributedZkUpdateProcessor.java:1245)
> at org.apache.solr.update.processor.DistributedZkUpdateProcessor.setupRequest(DistributedZkUpdateProcessor.java:582)
> at org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:239)
>
>
>
> We have a one collection with one shard, almost 400 million documents (~334GB).
>
> $ sysctl vm.nr_hugepages
> vm.nr_hugepages = 32768
> $ sysctl vm.max_map_count
> vm.max_map_count = 131072
>
> /etc/security/limits.conf
>
> * - core unlimited
> * - data unlimited
> * - priority unlimited
> * - fsize unlimited
> * - sigpending 513928
> * - memlock unlimited
> * - nofile 131072
> * - msgqueue 819200
> * - rtprio 0
> * - stack 8192
> * - cpu unlimited
> * - rss unlimited #virtual memory unlimited
> * - locks unlimited
> * soft nproc 65536
> * hard nproc 65536
> * - nofile 131072
>
>
>
> /etc/sysctl.conf
>
> vm.nr_hugepages = 32768
> vm.max_map_count = 131072
>
>
> Could you please provide me some advice to fix this error?
>
> Thanks,
>
> Emmanuel Altamirano