You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Vignan Malyala <ds...@gmail.com> on 2019/10/23 10:09:01 UTC
Solr Prod stopped yesterday - saya "insufficient memory for the Java
Runtime Environment"
*Solr prod stopped yesterday. How to prevent this.*
Solr heap info is : -Xms20g -Xmx20g
JVM Heap info. : -Xms528m -Xmx8g
Physical Ram - 32GB
Solr version - 6.6.1
Swap memory - 8g
*hc_err_pid.log got created with following info in it:*
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 16106127360 bytes for
committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# The process is running with CompressedOops enabled, and the Java Heap
may be blocking the growth of the native heap
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2749), pid=86291,
tid=0x00007f8822e47700
#
# JRE version: (8.0_211-b12) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.211-b12 mixed mode
linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core
dumping, try "ulimit -c unlimited" before starting Java again
#
*solr_gc.log shows following:*
2019-10-23T01:24:23.688+0530: 644433.457: Total time for which application
threads were stopped: 0.0028257 seconds, Stopping threads took: 0.0016863
seconds
Heap
par new generation total 4369088K, used 736363K [0x00000002c0000000,
0x0000000400000000, 0x0000000400000000)
eden space 3495296K, 13% used [0x00000002c0000000, 0x00000002dc9d5ed8,
0x0000000395560000)
from space 873792K, 30% used [0x00000003caab0000, 0x00000003daff4d80,
0x0000000400000000)
to space 873792K, 0% used [0x0000000395560000, 0x0000000395560000,
0x00000003caab0000)
concurrent mark-sweep generation total 15728640K, used 3807001K
[0x0000000400000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 45325K, capacity 47047K, committed 47324K, reserved
1091584K
class space used 4821K, capacity 5230K, committed 5340K, reserved
1048576K
Re: Solr Prod stopped yesterday - saya "insufficient memory for the
Java Runtime Environment"
Posted by Paras Lehana <pa...@indiamart.com>.
Hi Vignan,
I see this setting quite strange.
Also, if this is the case, you have allocated Solr more memory even than
maximum allowed for total JVM. Solr heap is a subset of JVM heap - do note
that!
On Wed, 23 Oct 2019 at 16:41, Vincenzo D'Amore <v....@gmail.com> wrote:
> Hi,
>
> I see this setting quite strange:
>
> Solr heap info is : -Xms20g -Xmx20g
> JVM Heap info. : -Xms528m -Xmx8g
>
> “Usually” Solr runs inside the jvm and you can have only one of these
> settings really active. I suggest to double check your memory
> configuration.
>
> Ciao,
> Vincenzo
>
> --
> skype: free.dev
>
> > On 23 Oct 2019, at 12:16, Vignan Malyala <ds...@gmail.com> wrote:
> >
> > Solr heap info is : -Xms20g -Xmx20g
> > JVM Heap info. : -Xms528m -Xmx8g
>
--
--
Regards,
*Paras Lehana* [65871]
Development Engineer, Auto-Suggest,
IndiaMART Intermesh Ltd.
8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303
Mob.: +91-9560911996
Work: 01203916600 | Extn: *8173*
--
IMPORTANT:
NEVER share your IndiaMART OTP/ Password with anyone.
Re: Solr Prod stopped yesterday - saya "insufficient memory for the Java Runtime Environment"
Posted by Vincenzo D'Amore <v....@gmail.com>.
Hi,
I see this setting quite strange:
Solr heap info is : -Xms20g -Xmx20g
JVM Heap info. : -Xms528m -Xmx8g
“Usually” Solr runs inside the jvm and you can have only one of these settings really active. I suggest to double check your memory configuration.
Ciao,
Vincenzo
--
skype: free.dev
> On 23 Oct 2019, at 12:16, Vignan Malyala <ds...@gmail.com> wrote:
>
> Solr heap info is : -Xms20g -Xmx20g
> JVM Heap info. : -Xms528m -Xmx8g
Re: Solr Prod stopped yesterday - saya "insufficient memory for the
Java Runtime Environment"
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/23/2019 9:08 AM, Vignan Malyala wrote:
> Ok. I have around 500 cores in my solr. So, how much heap I should allocate
> in solr and jvm?
> (Currently as I see, in solr.in.sh shows heap as - Xms 20g -Xmx 20g.
> And my system jvm heap shows -Xms 528m -Xmx 8g. I've re-checked it.)
We have no way of knowing how big a heap you need.
https://lucidworks.com/post/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
We can make an educated guess with certain information that has not been
provided, but it would only be a guess, and might be wrong.
As I said before, there is only one heap for a Java program. There is
no distinction between Java heap, Solr heap, system heap, or a heap with
any other name. If you have multiple entries in the commandline for
-Xmx or -Xms, only one of them is going to take effect, and I do not
know which one.
The error message you're getting in Java's error file means that there
wasn't enough memory available for Java to allocate what it wanted to
allocate, so Java crashed. This is a problem at the Java level, Solr is
not involved.
Thanks,
Shawn
Re: Solr Prod stopped yesterday - saya "insufficient memory for the
Java Runtime Environment"
Posted by Vignan Malyala <ds...@gmail.com>.
Ok. I have around 500 cores in my solr. So, how much heap I should allocate
in solr and jvm?
(Currently as I see, in solr.in.sh shows heap as - Xms 20g -Xmx 20g.
And my system jvm heap shows -Xms 528m -Xmx 8g. I've re-checked it.)
On Wed 23 Oct, 2019, 7:52 PM Shawn Heisey, <ap...@elyograg.org> wrote:
> On 10/23/2019 4:09 AM, Vignan Malyala wrote:
> > *Solr prod stopped yesterday. How to prevent this.*
> >
> > Solr heap info is : -Xms20g -Xmx20g
> > JVM Heap info. : -Xms528m -Xmx8g
>
> There is no such thing as a Solr heap separate from the JVM heap. There
> are multiple environment variables that can specify the heap size ...
> only one of those settings is actually going to take effect. I have not
> done any investigation to determine which one it will be.
>
> > Physical Ram - 32GB
> > Solr version - 6.6.1
> > Swap memory - 8g
> >
> > *hc_err_pid.log got created with following info in it:*
> > #
> > # There is insufficient memory for the Java Runtime Environment to
> continue.
> > # Native memory allocation (mmap) failed to map 16106127360 bytes for
> > committing reserved memory.
>
> This sounds like there is insufficient memory available when running
> Solr for the system to start Java with the configured settings. Based
> on this number, which is about 16GB, I'm betting that the heap size
> which took effect is the 20GB one, or maybe it got set to 16GB by
> another setting that you did not mention above.
>
> Your information says that there is 32GB total memory ... maybe there
> are other programs that are using up some of that memory before Solr
> attempts to start, and there is not enough memory left for Solr.
>
> Thanks,
> Shawn
>
Re: Solr Prod stopped yesterday - saya "insufficient memory for the
Java Runtime Environment"
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/23/2019 4:09 AM, Vignan Malyala wrote:
> *Solr prod stopped yesterday. How to prevent this.*
>
> Solr heap info is : -Xms20g -Xmx20g
> JVM Heap info. : -Xms528m -Xmx8g
There is no such thing as a Solr heap separate from the JVM heap. There
are multiple environment variables that can specify the heap size ...
only one of those settings is actually going to take effect. I have not
done any investigation to determine which one it will be.
> Physical Ram - 32GB
> Solr version - 6.6.1
> Swap memory - 8g
>
> *hc_err_pid.log got created with following info in it:*
> #
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (mmap) failed to map 16106127360 bytes for
> committing reserved memory.
This sounds like there is insufficient memory available when running
Solr for the system to start Java with the configured settings. Based
on this number, which is about 16GB, I'm betting that the heap size
which took effect is the 20GB one, or maybe it got set to 16GB by
another setting that you did not mention above.
Your information says that there is 32GB total memory ... maybe there
are other programs that are using up some of that memory before Solr
attempts to start, and there is not enough memory left for Solr.
Thanks,
Shawn