You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Luke Oak <lu...@gmail.com> on 2021/01/23 13:29:42 UTC
Cannot start solr because oom
Hi there,
I use default settings to start solr , I set heap to 6G, I created 10 collections with 1node and 1 replica, however, there is not much data at all, just 100 documents.
My server is 32 G memory and 4 core cpu, ssd drive 300g
It was ok when i created 5 collections. It got oom killed when 10 collections are created. Please, no data in new collections.
Also I found that solr doesn’t do garbage collection when the 6G is used ( from dashboard, jvm usage is reached 6 g)
Any idea or setting needs to be done?
Thanks
Derrick
Sent from my iPhone
Re: Cannot start solr because oom
Posted by Luke <lu...@gmail.com>.
SinceI changed heap size to 10G, I found that solr always uses around
6G-6.5G. Just wondering where I can set to limit memory usage, for example,
I just want to give solr 6G.
On Sun, Jan 24, 2021 at 1:51 PM Luke <lu...@gmail.com> wrote:
> looks like the solr-8983-console.log was overridden after I restarted Solr
> with 10G memory, I cannot find it anymore.
>
> as for how I install and start solr, I did as below
> 1. download binary file(8.7.0)
> 2. change configuration in solr.in.sh(setup external zk)
> 3. start it by ./bin/solr start &
>
> Thanks
> Derrick
>
> On Sun, Jan 24, 2021 at 2:26 AM Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 1/23/2021 6:41 PM, Luke wrote:
>> > I don't see any log in solr.log, but there is OutOfMemory error in
>> > solr-8983-console.log file.
>>
>> Do you have the entire text of that exception? Can you share it? That
>> is the real information that I am after here.
>>
>> I only asked how Solr was installed and started so I would be able to
>> help you figure out where the log files are, if that became necessary.
>> It seems that you know where they are.
>>
>> Thanks,
>> Shawn
>>
>
Re: Cannot start solr because oom
Posted by Luke <lu...@gmail.com>.
looks like the solr-8983-console.log was overridden after I restarted Solr
with 10G memory, I cannot find it anymore.
as for how I install and start solr, I did as below
1. download binary file(8.7.0)
2. change configuration in solr.in.sh(setup external zk)
3. start it by ./bin/solr start &
Thanks
Derrick
On Sun, Jan 24, 2021 at 2:26 AM Shawn Heisey <ap...@elyograg.org> wrote:
> On 1/23/2021 6:41 PM, Luke wrote:
> > I don't see any log in solr.log, but there is OutOfMemory error in
> > solr-8983-console.log file.
>
> Do you have the entire text of that exception? Can you share it? That
> is the real information that I am after here.
>
> I only asked how Solr was installed and started so I would be able to
> help you figure out where the log files are, if that became necessary.
> It seems that you know where they are.
>
> Thanks,
> Shawn
>
Re: Cannot start solr because oom
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/23/2021 6:41 PM, Luke wrote:
> I don't see any log in solr.log, but there is OutOfMemory error in
> solr-8983-console.log file.
Do you have the entire text of that exception? Can you share it? That
is the real information that I am after here.
I only asked how Solr was installed and started so I would be able to
help you figure out where the log files are, if that became necessary.
It seems that you know where they are.
Thanks,
Shawn
Re: Cannot start solr because oom
Posted by Luke <lu...@gmail.com>.
Shawn,
What version of Solr? 8.7.0
How is it installed and started? I download binary file and change
configuration in solr.in.sh, then start it by ./bin/solr start &
What OS? centos 7
Java version? openJDK 8
I don't see any log in solr.log, but there is OutOfMemory error in
solr-8983-console.log file.
I changed the heap to 10G now, it's ok now, but I am wondering why solr
uses 6.4G memory when there is no data at all. there are just 10 empty
collections.
also the way to start solr is matter?
>>>> the below is from my gc log, can anything be optimized?
{Heap before GC invocations=55 (full 0):
garbage-first heap total 10485760K, used 9360694K [0x0000000540000000,
0x0000000540405000, 0x00000007c0000000)
region size 4096K, 704 young (2883584K), 2 survivors (8192K)
Metaspace used 62884K, capacity 66378K, committed 66844K, reserved
1107968K
class space used 7238K, capacity 8128K, committed 8220K, reserved
1048576K
2021-01-24T09:17:17.467+0800: 39708.837: [GC pause (G1 Evacuation Pause)
(young) (initial-mark)
Desired survivor size 184549376 bytes, new threshold 15 (max 15)
- age 1: 860160 bytes, 860160 total
- age 2: 25376 bytes, 885536 total
- age 3: 293920 bytes, 1179456 total
- age 4: 1498520 bytes, 2677976 total
, 0.0078650 secs]
[Parallel Time: 5.6 ms, GC Workers: 4]
[GC Worker Start (ms): Min: 39708837.9, Avg: 39708837.9, Max:
39708838.0, Diff: 0.0]
[Ext Root Scanning (ms): Min: 1.5, Avg: 1.6, Max: 1.9, Diff: 0.4,
Sum: 6.6]
[Update RS (ms): Min: 0.7, Avg: 0.8, Max: 0.9, Diff: 0.2, Sum: 3.3]
[Processed Buffers: Min: 5, Avg: 10.8, Max: 25, Diff: 20, Sum: 43]
[Scan RS (ms): Min: 0.1, Avg: 0.2, Max: 0.2, Diff: 0.1, Sum: 0.8]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.9, Diff: 1.9,
Sum: 2.0]
[Object Copy (ms): Min: 0.7, Avg: 2.2, Max: 2.7, Diff: 2.0, Sum: 8.6]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Termination Attempts: Min: 2, Avg: 3.2, Max: 5, Diff: 3, Sum: 13]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum:
0.2]
[GC Worker Total (ms): Min: 5.3, Avg: 5.4, Max: 5.4, Diff: 0.0, Sum:
21.4]
[GC Worker End (ms): Min: 39708843.3, Avg: 39708843.3, Max:
39708843.3, Diff: 0.0]
[Code Root Fixup: 0.0 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.4 ms]
[Other: 1.9 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.8 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.0 ms]
[Humongous Register: 0.0 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 0.6 ms]
[Eden: 2808.0M(2808.0M)->0.0B(2824.0M) Survivors: 8192.0K->8192.0K Heap:
9141.3M(10240.0M)->6333.1M(10240.0M)]
Heap after GC invocations=56 (full 0):
garbage-first heap total 10485760K, used 6485044K [0x0000000540000000,
0x0000000540405000, 0x00000007c0000000)
region size 4096K, 2 young (8192K), 2 survivors (8192K)
Metaspace used 62884K, capacity 66378K, committed 66844K, reserved
1107968K
class space used 7238K, capacity 8128K, committed 8220K, reserved
1048576K
}
[Times: user=0.03 sys=0.00, real=0.01 secs]
2021-01-24T09:17:17.475+0800: 39708.845: [GC
concurrent-root-region-scan-start]
2021-01-24T09:17:17.475+0800: 39708.846: Total time for which application
threads were stopped: 0.0088115 seconds, Stopping threads took: 0.0000670
seconds
2021-01-24T09:17:17.481+0800: 39708.851: [GC
concurrent-root-region-scan-end, 0.0057470 secs]
2021-01-24T09:17:17.481+0800: 39708.851: [GC concurrent-mark-start]
2021-01-24T09:17:32.665+0800: 39724.035: [GC concurrent-mark-end,
15.1839300 secs]
2021-01-24T09:17:32.665+0800: 39724.036: [GC remark
2021-01-24T09:17:32.665+0800: 39724.036: [Finalize Marking, 0.0002351 secs]
2021-01-24T09:17:32.666+0800: 39724.036: [GC ref-proc, 0.0018365 secs]
2021-01-24T09:17:32.668+0800: 39724.038: [Unloading, 0.0125520 secs],
0.0339855 secs]
[Times: user=0.12 sys=0.00, real=0.04 secs]
2021-01-24T09:17:32.700+0800: 39724.070: Total time for which application
threads were stopped: 0.0347120 seconds, Stopping threads took: 0.0000616
seconds
2021-01-24T09:17:32.700+0800: 39724.071: [GC cleanup 6355M->6355M(10240M),
0.0164190 secs]
[Times: user=0.06 sys=0.00, real=0.01 secs]
Thanks
Derrick
On Sat, Jan 23, 2021 at 7:54 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 1/23/2021 6:29 AM, Luke Oak wrote:
> > I use default settings to start solr , I set heap to 6G, I created 10
> collections with 1node and 1 replica, however, there is not much data at
> all, just 100 documents.
> >
> > My server is 32 G memory and 4 core cpu, ssd drive 300g
> >
> > It was ok when i created 5 collections. It got oom killed when 10
> collections are created. Please, no data in new collections.
>
> What version of Solr? How is it installed and started? What OS? What
> Java version?
>
> Do you have the actual OutOfMemoryError text? If I remember correctly
> from my own reading, there are eight possible causes for OOME, and not
> all of them are related to memory. The actual exception, which will be
> recorded in the main Solr logfile if it is even recorded (sometimes it's
> not), will contain the reason for the error.
>
> A 6GB heap is definitely enough for a handful of empty cores. So my
> best guess is that another resource, possibly thread count or open
> files, is running out.
>
> > Also I found that solr doesn’t do garbage collection when the 6G is used
> ( from dashboard, jvm usage is reached 6 g)
>
> Sorry to be pedantic, but Solr doesn't EVER do Garbage Collection. Java
> does. And it is completely normal for the entire Java heap to be
> consumed on occasion, no matter what's happening. Solr does not expose
> any way to force a GC.
>
> Thanks,
> Shawn
>
Re: Cannot start solr because oom
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/23/2021 6:29 AM, Luke Oak wrote:
> I use default settings to start solr , I set heap to 6G, I created 10 collections with 1node and 1 replica, however, there is not much data at all, just 100 documents.
>
> My server is 32 G memory and 4 core cpu, ssd drive 300g
>
> It was ok when i created 5 collections. It got oom killed when 10 collections are created. Please, no data in new collections.
What version of Solr? How is it installed and started? What OS? What
Java version?
Do you have the actual OutOfMemoryError text? If I remember correctly
from my own reading, there are eight possible causes for OOME, and not
all of them are related to memory. The actual exception, which will be
recorded in the main Solr logfile if it is even recorded (sometimes it's
not), will contain the reason for the error.
A 6GB heap is definitely enough for a handful of empty cores. So my
best guess is that another resource, possibly thread count or open
files, is running out.
> Also I found that solr doesn’t do garbage collection when the 6G is used ( from dashboard, jvm usage is reached 6 g)
Sorry to be pedantic, but Solr doesn't EVER do Garbage Collection. Java
does. And it is completely normal for the entire Java heap to be
consumed on occasion, no matter what's happening. Solr does not expose
any way to force a GC.
Thanks,
Shawn