You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rohan Kasat <ro...@gmail.com> on 2019/08/16 14:23:58 UTC
Solr crash | GC issue
Hi All,
I have a Solr Cloud setup of 3 solr servers 7.5 version.
24GB heap memory is allocated to each solr server and i have around 655 GB
of data in indexes to be searched for.
Few last 2-3 days, the solr servers are crashing and am able to see the
heap memory is almost full but the CPU usage is just 1 %.
I am attaching the gc logs from 3 servers. Can you please help in analyzing
yje logs and comments to improve
https://gist.github.com/rohankasat/cee8203c0c12983d9839b7a59047733b
--
*Regards,Rohan Kasat*
RE: Solr crash | GC issue
Posted by Paul Russell <pa...@qflow.com>.
For quick analysis we use https://gceasy.io
Very information and quick turnaround.
Paul
Re: Solr crash | GC issue
Posted by Rohan Kasat <ro...@gmail.com>.
Thanks Shawn.
I saw that error when the solr crashed last time. Am waiting to see if it
happens again and to capture the compete error log.
Regards,
Rohan Kasat
On Fri, Aug 16, 2019 at 2:36 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 8/16/2019 11:59 AM, Rohan Kasat wrote:
> > I see the OOM file getting created with "not much heap space" as the
> error
>
> Can you get the precise error cause? I haven't ever seen that
> particular text before. If you can paste the entire error (which will
> be many lines), that can be helpful.
>
> > Shawn, i have tried your CMS settings too and now will try increasing the
> > heap memory, hope it works this time.
>
> Changing GC tuning can never fix an OOME problem. The only way to fix
> it is to increase the resource that's running out or adjust things so
> less of that resource is needed.
>
> Thanks,
> Shawn
>
--
*Regards,Rohan Kasat*
Re: Solr crash | GC issue
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/16/2019 11:59 AM, Rohan Kasat wrote:
> I see the OOM file getting created with "not much heap space" as the error
Can you get the precise error cause? I haven't ever seen that
particular text before. If you can paste the entire error (which will
be many lines), that can be helpful.
> Shawn, i have tried your CMS settings too and now will try increasing the
> heap memory, hope it works this time.
Changing GC tuning can never fix an OOME problem. The only way to fix
it is to increase the resource that's running out or adjust things so
less of that resource is needed.
Thanks,
Shawn
Re: Solr crash | GC issue
Posted by Rohan Kasat <ro...@gmail.com>.
Thanks Shawn and Paul.
I tried using the https://gceasy.io/ but was not able to understand much.
I see the OOM file getting created with "not much heap space" as the error
.
Shawn, i have tried your CMS settings too and now will try increasing the
heap memory, hope it works this time.
Any things specific i should be checking ?
Regards,
Rohan Kasat
On Fri, Aug 16, 2019 at 12:23 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 8/16/2019 8:23 AM, Rohan Kasat wrote:
> > I have a Solr Cloud setup of 3 solr servers 7.5 version.
> > 24GB heap memory is allocated to each solr server and i have around 655
> GB
> > of data in indexes to be searched for.
> >
> > Few last 2-3 days, the solr servers are crashing and am able to see the
> > heap memory is almost full but the CPU usage is just 1 %.
> >
> > I am attaching the gc logs from 3 servers. Can you please help in
> analyzing
> > yje logs and comments to improve
> >
> > https://gist.github.com/rohankasat/cee8203c0c12983d9839b7a59047733b
>
> These three GC logs do not indicate that all the heap is used.
>
> The peak heap usage during these GC logs is 18.86GB, 19.42GB, and
> 18.91GB. That's quite a bit below the 24GB max.
>
> There are some very long GC pauses recorded. Increasing the heap size
> MIGHT help with that, or it might not.
>
> The typical way that Solr appears to "crash" is when an OutOfMemoryError
> exception is thrown, at which time a Solr instance that is running on an
> OS like Linux will kill itself with a -9 signal. This scripting is not
> present when starting on Windows.
>
> An OOME can be thrown for a resource other than memory, so despite the
> exception name, it might not actually be memory that has been depleted.
> The exception will need to be examined to learn why it was thrown.
>
> GC logs do not indicate the cause of OOME. If that information is
> logged at all, and it might not be, it will be in solr.log.
>
> Looking at the GC logs to see how your Solr is laid out... the following
> command might find the cause, if it was logged, and if the relevant log
> has not been rotated out:
>
> grep -r OutOfMemory /apps/solr/solr_data/logs/*
>
> At the very least it might help you find out which log file to
> investigate further.
>
> Thanks,
> Shawn
>
--
*Regards,Rohan Kasat*
Re: Solr crash | GC issue
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/16/2019 8:23 AM, Rohan Kasat wrote:
> I have a Solr Cloud setup of 3 solr servers 7.5 version.
> 24GB heap memory is allocated to each solr server and i have around 655 GB
> of data in indexes to be searched for.
>
> Few last 2-3 days, the solr servers are crashing and am able to see the
> heap memory is almost full but the CPU usage is just 1 %.
>
> I am attaching the gc logs from 3 servers. Can you please help in analyzing
> yje logs and comments to improve
>
> https://gist.github.com/rohankasat/cee8203c0c12983d9839b7a59047733b
These three GC logs do not indicate that all the heap is used.
The peak heap usage during these GC logs is 18.86GB, 19.42GB, and
18.91GB. That's quite a bit below the 24GB max.
There are some very long GC pauses recorded. Increasing the heap size
MIGHT help with that, or it might not.
The typical way that Solr appears to "crash" is when an OutOfMemoryError
exception is thrown, at which time a Solr instance that is running on an
OS like Linux will kill itself with a -9 signal. This scripting is not
present when starting on Windows.
An OOME can be thrown for a resource other than memory, so despite the
exception name, it might not actually be memory that has been depleted.
The exception will need to be examined to learn why it was thrown.
GC logs do not indicate the cause of OOME. If that information is
logged at all, and it might not be, it will be in solr.log.
Looking at the GC logs to see how your Solr is laid out... the following
command might find the cause, if it was logged, and if the relevant log
has not been rotated out:
grep -r OutOfMemory /apps/solr/solr_data/logs/*
At the very least it might help you find out which log file to
investigate further.
Thanks,
Shawn