You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Satya Marivada <sa...@gmail.com> on 2017/06/14 15:26:29 UTC
Out of Memory Errors
Hi,
I am getting Out of Memory Errors after a while on solr-6.3.0.
The -XX:OnOutOfMemoryError=/sanfs/mnt/vol01/solr/solr-6.3.0/bin/oom_solr.sh
just kills the jvm right after.
Using Jconsole, I see the nice triangle pattern, where it uses the heap and
being reclaimed back.
The heap size is set at 3g. The index size hosted on that particular node
is 17G.
java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4
-XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
-XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
Looking at the solr_gc.log.0, the eden space is being used 100% all the
while and being successfully reclaimed. So don't think that has go to do
with it.
Apart from that in the solr.log, I see exceptions that are aftermath of
killing the jvm
org.eclipse.jetty.io.EofException: Closed
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:383)
at
org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:90)
at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:213)
at
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:206)
at
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:136)
Any suggestions on how to go about it.
Thanks,
Satya
Re: Out of Memory Errors
Posted by Susheel Kumar <su...@gmail.com>.
The attachment will not come thru. Can you upload thru dropbox / other
sharing sites etc.
On Wed, Jun 14, 2017 at 12:41 PM, Satya Marivada <sa...@gmail.com>
wrote:
> Susheel, Please see attached. There heap towards the end of graph has
> spiked
>
>
>
> On Wed, Jun 14, 2017 at 11:46 AM Susheel Kumar <su...@gmail.com>
> wrote:
>
>> You may have gc logs saved when OOM happened. Can you draw it in GC Viewer
>> or so and share.
>>
>> Thnx
>>
>> On Wed, Jun 14, 2017 at 11:26 AM, Satya Marivada <
>> satya.chaitanya@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I am getting Out of Memory Errors after a while on solr-6.3.0.
>> > The -XX:OnOutOfMemoryError=/sanfs/mnt/vol01/solr/solr-6.3.0/bin/
>> oom_solr.sh
>> > just kills the jvm right after.
>> > Using Jconsole, I see the nice triangle pattern, where it uses the heap
>> > and being reclaimed back.
>> >
>> > The heap size is set at 3g. The index size hosted on that particular
>> node
>> > is 17G.
>> >
>> > java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4
>> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
>> > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4
>> > -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
>> > -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:
>> > CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
>> >
>> > Looking at the solr_gc.log.0, the eden space is being used 100% all the
>> > while and being successfully reclaimed. So don't think that has go to do
>> > with it.
>> >
>> > Apart from that in the solr.log, I see exceptions that are aftermath of
>> > killing the jvm
>> >
>> > org.eclipse.jetty.io.EofException: Closed
>> > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.
>> java:383)
>> > at org.apache.commons.io.output.ProxyOutputStream.write(
>> > ProxyOutputStream.java:90)
>> > at org.apache.solr.common.util.FastOutputStream.flush(
>> > FastOutputStream.java:213)
>> > at org.apache.solr.common.util.FastOutputStream.flushBuffer(
>> > FastOutputStream.java:206)
>> > at org.apache.solr.common.util.JavaBinCodec.marshal(
>> > JavaBinCodec.java:136)
>> >
>> > Any suggestions on how to go about it.
>> >
>> > Thanks,
>> > Satya
>> >
>>
>
Re: Out of Memory Errors
Posted by Satya Marivada <sa...@gmail.com>.
Susheel, Please see attached. There heap towards the end of graph has
spiked
On Wed, Jun 14, 2017 at 11:46 AM Susheel Kumar <su...@gmail.com>
wrote:
> You may have gc logs saved when OOM happened. Can you draw it in GC Viewer
> or so and share.
>
> Thnx
>
> On Wed, Jun 14, 2017 at 11:26 AM, Satya Marivada <
> satya.chaitanya@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am getting Out of Memory Errors after a while on solr-6.3.0.
> > The
> -XX:OnOutOfMemoryError=/sanfs/mnt/vol01/solr/solr-6.3.0/bin/oom_solr.sh
> > just kills the jvm right after.
> > Using Jconsole, I see the nice triangle pattern, where it uses the heap
> > and being reclaimed back.
> >
> > The heap size is set at 3g. The index size hosted on that particular node
> > is 17G.
> >
> > java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4
> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4
> > -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
> > -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:
> > CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
> >
> > Looking at the solr_gc.log.0, the eden space is being used 100% all the
> > while and being successfully reclaimed. So don't think that has go to do
> > with it.
> >
> > Apart from that in the solr.log, I see exceptions that are aftermath of
> > killing the jvm
> >
> > org.eclipse.jetty.io.EofException: Closed
> > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:383)
> > at org.apache.commons.io.output.ProxyOutputStream.write(
> > ProxyOutputStream.java:90)
> > at org.apache.solr.common.util.FastOutputStream.flush(
> > FastOutputStream.java:213)
> > at org.apache.solr.common.util.FastOutputStream.flushBuffer(
> > FastOutputStream.java:206)
> > at org.apache.solr.common.util.JavaBinCodec.marshal(
> > JavaBinCodec.java:136)
> >
> > Any suggestions on how to go about it.
> >
> > Thanks,
> > Satya
> >
>
Re: Out of Memory Errors
Posted by Susheel Kumar <su...@gmail.com>.
You may have gc logs saved when OOM happened. Can you draw it in GC Viewer
or so and share.
Thnx
On Wed, Jun 14, 2017 at 11:26 AM, Satya Marivada <sa...@gmail.com>
wrote:
> Hi,
>
> I am getting Out of Memory Errors after a while on solr-6.3.0.
> The -XX:OnOutOfMemoryError=/sanfs/mnt/vol01/solr/solr-6.3.0/bin/oom_solr.sh
> just kills the jvm right after.
> Using Jconsole, I see the nice triangle pattern, where it uses the heap
> and being reclaimed back.
>
> The heap size is set at 3g. The index size hosted on that particular node
> is 17G.
>
> java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4
> -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
> -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:
> CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
>
> Looking at the solr_gc.log.0, the eden space is being used 100% all the
> while and being successfully reclaimed. So don't think that has go to do
> with it.
>
> Apart from that in the solr.log, I see exceptions that are aftermath of
> killing the jvm
>
> org.eclipse.jetty.io.EofException: Closed
> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:383)
> at org.apache.commons.io.output.ProxyOutputStream.write(
> ProxyOutputStream.java:90)
> at org.apache.solr.common.util.FastOutputStream.flush(
> FastOutputStream.java:213)
> at org.apache.solr.common.util.FastOutputStream.flushBuffer(
> FastOutputStream.java:206)
> at org.apache.solr.common.util.JavaBinCodec.marshal(
> JavaBinCodec.java:136)
>
> Any suggestions on how to go about it.
>
> Thanks,
> Satya
>