You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Clemens Wyss DEV <cl...@mysign.ch> on 2015/06/03 08:20:12 UTC
Solr OutOfMemory but no heap and dump and oo_solr.sh is not triggered
Context: Lucene 5.1, Java 8 on debian. 24G of RAM whereof 16G available for Solr.
I am seeing the following OOMs:
ERROR - 2015-06-03 05:17:13.317; [ customer-1-de_CH_1] org.apache.solr.common.SolrException; null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:854)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:463)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
WARN - 2015-06-03 05:17:13.319; [ customer-1-de_CH_1] org.eclipse.jetty.servlet.ServletHandler; Error for /solr/customer-1-de_CH_1/suggest_phrase
java.lang.OutOfMemoryError: Java heap space
The full commandline is
/usr/local/java/bin/java -server -Xss256k -Xms16G
-Xmx16G -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/logs/solr_gc.log -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Dsolr.solr.home=/opt/solr/data -Dsolr.install.dir=/usr/local/solr -Dlog4j.configuration=file:/opt/solr/log4j.properties
-jar start.jar -XX:OnOutOfMemoryError=/usr/local/solr/bin/oom_solr.sh 8983 /opt/solr/logs OPTIONS=default,rewrite
So I'd expect /usr/local/solr/bin/oom_solr.sh tob e triggered. But this does not seem to "happen". What am I missing? Is it o to pull a heapdump from Solr before killing/rebooting in oom_solr.sh?
Also I would like to know what query parameters were sent to /solr/customer-1-de_CH_1/suggest_phrase (which may be the reason fort he OOM ...
AW: Solr OutOfMemory but no heap and dump and oo_solr.sh is not
triggered
Posted by Clemens Wyss DEV <cl...@mysign.ch>.
Sorry for the delay -> https://issues.apache.org/jira/browse/SOLR-7646
-----Ursprüngliche Nachricht-----
Von: Erick Erickson [mailto:erickerickson@gmail.com]
Gesendet: Mittwoch, 3. Juni 2015 17:39
An: solr-user@lucene.apache.org
Betreff: Re: Solr OutOfMemory but no heap and dump and oo_solr.sh is not triggered
bq: what exactly should I file? What needs to be added/appended to the issue?
Just what Mark said, title it something like OOM exception wrapped in runtime exception
Include your original post and that you were asked to open the JIRA after discussion on the user's list. Don't worry too much, the title & etc. can be changed after as things become clearer.
Best,
Erick
On Wed, Jun 3, 2015 at 5:58 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
> Hi Mark,
> what exactly should I file? What needs to be added/appended to the issue?
>
> Regards
> Clemens
>
> -----Ursprüngliche Nachricht-----
> Von: Mark Miller [mailto:markrmiller@gmail.com]
> Gesendet: Mittwoch, 3. Juni 2015 14:23
> An: solr-user@lucene.apache.org
> Betreff: Re: Solr OutOfMemory but no heap and dump and oo_solr.sh is
> not triggered
>
> We will have to a find a way to deal with this long term. Browsing the code I can see a variety of places where problem exception handling has been introduced since this all was fixed.
>
> - Mark
>
> On Wed, Jun 3, 2015 at 8:19 AM Mark Miller <ma...@gmail.com> wrote:
>
>> File a JIRA issue please. That OOM Exception is getting wrapped in a
>> RuntimeException it looks. Bug.
>>
>> - Mark
>>
>>
>> On Wed, Jun 3, 2015 at 2:20 AM Clemens Wyss DEV
>> <cl...@mysign.ch>
>> wrote:
>>
>>> Context: Lucene 5.1, Java 8 on debian. 24G of RAM whereof 16G
>>> available for Solr.
>>>
>>> I am seeing the following OOMs:
>>> ERROR - 2015-06-03 05:17:13.317; [ customer-1-de_CH_1]
>>> org.apache.solr.common.SolrException; null:java.lang.RuntimeException:
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:854)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:463)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>>> at
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
>>> at
>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
>>> at
>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>>> at org.eclipse.jetty.server.Server.handle(Server.java:368)
>>> at
>>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
>>> at
>>> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
>>> at
>>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
>>> at
>>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
>>> at
>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>> at
>>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>>> at
>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
>>> at
>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>> WARN - 2015-06-03 05:17:13.319; [ customer-1-de_CH_1]
>>> org.eclipse.jetty.servlet.ServletHandler; Error for
>>> /solr/customer-1-de_CH_1/suggest_phrase
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> The full commandline is
>>> /usr/local/java/bin/java -server -Xss256k -Xms16G -Xmx16G
>>> -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90
>>> -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
>>> -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m
>>> -XX:+UseCMSInitiatingOccupancyOnly
>>> -XX:CMSInitiatingOccupancyFraction=50
>>> -XX:CMSMaxAbortablePrecleanTime=6000
>>> -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled
>>> -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails
>>> -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
>>> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
>>> -Xloggc:/opt/solr/logs/solr_gc.log
>>> -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks
>>> -Duser.timezone=UTC -Dsolr.solr.home=/opt/solr/data
>>> -Dsolr.install.dir=/usr/local/solr
>>> -Dlog4j.configuration=file:/opt/solr/log4j.properties
>>> -jar start.jar
>>> -XX:OnOutOfMemoryError=/usr/local/solr/bin/oom_solr.sh
>>> 8983 /opt/solr/logs OPTIONS=default,rewrite
>>>
>>> So I'd expect /usr/local/solr/bin/oom_solr.sh tob e triggered. But
>>> this does not seem to "happen". What am I missing? Is it o to pull a
>>> heapdump from Solr before killing/rebooting in oom_solr.sh?
>>>
>>> Also I would like to know what query parameters were sent to
>>> /solr/customer-1-de_CH_1/suggest_phrase (which may be the reason
>>> fort he OOM ...
>>>
>>>
>>> --
>> - Mark
>> about.me/markrmiller
>>
> --
> - Mark
> about.me/markrmiller
Re: Solr OutOfMemory but no heap and dump and oo_solr.sh is not triggered
Posted by Erick Erickson <er...@gmail.com>.
bq: what exactly should I file? What needs to be added/appended to the issue?
Just what Mark said, title it something like
OOM exception wrapped in runtime exception
Include your original post and that you were asked to open the JIRA
after discussion on the user's list. Don't worry too much, the title &
etc. can be changed after as things become clearer.
Best,
Erick
On Wed, Jun 3, 2015 at 5:58 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
> Hi Mark,
> what exactly should I file? What needs to be added/appended to the issue?
>
> Regards
> Clemens
>
> -----Ursprüngliche Nachricht-----
> Von: Mark Miller [mailto:markrmiller@gmail.com]
> Gesendet: Mittwoch, 3. Juni 2015 14:23
> An: solr-user@lucene.apache.org
> Betreff: Re: Solr OutOfMemory but no heap and dump and oo_solr.sh is not triggered
>
> We will have to a find a way to deal with this long term. Browsing the code I can see a variety of places where problem exception handling has been introduced since this all was fixed.
>
> - Mark
>
> On Wed, Jun 3, 2015 at 8:19 AM Mark Miller <ma...@gmail.com> wrote:
>
>> File a JIRA issue please. That OOM Exception is getting wrapped in a
>> RuntimeException it looks. Bug.
>>
>> - Mark
>>
>>
>> On Wed, Jun 3, 2015 at 2:20 AM Clemens Wyss DEV <cl...@mysign.ch>
>> wrote:
>>
>>> Context: Lucene 5.1, Java 8 on debian. 24G of RAM whereof 16G
>>> available for Solr.
>>>
>>> I am seeing the following OOMs:
>>> ERROR - 2015-06-03 05:17:13.317; [ customer-1-de_CH_1]
>>> org.apache.solr.common.SolrException; null:java.lang.RuntimeException:
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:854)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:463)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>>> at
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
>>> at
>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
>>> at
>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>>> at org.eclipse.jetty.server.Server.handle(Server.java:368)
>>> at
>>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
>>> at
>>> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
>>> at
>>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
>>> at
>>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
>>> at
>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>> at
>>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>>> at
>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
>>> at
>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>> WARN - 2015-06-03 05:17:13.319; [ customer-1-de_CH_1]
>>> org.eclipse.jetty.servlet.ServletHandler; Error for
>>> /solr/customer-1-de_CH_1/suggest_phrase
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> The full commandline is
>>> /usr/local/java/bin/java -server -Xss256k -Xms16G -Xmx16G
>>> -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90
>>> -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
>>> -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m
>>> -XX:+UseCMSInitiatingOccupancyOnly
>>> -XX:CMSInitiatingOccupancyFraction=50
>>> -XX:CMSMaxAbortablePrecleanTime=6000
>>> -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc
>>> -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>>> -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution
>>> -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/logs/solr_gc.log
>>> -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks
>>> -Duser.timezone=UTC -Dsolr.solr.home=/opt/solr/data
>>> -Dsolr.install.dir=/usr/local/solr
>>> -Dlog4j.configuration=file:/opt/solr/log4j.properties
>>> -jar start.jar -XX:OnOutOfMemoryError=/usr/local/solr/bin/oom_solr.sh
>>> 8983 /opt/solr/logs OPTIONS=default,rewrite
>>>
>>> So I'd expect /usr/local/solr/bin/oom_solr.sh tob e triggered. But
>>> this does not seem to "happen". What am I missing? Is it o to pull a
>>> heapdump from Solr before killing/rebooting in oom_solr.sh?
>>>
>>> Also I would like to know what query parameters were sent to
>>> /solr/customer-1-de_CH_1/suggest_phrase (which may be the reason fort
>>> he OOM ...
>>>
>>>
>>> --
>> - Mark
>> about.me/markrmiller
>>
> --
> - Mark
> about.me/markrmiller
AW: Solr OutOfMemory but no heap and dump and oo_solr.sh is not
triggered
Posted by Clemens Wyss DEV <cl...@mysign.ch>.
Hi Mark,
what exactly should I file? What needs to be added/appended to the issue?
Regards
Clemens
-----Ursprüngliche Nachricht-----
Von: Mark Miller [mailto:markrmiller@gmail.com]
Gesendet: Mittwoch, 3. Juni 2015 14:23
An: solr-user@lucene.apache.org
Betreff: Re: Solr OutOfMemory but no heap and dump and oo_solr.sh is not triggered
We will have to a find a way to deal with this long term. Browsing the code I can see a variety of places where problem exception handling has been introduced since this all was fixed.
- Mark
On Wed, Jun 3, 2015 at 8:19 AM Mark Miller <ma...@gmail.com> wrote:
> File a JIRA issue please. That OOM Exception is getting wrapped in a
> RuntimeException it looks. Bug.
>
> - Mark
>
>
> On Wed, Jun 3, 2015 at 2:20 AM Clemens Wyss DEV <cl...@mysign.ch>
> wrote:
>
>> Context: Lucene 5.1, Java 8 on debian. 24G of RAM whereof 16G
>> available for Solr.
>>
>> I am seeing the following OOMs:
>> ERROR - 2015-06-03 05:17:13.317; [ customer-1-de_CH_1]
>> org.apache.solr.common.SolrException; null:java.lang.RuntimeException:
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:854)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:463)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>> at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>> at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>> at org.eclipse.jetty.server.Server.handle(Server.java:368)
>> at
>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
>> at
>> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
>> at
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
>> at
>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
>> at
>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>> at
>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>> at
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
>> at
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> WARN - 2015-06-03 05:17:13.319; [ customer-1-de_CH_1]
>> org.eclipse.jetty.servlet.ServletHandler; Error for
>> /solr/customer-1-de_CH_1/suggest_phrase
>> java.lang.OutOfMemoryError: Java heap space
>>
>> The full commandline is
>> /usr/local/java/bin/java -server -Xss256k -Xms16G -Xmx16G
>> -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90
>> -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
>> -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m
>> -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:CMSInitiatingOccupancyFraction=50
>> -XX:CMSMaxAbortablePrecleanTime=6000
>> -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc
>> -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>> -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution
>> -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/logs/solr_gc.log
>> -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks
>> -Duser.timezone=UTC -Dsolr.solr.home=/opt/solr/data
>> -Dsolr.install.dir=/usr/local/solr
>> -Dlog4j.configuration=file:/opt/solr/log4j.properties
>> -jar start.jar -XX:OnOutOfMemoryError=/usr/local/solr/bin/oom_solr.sh
>> 8983 /opt/solr/logs OPTIONS=default,rewrite
>>
>> So I'd expect /usr/local/solr/bin/oom_solr.sh tob e triggered. But
>> this does not seem to "happen". What am I missing? Is it o to pull a
>> heapdump from Solr before killing/rebooting in oom_solr.sh?
>>
>> Also I would like to know what query parameters were sent to
>> /solr/customer-1-de_CH_1/suggest_phrase (which may be the reason fort
>> he OOM ...
>>
>>
>> --
> - Mark
> about.me/markrmiller
>
--
- Mark
about.me/markrmiller
Re: Solr OutOfMemory but no heap and dump and oo_solr.sh is not triggered
Posted by Mark Miller <ma...@gmail.com>.
We will have to a find a way to deal with this long term. Browsing the code
I can see a variety of places where problem exception handling has been
introduced since this all was fixed.
- Mark
On Wed, Jun 3, 2015 at 8:19 AM Mark Miller <ma...@gmail.com> wrote:
> File a JIRA issue please. That OOM Exception is getting wrapped in a
> RuntimeException it looks. Bug.
>
> - Mark
>
>
> On Wed, Jun 3, 2015 at 2:20 AM Clemens Wyss DEV <cl...@mysign.ch>
> wrote:
>
>> Context: Lucene 5.1, Java 8 on debian. 24G of RAM whereof 16G available
>> for Solr.
>>
>> I am seeing the following OOMs:
>> ERROR - 2015-06-03 05:17:13.317; [ customer-1-de_CH_1]
>> org.apache.solr.common.SolrException; null:java.lang.RuntimeException:
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:854)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:463)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>> at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>> at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>> at org.eclipse.jetty.server.Server.handle(Server.java:368)
>> at
>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
>> at
>> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
>> at
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
>> at
>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
>> at
>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>> at
>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>> at
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
>> at
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> WARN - 2015-06-03 05:17:13.319; [ customer-1-de_CH_1]
>> org.eclipse.jetty.servlet.ServletHandler; Error for
>> /solr/customer-1-de_CH_1/suggest_phrase
>> java.lang.OutOfMemoryError: Java heap space
>>
>> The full commandline is
>> /usr/local/java/bin/java -server -Xss256k -Xms16G
>> -Xmx16G -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90
>> -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
>> -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
>> -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc
>> -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>> -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution
>> -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/logs/solr_gc.log
>> -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC
>> -Dsolr.solr.home=/opt/solr/data -Dsolr.install.dir=/usr/local/solr
>> -Dlog4j.configuration=file:/opt/solr/log4j.properties
>> -jar start.jar -XX:OnOutOfMemoryError=/usr/local/solr/bin/oom_solr.sh
>> 8983 /opt/solr/logs OPTIONS=default,rewrite
>>
>> So I'd expect /usr/local/solr/bin/oom_solr.sh tob e triggered. But this
>> does not seem to "happen". What am I missing? Is it o to pull a heapdump
>> from Solr before killing/rebooting in oom_solr.sh?
>>
>> Also I would like to know what query parameters were sent to
>> /solr/customer-1-de_CH_1/suggest_phrase (which may be the reason fort he
>> OOM ...
>>
>>
>> --
> - Mark
> about.me/markrmiller
>
--
- Mark
about.me/markrmiller
Re: Solr OutOfMemory but no heap and dump and oo_solr.sh is not triggered
Posted by Mark Miller <ma...@gmail.com>.
File a JIRA issue please. That OOM Exception is getting wrapped in a
RuntimeException it looks. Bug.
- Mark
On Wed, Jun 3, 2015 at 2:20 AM Clemens Wyss DEV <cl...@mysign.ch>
wrote:
> Context: Lucene 5.1, Java 8 on debian. 24G of RAM whereof 16G available
> for Solr.
>
> I am seeing the following OOMs:
> ERROR - 2015-06-03 05:17:13.317; [ customer-1-de_CH_1]
> org.apache.solr.common.SolrException; null:java.lang.RuntimeException:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:854)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:463)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> WARN - 2015-06-03 05:17:13.319; [ customer-1-de_CH_1]
> org.eclipse.jetty.servlet.ServletHandler; Error for
> /solr/customer-1-de_CH_1/suggest_phrase
> java.lang.OutOfMemoryError: Java heap space
>
> The full commandline is
> /usr/local/java/bin/java -server -Xss256k -Xms16G
> -Xmx16G -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90
> -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
> -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
> -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc
> -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution
> -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/logs/solr_gc.log
> -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC
> -Dsolr.solr.home=/opt/solr/data -Dsolr.install.dir=/usr/local/solr
> -Dlog4j.configuration=file:/opt/solr/log4j.properties
> -jar start.jar -XX:OnOutOfMemoryError=/usr/local/solr/bin/oom_solr.sh 8983
> /opt/solr/logs OPTIONS=default,rewrite
>
> So I'd expect /usr/local/solr/bin/oom_solr.sh tob e triggered. But this
> does not seem to "happen". What am I missing? Is it o to pull a heapdump
> from Solr before killing/rebooting in oom_solr.sh?
>
> Also I would like to know what query parameters were sent to
> /solr/customer-1-de_CH_1/suggest_phrase (which may be the reason fort he
> OOM ...
>
>
> --
- Mark
about.me/markrmiller
Re: AW: Solr OutOfMemory but no heap and dump and oo_solr.sh is not
triggered
Posted by Shawn Heisey <ap...@elyograg.org>.
On 6/3/2015 1:41 AM, Clemens Wyss DEV wrote:
>> The oom script just kills Solr with the KILL signal (-9) and logs the kill.
> I know. But my feeling is, that not even this "happens", i.e. the script is not being executed. At least I see no solr_oom_killer-$SOLR_PORT-$NOW.log file ...
>
> Btw:
> Who re-starts solr after it's been killed?
I'm not sure what to think here. I wonder if the Java you are using is
broken?
Restarting would most likely need to be handled by you. You could put
Solr under the management of something that keeps it running, like
heartbeat, pacemaker, or the supervisor program that comes with qmail,
whose name I can never remember.
You could add something to send you an email every time the OOM script
is run, so you know it has been killed. If you have sized the memory
appropriately, the OOM killer will never run.
>> I don't know if a heap dump on OOM is compatible with the OOM script.
>> If Java chooses to run the OOM script before the heap dump is done, the process
>> will be killed before the heap finishes dumping.
> What if I did a jmap-call in the oom-script before killing the process?
You very likely could add that, but be aware of how much time anything
you add will take. If your cloud is sufficiently redundant, then it
probably won't matter if one node is down for several minutes.
Thanks,
Shawn
AW: Solr OutOfMemory but no heap and dump and oo_solr.sh is not
triggered
Posted by Clemens Wyss DEV <cl...@mysign.ch>.
Ciao Shawn,
thanks for your reply.
> The oom script just kills Solr with the KILL signal (-9) and logs the kill.
I know. But my feeling is, that not even this "happens", i.e. the script is not being executed. At least I see no solr_oom_killer-$SOLR_PORT-$NOW.log file ...
Btw:
Who re-starts solr after it's been killed?
> FYI, the stacktrace on the OOM error, especially in a multi-threaded app like Solr,
>will frequently be completely useless in tracking down the problem.
I agree
> I don't know if a heap dump on OOM is compatible with the OOM script.
>If Java chooses to run the OOM script before the heap dump is done, the process
>will be killed before the heap finishes dumping.
What if I did a jmap-call in the oom-script before killing the process?
-Clemens
-----Ursprüngliche Nachricht-----
Von: Shawn Heisey [mailto:apache@elyograg.org]
Gesendet: Mittwoch, 3. Juni 2015 09:16
An: solr-user@lucene.apache.org
Betreff: Re: Solr OutOfMemory but no heap and dump and oo_solr.sh is not triggered
On 6/3/2015 12:20 AM, Clemens Wyss DEV wrote:
> Context: Lucene 5.1, Java 8 on debian. 24G of RAM whereof 16G available for Solr.
>
> I am seeing the following OOMs:
> ERROR - 2015-06-03 05:17:13.317; [ customer-1-de_CH_1] org.apache.solr.common.SolrException; null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
<snip>
> Caused by: java.lang.OutOfMemoryError: Java heap space
> WARN - 2015-06-03 05:17:13.319; [ customer-1-de_CH_1] org.eclipse.jetty.servlet.ServletHandler; Error for /solr/customer-1-de_CH_1/suggest_phrase
> java.lang.OutOfMemoryError: Java heap space
>
> The full commandline is
> /usr/local/java/bin/java -server -Xss256k -Xms16G -Xmx16G
> -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90
> -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
> -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=50
> -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled
> -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -Xloggc:/opt/solr/logs/solr_gc.log -Djetty.port=8983 -DSTOP.PORT=7983
> -DSTOP.KEY=solrrocks -Duser.timezone=UTC
> -Dsolr.solr.home=/opt/solr/data -Dsolr.install.dir=/usr/local/solr
> -Dlog4j.configuration=file:/opt/solr/log4j.properties
> -jar start.jar -XX:OnOutOfMemoryError=/usr/local/solr/bin/oom_solr.sh
> 8983 /opt/solr/logs OPTIONS=default,rewrite
>
> So I'd expect /usr/local/solr/bin/oom_solr.sh tob e triggered. But this does not seem to "happen". What am I missing? Is it o to pull a heapdump from Solr before killing/rebooting in oom_solr.sh?
>
> Also I would like to know what query parameters were sent to /solr/customer-1-de_CH_1/suggest_phrase (which may be the reason fort he OOM ...
The oom script just kills Solr with the KILL signal (-9) and logs the kill. That's it. It does not attempt to make a heap dump. If you
*want* to dump the heap on OOM, you can, with some additional options:
http://stackoverflow.com/questions/542979/using-heapdumponoutofmemoryerror-parameter-for-heap-dump-for-jboss/20496376#20496376
I don't know if a heap dump on OOM is compatible with the OOM script.
If Java chooses to run the OOM script before the heap dump is done, the process will be killed before the heap finishes dumping.
FYI, the stacktrace on the OOM error, especially in a multi-threaded app like Solr, will frequently be completely useless in tracking down the problem. The thread that makes the triggering memory allocation may be completely unrelated. This error happened on a suggest handler ... but the large memory allocations may be happening in a completely different part of the code.
We have not had any recent indications of a memory leak in Solr. Memory leaks in Solr *do* happen, but they are usually caught by the tests.
which run in a minimal memory space. The project has continuous integration servers set up that run all the tests many times per day.
If you are running out of heap with 16GB allocated, then either your Solr installation is enormous or you've got a configuration that's not tuned properly. With a very large Solr installation, you may need to simply allocate more memory to the heap ... which may mean that you'll need to install more memory in the server. The alternative would be figuring out where you can change your configuration to reduce memory requirements.
Here's some incomplete info on settings and situations that can require a very large heap:
https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
To provide much help, we'll need lots of details about your system ...
number of documents in all cores, total index size on disk, your config, possibly your schema, and maybe a few other things I haven't thought of yet.
Thanks,
Shawn
Re: Solr OutOfMemory but no heap and dump and oo_solr.sh is not triggered
Posted by Shawn Heisey <ap...@elyograg.org>.
On 6/3/2015 12:20 AM, Clemens Wyss DEV wrote:
> Context: Lucene 5.1, Java 8 on debian. 24G of RAM whereof 16G available for Solr.
>
> I am seeing the following OOMs:
> ERROR - 2015-06-03 05:17:13.317; [ customer-1-de_CH_1] org.apache.solr.common.SolrException; null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
<snip>
> Caused by: java.lang.OutOfMemoryError: Java heap space
> WARN - 2015-06-03 05:17:13.319; [ customer-1-de_CH_1] org.eclipse.jetty.servlet.ServletHandler; Error for /solr/customer-1-de_CH_1/suggest_phrase
> java.lang.OutOfMemoryError: Java heap space
>
> The full commandline is
> /usr/local/java/bin/java -server -Xss256k -Xms16G
> -Xmx16G -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/logs/solr_gc.log -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Dsolr.solr.home=/opt/solr/data -Dsolr.install.dir=/usr/local/solr -Dlog4j.configuration=file:/opt/solr/log4j.properties
> -jar start.jar -XX:OnOutOfMemoryError=/usr/local/solr/bin/oom_solr.sh 8983 /opt/solr/logs OPTIONS=default,rewrite
>
> So I'd expect /usr/local/solr/bin/oom_solr.sh tob e triggered. But this does not seem to "happen". What am I missing? Is it o to pull a heapdump from Solr before killing/rebooting in oom_solr.sh?
>
> Also I would like to know what query parameters were sent to /solr/customer-1-de_CH_1/suggest_phrase (which may be the reason fort he OOM ...
The oom script just kills Solr with the KILL signal (-9) and logs the
kill. That's it. It does not attempt to make a heap dump. If you
*want* to dump the heap on OOM, you can, with some additional options:
http://stackoverflow.com/questions/542979/using-heapdumponoutofmemoryerror-parameter-for-heap-dump-for-jboss/20496376#20496376
I don't know if a heap dump on OOM is compatible with the OOM script.
If Java chooses to run the OOM script before the heap dump is done, the
process will be killed before the heap finishes dumping.
FYI, the stacktrace on the OOM error, especially in a multi-threaded app
like Solr, will frequently be completely useless in tracking down the
problem. The thread that makes the triggering memory allocation may be
completely unrelated. This error happened on a suggest handler ... but
the large memory allocations may be happening in a completely different
part of the code.
We have not had any recent indications of a memory leak in Solr. Memory
leaks in Solr *do* happen, but they are usually caught by the tests.
which run in a minimal memory space. The project has continuous
integration servers set up that run all the tests many times per day.
If you are running out of heap with 16GB allocated, then either your
Solr installation is enormous or you've got a configuration that's not
tuned properly. With a very large Solr installation, you may need to
simply allocate more memory to the heap ... which may mean that you'll
need to install more memory in the server. The alternative would be
figuring out where you can change your configuration to reduce memory
requirements.
Here's some incomplete info on settings and situations that can require
a very large heap:
https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
To provide much help, we'll need lots of details about your system ...
number of documents in all cores, total index size on disk, your config,
possibly your schema, and maybe a few other things I haven't thought of yet.
Thanks,
Shawn