You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Grzegorz Sobczyk <gs...@gmail.com> on 2013/08/01 15:24:53 UTC
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Today I found in solr logs exception: java.lang.OutOfMemoryError: Requested
array size exceeds VM limit.
At that time memory usage was ~200MB / Xmx3g
Env looks like this:
3x standalone zK (Java 7)
3x Solr 4.2.1 on Tomcat (Java 7)
Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux
One Solr and one ZK on single host: lmsiprse01, lmsiprse02, lmsiprse03
Before exception I start restarting Solr:
lmsiprse01:
[2013-08-01 05:23:43]: /etc/init.d/tomcat6-1 stop
[2013-08-01 05:25:09]: /etc/init.d/tomcat6-1 start
lmsiprse02 (leader):
2013-08-01 05:27:21]: /etc/init.d/tomcat6-1 stop
2013-08-01 05:29:31]: /etc/init.d/tomcat6-1 start
lmsiprse03:
[2013-08-01 05:25:48]: /etc/init.d/tomcat6-1 stop
[2013-08-01 05:26:42]: /etc/init.d/tomcat6-1 start
and error shows up at 2013 5:27:26 on lmsiprse01
Is anybody knows what happened?
fragment of log looks:
sie 01, 2013 5:27:26 AM org.apache.solr.core.SolrCore execute
INFO: [products] webapp=/solr path=/select
params={facet=true&start=0&q=&facet.limit=-1&facet.field=attribute_u-typ&facet.field=attribute_u-gama-kolorystyczna&facet.field=brand_name&wt=javabin&fq=node_id:1056&version=2&rows=0}
hits=1241 status=0 QTime=33
sie 01, 2013 5:27:26 AM org.apache.solr.common.SolrException log
SEVERE: null:java.lang.RuntimeException: java.lang.OutOfMemoryError:
Requested array size exceeds VM limit
at
org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:653)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:366)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:64)
at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:37)
at
org.apache.solr.handler.component.ShardFieldSortedHitQueue.<init>(ShardDoc.java:113)
at
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:766)
at
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:625)
at
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:604)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1817)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
... 13 more
--
Grzegorz Sobczyk
Re: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Posted by Erick Erickson <er...@gmail.com>.
Well, this is probably not a rogue query. You can test this, of course
by replaying all your queries on a test system. My guess is that it's
just too much stuff on too small a box.
Or you could have poorly configured Solr parameters. I've seen, for
instance, the filterCache sized at 1M. Which runs fine for a long time
until indexing stops and the filterCache fills up. Which from
your picture seems like a possibility, you have an increasing
memory usage for days.
I'd suggest something like jhat/jmap to find out where the memory
is being used up, a review of your configs, things like that.
Best
Erick
On Fri, Aug 2, 2013 at 7:44 AM, Grzegorz Sobczyk <gs...@gmail.com> wrote:
> Solr parameters listed in dashboard:
> -DzkHost=localhost:2181,172.27.5.121:2181,172.27.5.122:2181
> -XX:+UseConcMarkSweepGC
> -Xmx3072m
> -Djava.awt.headless=true
>
> Mem usage in last days: http://i42.tinypic.com/29z5rew.png
>
> It's production system and there's too many requests to detect which query
> is buggy
> some lines before every error (result of: zgrep -B 200 "SEVERE:
> null:java.lang.RuntimeException: java.lang.OutOfMemoryError"
> daemon.err.2.gz > out.log )
> https://gist.github.com/gsobczyk/65e2ab9bb13a99aec89e
>
> I've tried invoking recent request before exception but without success of
> reproduce error
>
>
>
> On 1 August 2013 20:39, Erick Erickson <er...@gmail.com> wrote:
>
> > What are the memory parameters you start Solr with? The Solr admin page
> > will tell you how much memory the JVM has.
> >
> > Also, cut/paste the queries you're running when you see this.
> >
> > Best
> > Erick
> >
> >
> > On Thu, Aug 1, 2013 at 9:50 AM, Grzegorz Sobczyk <gs...@gmail.com>
> > wrote:
> >
> > > after node starts in log I have only few requests:
> > > https://gist.github.com/gsobczyk/6131503#file-solr-oom-log
> > > this error occurred multiple times
> > >
> > >
> > >
> > >
> > > On 1 August 2013 15:33, Rafał Kuć <r....@solr.pl> wrote:
> > >
> > > > Hello!
> > > >
> > > > The exception you've shown tells you that Solr tried to allocate an
> > > > array that exceeded heap size. Do you use some custom sorts? Did you
> > > > send large bulks during the time that the exception occurred?
> > > >
> > > > --
> > > > Regards,
> > > > Rafał Kuć
> > > > Sematext :: http://sematext.com/ :: Solr - Lucene - ElasticSearch
> > > >
> > > > > Today I found in solr logs exception: java.lang.OutOfMemoryError:
> > > > Requested
> > > > > array size exceeds VM limit.
> > > > > At that time memory usage was ~200MB / Xmx3g
> > > >
> > > > > Env looks like this:
> > > > > 3x standalone zK (Java 7)
> > > > > 3x Solr 4.2.1 on Tomcat (Java 7)
> > > > > Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux
> > > > > One Solr and one ZK on single host: lmsiprse01, lmsiprse02,
> > lmsiprse03
> > > >
> > > > > Before exception I start restarting Solr:
> > > > > lmsiprse01:
> > > > > [2013-08-01 05:23:43]: /etc/init.d/tomcat6-1 stop
> > > > > [2013-08-01 05:25:09]: /etc/init.d/tomcat6-1 start
> > > > > lmsiprse02 (leader):
> > > > > 2013-08-01 05:27:21]: /etc/init.d/tomcat6-1 stop
> > > > > 2013-08-01 05:29:31]: /etc/init.d/tomcat6-1 start
> > > > > lmsiprse03:
> > > > > [2013-08-01 05:25:48]: /etc/init.d/tomcat6-1 stop
> > > > > [2013-08-01 05:26:42]: /etc/init.d/tomcat6-1 start
> > > >
> > > > > and error shows up at 2013 5:27:26 on lmsiprse01
> > > >
> > > > > Is anybody knows what happened?
> > > >
> > > > > fragment of log looks:
> > > > > sie 01, 2013 5:27:26 AM org.apache.solr.core.SolrCore execute
> > > > > INFO: [products] webapp=/solr path=/select
> > > > >
> > > >
> > >
> >
> params={facet=true&start=0&q=&facet.limit=-1&facet.field=attribute_u-typ&facet.field=attribute_u-gama-kolorystyczna&facet.field=brand_name&wt=javabin&fq=node_id:1056&version=2&rows=0}
> > > > > hits=1241 status=0 QTime=33
> > > > > sie 01, 2013 5:27:26 AM org.apache.solr.common.SolrException log
> > > > > SEVERE: null:java.lang.RuntimeException:
> java.lang.OutOfMemoryError:
> > > > > Requested array size exceeds VM limit
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:653)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:366)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
> > > > > at
> > > > >
> > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> > > > > at java.lang.Thread.run(Thread.java:724)
> > > > > Caused by: java.lang.OutOfMemoryError: Requested array size exceeds
> > VM
> > > > limit
> > > > > at
> org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:64)
> > > > > at
> org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:37)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.component.ShardFieldSortedHitQueue.<init>(ShardDoc.java:113)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:766)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:625)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:604)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > > > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1817)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> > > > > ... 13 more
> > > >
> > > >
> > >
> > >
> > > --
> > > Grzegorz Sobczyk
> > >
> >
>
>
>
> --
> Grzegorz Sobczyk
>
Re: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Posted by Grzegorz Sobczyk <gs...@gmail.com>.
Solr parameters listed in dashboard:
-DzkHost=localhost:2181,172.27.5.121:2181,172.27.5.122:2181
-XX:+UseConcMarkSweepGC
-Xmx3072m
-Djava.awt.headless=true
Mem usage in last days: http://i42.tinypic.com/29z5rew.png
It's production system and there's too many requests to detect which query
is buggy
some lines before every error (result of: zgrep -B 200 "SEVERE:
null:java.lang.RuntimeException: java.lang.OutOfMemoryError"
daemon.err.2.gz > out.log )
https://gist.github.com/gsobczyk/65e2ab9bb13a99aec89e
I've tried invoking recent request before exception but without success of
reproduce error
On 1 August 2013 20:39, Erick Erickson <er...@gmail.com> wrote:
> What are the memory parameters you start Solr with? The Solr admin page
> will tell you how much memory the JVM has.
>
> Also, cut/paste the queries you're running when you see this.
>
> Best
> Erick
>
>
> On Thu, Aug 1, 2013 at 9:50 AM, Grzegorz Sobczyk <gs...@gmail.com>
> wrote:
>
> > after node starts in log I have only few requests:
> > https://gist.github.com/gsobczyk/6131503#file-solr-oom-log
> > this error occurred multiple times
> >
> >
> >
> >
> > On 1 August 2013 15:33, Rafał Kuć <r....@solr.pl> wrote:
> >
> > > Hello!
> > >
> > > The exception you've shown tells you that Solr tried to allocate an
> > > array that exceeded heap size. Do you use some custom sorts? Did you
> > > send large bulks during the time that the exception occurred?
> > >
> > > --
> > > Regards,
> > > Rafał Kuć
> > > Sematext :: http://sematext.com/ :: Solr - Lucene - ElasticSearch
> > >
> > > > Today I found in solr logs exception: java.lang.OutOfMemoryError:
> > > Requested
> > > > array size exceeds VM limit.
> > > > At that time memory usage was ~200MB / Xmx3g
> > >
> > > > Env looks like this:
> > > > 3x standalone zK (Java 7)
> > > > 3x Solr 4.2.1 on Tomcat (Java 7)
> > > > Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux
> > > > One Solr and one ZK on single host: lmsiprse01, lmsiprse02,
> lmsiprse03
> > >
> > > > Before exception I start restarting Solr:
> > > > lmsiprse01:
> > > > [2013-08-01 05:23:43]: /etc/init.d/tomcat6-1 stop
> > > > [2013-08-01 05:25:09]: /etc/init.d/tomcat6-1 start
> > > > lmsiprse02 (leader):
> > > > 2013-08-01 05:27:21]: /etc/init.d/tomcat6-1 stop
> > > > 2013-08-01 05:29:31]: /etc/init.d/tomcat6-1 start
> > > > lmsiprse03:
> > > > [2013-08-01 05:25:48]: /etc/init.d/tomcat6-1 stop
> > > > [2013-08-01 05:26:42]: /etc/init.d/tomcat6-1 start
> > >
> > > > and error shows up at 2013 5:27:26 on lmsiprse01
> > >
> > > > Is anybody knows what happened?
> > >
> > > > fragment of log looks:
> > > > sie 01, 2013 5:27:26 AM org.apache.solr.core.SolrCore execute
> > > > INFO: [products] webapp=/solr path=/select
> > > >
> > >
> >
> params={facet=true&start=0&q=&facet.limit=-1&facet.field=attribute_u-typ&facet.field=attribute_u-gama-kolorystyczna&facet.field=brand_name&wt=javabin&fq=node_id:1056&version=2&rows=0}
> > > > hits=1241 status=0 QTime=33
> > > > sie 01, 2013 5:27:26 AM org.apache.solr.common.SolrException log
> > > > SEVERE: null:java.lang.RuntimeException: java.lang.OutOfMemoryError:
> > > > Requested array size exceeds VM limit
> > > > at
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:653)
> > > > at
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:366)
> > > > at
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
> > > > at
> > > >
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > > > at
> > > >
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > > > at
> > > >
> > >
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > > > at
> > > >
> > >
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > > > at
> > > >
> > >
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > > > at
> > > >
> > >
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > > > at
> > > >
> > >
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> > > > at
> > > >
> > >
> >
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> > > > at
> > > >
> > >
> >
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
> > > > at
> > > >
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> > > > at java.lang.Thread.run(Thread.java:724)
> > > > Caused by: java.lang.OutOfMemoryError: Requested array size exceeds
> VM
> > > limit
> > > > at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:64)
> > > > at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:37)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.component.ShardFieldSortedHitQueue.<init>(ShardDoc.java:113)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:766)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:625)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:604)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1817)
> > > > at
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
> > > > at
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> > > > ... 13 more
> > >
> > >
> >
> >
> > --
> > Grzegorz Sobczyk
> >
>
--
Grzegorz Sobczyk
Re: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Posted by Erick Erickson <er...@gmail.com>.
What are the memory parameters you start Solr with? The Solr admin page
will tell you how much memory the JVM has.
Also, cut/paste the queries you're running when you see this.
Best
Erick
On Thu, Aug 1, 2013 at 9:50 AM, Grzegorz Sobczyk <gs...@gmail.com> wrote:
> after node starts in log I have only few requests:
> https://gist.github.com/gsobczyk/6131503#file-solr-oom-log
> this error occurred multiple times
>
>
>
>
> On 1 August 2013 15:33, Rafał Kuć <r....@solr.pl> wrote:
>
> > Hello!
> >
> > The exception you've shown tells you that Solr tried to allocate an
> > array that exceeded heap size. Do you use some custom sorts? Did you
> > send large bulks during the time that the exception occurred?
> >
> > --
> > Regards,
> > Rafał Kuć
> > Sematext :: http://sematext.com/ :: Solr - Lucene - ElasticSearch
> >
> > > Today I found in solr logs exception: java.lang.OutOfMemoryError:
> > Requested
> > > array size exceeds VM limit.
> > > At that time memory usage was ~200MB / Xmx3g
> >
> > > Env looks like this:
> > > 3x standalone zK (Java 7)
> > > 3x Solr 4.2.1 on Tomcat (Java 7)
> > > Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux
> > > One Solr and one ZK on single host: lmsiprse01, lmsiprse02, lmsiprse03
> >
> > > Before exception I start restarting Solr:
> > > lmsiprse01:
> > > [2013-08-01 05:23:43]: /etc/init.d/tomcat6-1 stop
> > > [2013-08-01 05:25:09]: /etc/init.d/tomcat6-1 start
> > > lmsiprse02 (leader):
> > > 2013-08-01 05:27:21]: /etc/init.d/tomcat6-1 stop
> > > 2013-08-01 05:29:31]: /etc/init.d/tomcat6-1 start
> > > lmsiprse03:
> > > [2013-08-01 05:25:48]: /etc/init.d/tomcat6-1 stop
> > > [2013-08-01 05:26:42]: /etc/init.d/tomcat6-1 start
> >
> > > and error shows up at 2013 5:27:26 on lmsiprse01
> >
> > > Is anybody knows what happened?
> >
> > > fragment of log looks:
> > > sie 01, 2013 5:27:26 AM org.apache.solr.core.SolrCore execute
> > > INFO: [products] webapp=/solr path=/select
> > >
> >
> params={facet=true&start=0&q=&facet.limit=-1&facet.field=attribute_u-typ&facet.field=attribute_u-gama-kolorystyczna&facet.field=brand_name&wt=javabin&fq=node_id:1056&version=2&rows=0}
> > > hits=1241 status=0 QTime=33
> > > sie 01, 2013 5:27:26 AM org.apache.solr.common.SolrException log
> > > SEVERE: null:java.lang.RuntimeException: java.lang.OutOfMemoryError:
> > > Requested array size exceeds VM limit
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:653)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:366)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
> > > at
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > > at
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > > at
> > >
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > > at
> > >
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > > at
> > >
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > > at
> > >
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > > at
> > >
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > > at
> > >
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> > > at
> > >
> >
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> > > at
> > >
> >
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
> > > at
> > > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> > > at java.lang.Thread.run(Thread.java:724)
> > > Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM
> > limit
> > > at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:64)
> > > at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:37)
> > > at
> > >
> >
> org.apache.solr.handler.component.ShardFieldSortedHitQueue.<init>(ShardDoc.java:113)
> > > at
> > >
> >
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:766)
> > > at
> > >
> >
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:625)
> > > at
> > >
> >
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:604)
> > > at
> > >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)
> > > at
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1817)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> > > ... 13 more
> >
> >
>
>
> --
> Grzegorz Sobczyk
>
Re: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Posted by Grzegorz Sobczyk <gs...@gmail.com>.
after node starts in log I have only few requests:
https://gist.github.com/gsobczyk/6131503#file-solr-oom-log
this error occurred multiple times
On 1 August 2013 15:33, Rafał Kuć <r....@solr.pl> wrote:
> Hello!
>
> The exception you've shown tells you that Solr tried to allocate an
> array that exceeded heap size. Do you use some custom sorts? Did you
> send large bulks during the time that the exception occurred?
>
> --
> Regards,
> Rafał Kuć
> Sematext :: http://sematext.com/ :: Solr - Lucene - ElasticSearch
>
> > Today I found in solr logs exception: java.lang.OutOfMemoryError:
> Requested
> > array size exceeds VM limit.
> > At that time memory usage was ~200MB / Xmx3g
>
> > Env looks like this:
> > 3x standalone zK (Java 7)
> > 3x Solr 4.2.1 on Tomcat (Java 7)
> > Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux
> > One Solr and one ZK on single host: lmsiprse01, lmsiprse02, lmsiprse03
>
> > Before exception I start restarting Solr:
> > lmsiprse01:
> > [2013-08-01 05:23:43]: /etc/init.d/tomcat6-1 stop
> > [2013-08-01 05:25:09]: /etc/init.d/tomcat6-1 start
> > lmsiprse02 (leader):
> > 2013-08-01 05:27:21]: /etc/init.d/tomcat6-1 stop
> > 2013-08-01 05:29:31]: /etc/init.d/tomcat6-1 start
> > lmsiprse03:
> > [2013-08-01 05:25:48]: /etc/init.d/tomcat6-1 stop
> > [2013-08-01 05:26:42]: /etc/init.d/tomcat6-1 start
>
> > and error shows up at 2013 5:27:26 on lmsiprse01
>
> > Is anybody knows what happened?
>
> > fragment of log looks:
> > sie 01, 2013 5:27:26 AM org.apache.solr.core.SolrCore execute
> > INFO: [products] webapp=/solr path=/select
> >
> params={facet=true&start=0&q=&facet.limit=-1&facet.field=attribute_u-typ&facet.field=attribute_u-gama-kolorystyczna&facet.field=brand_name&wt=javabin&fq=node_id:1056&version=2&rows=0}
> > hits=1241 status=0 QTime=33
> > sie 01, 2013 5:27:26 AM org.apache.solr.common.SolrException log
> > SEVERE: null:java.lang.RuntimeException: java.lang.OutOfMemoryError:
> > Requested array size exceeds VM limit
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:653)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:366)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > at
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > at
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > at
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > at
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > at
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > at
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> > at
> >
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> > at
> >
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
> > at
> > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> > at java.lang.Thread.run(Thread.java:724)
> > Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM
> limit
> > at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:64)
> > at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:37)
> > at
> >
> org.apache.solr.handler.component.ShardFieldSortedHitQueue.<init>(ShardDoc.java:113)
> > at
> >
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:766)
> > at
> >
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:625)
> > at
> >
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:604)
> > at
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1817)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> > ... 13 more
>
>
--
Grzegorz Sobczyk
Re: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Posted by Rafał Kuć <r....@solr.pl>.
Hello!
The exception you've shown tells you that Solr tried to allocate an
array that exceeded heap size. Do you use some custom sorts? Did you
send large bulks during the time that the exception occurred?
--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - ElasticSearch
> Today I found in solr logs exception: java.lang.OutOfMemoryError: Requested
> array size exceeds VM limit.
> At that time memory usage was ~200MB / Xmx3g
> Env looks like this:
> 3x standalone zK (Java 7)
> 3x Solr 4.2.1 on Tomcat (Java 7)
> Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux
> One Solr and one ZK on single host: lmsiprse01, lmsiprse02, lmsiprse03
> Before exception I start restarting Solr:
> lmsiprse01:
> [2013-08-01 05:23:43]: /etc/init.d/tomcat6-1 stop
> [2013-08-01 05:25:09]: /etc/init.d/tomcat6-1 start
> lmsiprse02 (leader):
> 2013-08-01 05:27:21]: /etc/init.d/tomcat6-1 stop
> 2013-08-01 05:29:31]: /etc/init.d/tomcat6-1 start
> lmsiprse03:
> [2013-08-01 05:25:48]: /etc/init.d/tomcat6-1 stop
> [2013-08-01 05:26:42]: /etc/init.d/tomcat6-1 start
> and error shows up at 2013 5:27:26 on lmsiprse01
> Is anybody knows what happened?
> fragment of log looks:
> sie 01, 2013 5:27:26 AM org.apache.solr.core.SolrCore execute
> INFO: [products] webapp=/solr path=/select
> params={facet=true&start=0&q=&facet.limit=-1&facet.field=attribute_u-typ&facet.field=attribute_u-gama-kolorystyczna&facet.field=brand_name&wt=javabin&fq=node_id:1056&version=2&rows=0}
> hits=1241 status=0 QTime=33
> sie 01, 2013 5:27:26 AM org.apache.solr.common.SolrException log
> SEVERE: null:java.lang.RuntimeException: java.lang.OutOfMemoryError:
> Requested array size exceeds VM limit
> at
> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:653)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:366)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
> at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:64)
> at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:37)
> at
> org.apache.solr.handler.component.ShardFieldSortedHitQueue.<init>(ShardDoc.java:113)
> at
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:766)
> at
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:625)
> at
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:604)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1817)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> ... 13 more
Re: java.lang.OutOfMemoryError: Requested array size exceeds VM
limit
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Thu, 2013-08-01 at 15:24 +0200, Grzegorz Sobczyk wrote:
> Today I found in solr logs exception: java.lang.OutOfMemoryError: Requested
> array size exceeds VM limit.
> At that time memory usage was ~200MB / Xmx3g
[...]
> Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:64)
> at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:37)
> at
> org.apache.solr.handler.component.ShardFieldSortedHitQueue.<init>(ShardDoc.java:113)
Are you requesting a very large number of results? Integer.MAX_VALUE
perhaps? If so, you need to change that to a more manageable number.
- Toke Eskildsen, State and University Library, Denmark