You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Alex Benjamen <ab...@friendster.com> on 2008/01/28 23:44:25 UTC
SEVERE: java.lang.OutOfMemoryError: Java heap space
We're now running several solr instances on quad-cores and getting fairly good RPS even on
the largest index (26MM documents) after implementing faceted queries. Things are looking good
except for this OutOfMemoryError which occurs every 2 hrs at peak.
Note: I have browsed, searched the forums for this error and followed the most common advice of
increasing the memory allocation for the JVM:
/usr/bin/java -DSTOP.PORT=8805 -DSTOP.KEY=solrstop -Xmx3584M -Xms1024M -jar start.jar
I have also reduced the autowarmcounts to 100 from 2000. Still, after running a couple of hrs or
so solr/ping returns 500 error with (see below) the error in the log. We're running on jetty and
the index size is 3.7Gb with 26MM documents. I don't see this error in other solr instances which
have an index of 1.3Gb and 8MM docs (those simply hang, which is another topic, but it's more rare)
I could allocate more physical memory, but I can't seem to increase the -Xmx option to 3800 I get
an error : "Could not reserve enough space for object heap", even though I have more than 4Gb free.
(We're running on Intel quad core 64bit) When I try strace I'm seeing mmap2 errors.
So, question which comes to mind - can this problem be solved otherwise? I'm sure others have used
larger indexes than this. What other settings can I tweak to get rid of this error... I'm currently running
a healthcheck and simply restart the solr instance when I get a 500 error on the solr/ping. But this is
ugly and bad for cache... Any ideas?
Thanks in advance!
-Alex
Jan 24, 2008 3:25:44 PM org.apache.solr.core.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
at org.apache.solr.util.OpenBitSet.clone(OpenBitSet.java:539)
at org.apache.solr.search.DocSetBase.intersection(DocSet.java:201)
at org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:568)
at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:788)
at org.apache.solr.search.SolrIndexSearcher.getDocList(SolrIndexSearcher.java:698)
at org.apache.solr.request.StandardRequestHandler.handleRequestBody(StandardRequestHandler.java:122)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:77)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:658)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:191)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:159)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:821)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
RE: SEVERE: java.lang.OutOfMemoryError: Java heap space
Posted by Alex Benjamen <ab...@friendster.com>.
Thanks to all who responded. Things are running well! The IBM version
of the JRE for Intel 64 seems to run good, and the stalling issue has dissappeared.
(when the solr instance stops responding and freezes up)
What I learned is that solr is a great product but needs "tuning" to fit the usage.
Changing some config settings can make the difference between a usable and
unusable system.
I guess for most smaller indeces things will run fine out of the box, but if you
have a large index then you have to play around a bit
-Alex
------------------------------------------------------------------------------------------------
>Install the AMD64 version. (Confusingly, AMD64 is a spec name for
>EM64T, which is now what both AMD and Intel use)
>If that still doesn't work, is it possible that your machine/kernel is
>not set up to support 64 bit?
I was confused by the naming convention. Seems to work fine now, well,
I mean fine that it can be started with a larger heap. We'll see how it runs overnight
Thanks again,
Alex
RE: SEVERE: java.lang.OutOfMemoryError: Java heap space
Posted by Alex Benjamen <ab...@friendster.com>.
>Install the AMD64 version. (Confusingly, AMD64 is a spec name for
>EM64T, which is now what both AMD and Intel use)
>If that still doesn't work, is it possible that your machine/kernel is
>not set up to support 64 bit?
I was confused by the naming convention. Seems to work fine now, well,
I mean fine that it can be started with a larger heap. We'll see how it runs overnight
Thanks again,
Alex
Re: SEVERE: java.lang.OutOfMemoryError: Java heap space
Posted by Brian Whitman <br...@variogr.am>.
>
> But on Intel, where I'm having the problem it shows:
> java version "1.6.0_10-ea"
> Java(TM) SE Runtime Environment (build 1.6.0_10-ea-b10)
> Java HotSpot(TM) Server VM (build 11.0-b09, mixed mode)
>
> I can't seem to find the Intel 64 bit JDK binary, can you pls. send
> me the link?
> I was downloading from here:
> http://download.java.net/jdk6/
Install the AMD64 version. (Confusingly, AMD64 is a spec name for
EM64T, which is now what both AMD and Intel use)
If that still doesn't work, is it possible that your machine/kernel is
not set up to support 64 bit?
RE: SEVERE: java.lang.OutOfMemoryError: Java heap space
Posted by Alex Benjamen <ab...@friendster.com>.
>We use 10GB of ram in one of our solr installs. You need to make sure
>your java is 64 bit though. Alex, what does your java -version show?
>Mine shows
>java version "1.6.0_03"
>Java(TM) SE Runtime Environment (build 1.6.0_03-b05)
>Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_03-b05, mixed mode)
>And I run it with -mx10000m -ms5000m
Brian, good point - I'm running on both AMD and Intel platform, and on AMD
it does say :
$ java -version
java version "1.6.0_10-ea"
Java(TM) SE Runtime Environment (build 1.6.0_10-ea-b07)
Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_10-ea-b07, mixed mode)
But on Intel, where I'm having the problem it shows:
java version "1.6.0_10-ea"
Java(TM) SE Runtime Environment (build 1.6.0_10-ea-b10)
Java HotSpot(TM) Server VM (build 11.0-b09, mixed mode)
I can't seem to find the Intel 64 bit JDK binary, can you pls. send me the link?
I was downloading from here:
http://download.java.net/jdk6/
Thanks!
-Alex
Re: SEVERE: java.lang.OutOfMemoryError: Java heap space
Posted by Brian Whitman <br...@variogr.am>.
On Jan 28, 2008, at 7:06 PM, Leonardo Santagada wrote:
> On 28/01/2008, at 20:44, Alex Benjamen wrote:
>>
>
>> I could allocate more physical memory, but I can't seem to increase
>> the -Xmx option to 3800 I get
>> an error : "Could not reserve enough space for object heap", even
>> though I have more than 4Gb free.
>> (We're running on Intel quad core 64bit) When I try strace I'm
>> seeing mmap2 errors.
> ]
>
> I don't know much about java... but can you get any program to map
> more than 4gb of memory? I know windows has hard limits on how much
> memory you can map to one process and linux I think has some limit
> too. Of course it can be configured but maybe it is just a system
> configuration problem.
We use 10GB of ram in one of our solr installs. You need to make sure
your java is 64 bit though. Alex, what does your java -version show?
Mine shows
java version "1.6.0_03"
Java(TM) SE Runtime Environment (build 1.6.0_03-b05)
Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_03-b05, mixed mode)
And I run it with -mx10000m -ms5000m
Re: SEVERE: java.lang.OutOfMemoryError: Java heap space
Posted by Leonardo Santagada <sa...@gmail.com>.
On 28/01/2008, at 20:44, Alex Benjamen wrote:
> Note: I have browsed, searched the forums for this error and
> followed the most common advice of
> increasing the memory allocation for the JVM:
> /usr/bin/java -DSTOP.PORT=8805 -DSTOP.KEY=solrstop -Xmx3584M -
> Xms1024M -jar start.jar
> [snip]
> I could allocate more physical memory, but I can't seem to increase
> the -Xmx option to 3800 I get
> an error : "Could not reserve enough space for object heap", even
> though I have more than 4Gb free.
> (We're running on Intel quad core 64bit) When I try strace I'm
> seeing mmap2 errors.
I don't know much about java... but can you get any program to map
more than 4gb of memory? I know windows has hard limits on how much
memory you can map to one process and linux I think has some limit
too. Of course it can be configured but maybe it is just a system
configuration problem.
But still there should be a way to tell jetty to only have a certain
amount of threads/process/requests at a time, so you never risk the
problem of getting out of memory problems.
--
Leonardo Santagada