You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Torben Greulich <to...@s24.com> on 2013/12/16 10:34:30 UTC
OutOfMemoryError in RamUsageEstimator in solr 4.6
Hi,
we get a OutOfMemoryError in RamUsageEstimator and are a little bit
confused about the error.
We are using solr 4.6 and are confused about the Lucene42DocValuesProducer.
We checked current solr code and found that Lucene42NormsFormat will be
returned as NormFormat in Lucene46Codec and so the Lucene42DocValuesProducer
will be returned. Is there any special reason not to have a higher version
of NormsFormat?
This is our StackTrace:
SolrException|null:java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.util.RamUsageEstimator$IdentityHashSet.allocateBuffers(RamUsageEstimator.java:762)
at
org.apache.lucene.util.RamUsageEstimator$IdentityHashSet.expandAndRehash(RamUsageEstimator.java:736)
at
org.apache.lucene.util.RamUsageEstimator$IdentityHashSet.add(RamUsageEstimator.java:678)
at
org.apache.lucene.util.RamUsageEstimator.measureObjectSize(RamUsageEstimator.java:437)
at
org.apache.lucene.util.RamUsageEstimator.sizeOf(RamUsageEstimator.java:350)
at
org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.ramBytesUsed(Lucene42DocValuesProducer.java:194)
at
org.apache.lucene.index.SegmentCoreReaders.ramBytesUsed(SegmentCoreReaders.java:195)
at
org.apache.lucene.index.SegmentReader.ramBytesUsed(SegmentReader.java:558)
at
org.apache.solr.handler.admin.LukeRequestHandler.getIndexHeapUsed(LukeRequestHandler.java:579)
at
org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:558)
at
org.apache.solr.handler.admin.CoreAdminHandler.getCoreStatus(CoreAdminHandler.java:1044)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:685)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:167)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
...
Please contact me, if there are further questions.
Best regards Torben Greulich
--
Torben Greulich · Entwickler
Backend Development
shopping24 internet group
Wandsbeker Straße 3-7 · 22172 Hamburg
Telefon: +49 (0) 40 6461 xxxx · Fax: +49 (0) 40 6461 7879
torben.greulich@s24.com · www.s24.com
AG Hamburg HRB 63371
vertreten durch Dr. Björn Schäfers und Martin Mildner
Re: OutOfMemoryError in RamUsageEstimator in solr 4.6
Posted by Torben Greulich <to...@s24.com>.
Hi Shawn,
thanks for your reply. But we don't think that this is really a OOM error,
because we already increased the heap to 64gb and the OOM occurs at a usage
of 30-40gb. So solr would allocate more than 20gb at once. this sounds a
little bit too much.
Furthermore we found Lucene45DocValuesProducer and wondered why it exists
and isn't used here. Lucene45DocValuesProducer.ramBytesUsed() also looks
different to Lucene42DocValuesProducer.ramBytesUsed()
Best Regards
Torben
2013/12/16 Shawn Heisey <so...@elyograg.org>
> On 12/16/2013 2:34 AM, Torben Greulich wrote:
> > we get a OutOfMemoryError in RamUsageEstimator and are a little bit
> > confused about the error.
> > We are using solr 4.6 and are confused about the
> Lucene42DocValuesProducer.
> > We checked current solr code and found that Lucene42NormsFormat will be
> > returned as NormFormat in Lucene46Codec and so the
> Lucene42DocValuesProducer
> > will be returned. Is there any special reason not to have a higher
> version
> > of NormsFormat?
>
> The format for that part of the index apparently hasn't changed since
> Lucene 4.2. Something else must have changed about the default index
> format, or I expect that there would not be a 4.6 specific Codec object.
> From what I've seen, the verion number encoded in the class name
> doesn't change until there's a new class needed for the default codec.
>
> A java OutOfMemoryError means that you need to increase the max heap
> when you start the program. It looks like you're trying to get a core
> status, so this is probably happening when using the admin UI. To put
> it quite simply, your Java max heap is not big enough to handle what
> Solr has been asked to do.
>
> http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
>
> Thanks,
> Shawn
>
>
--
Torben Greulich · Entwickler
Backend Development
shopping24 internet group
Wandsbeker Straße 3-7 · 22172 Hamburg
Telefon: +49 (0) 40 6461 xxxx · Fax: +49 (0) 40 6461 7879
torben.greulich@s24.com · www.s24.com
AG Hamburg HRB 63371
vertreten durch Dr. Björn Schäfers und Martin Mildner
Re: OutOfMemoryError in RamUsageEstimator in solr 4.6
Posted by Shawn Heisey <so...@elyograg.org>.
On 12/16/2013 2:34 AM, Torben Greulich wrote:
> we get a OutOfMemoryError in RamUsageEstimator and are a little bit
> confused about the error.
> We are using solr 4.6 and are confused about the Lucene42DocValuesProducer.
> We checked current solr code and found that Lucene42NormsFormat will be
> returned as NormFormat in Lucene46Codec and so the Lucene42DocValuesProducer
> will be returned. Is there any special reason not to have a higher version
> of NormsFormat?
The format for that part of the index apparently hasn't changed since
Lucene 4.2. Something else must have changed about the default index
format, or I expect that there would not be a 4.6 specific Codec object.
From what I've seen, the verion number encoded in the class name
doesn't change until there's a new class needed for the default codec.
A java OutOfMemoryError means that you need to increase the max heap
when you start the program. It looks like you're trying to get a core
status, so this is probably happening when using the admin UI. To put
it quite simply, your Java max heap is not big enough to handle what
Solr has been asked to do.
http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
Thanks,
Shawn