You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Joe Calderon <ca...@gmail.com> on 2009/10/02 01:14:17 UTC
JVM OOM when using field collapse component
i gotten two different out of memory errors while using the field
collapsing component, using the latest patch (2009-09-26) and the
latest nightly,
has anyone else encountered similar problems? my collection is 5
million results but ive gotten the error collapsing as little as a few
thousand
SEVERE: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:173)
at org.apache.lucene.util.OpenBitSet.ensureCapacityWords(OpenBitSet.java:749)
at org.apache.lucene.util.OpenBitSet.ensureCapacity(OpenBitSet.java:757)
at org.apache.lucene.util.OpenBitSet.expandingWordNum(OpenBitSet.java:292)
at org.apache.lucene.util.OpenBitSet.set(OpenBitSet.java:233)
at org.apache.solr.search.AbstractDocumentCollapser.addCollapsedDoc(AbstractDocumentCollapser.java:402)
at org.apache.solr.search.NonAdjacentDocumentCollapser.doCollapsing(NonAdjacentDocumentCollapser.java:115)
at org.apache.solr.search.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:208)
at org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:98)
at org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:66)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
SEVERE: java.lang.OutOfMemoryError: Java heap space
at org.apache.solr.util.DocSetScoreCollector.<init>(DocSetScoreCollector.java:44)
at org.apache.solr.search.NonAdjacentDocumentCollapser.doQuery(NonAdjacentDocumentCollapser.java:68)
at org.apache.solr.search.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:205)
at org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:98)
at org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:66)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
Re: JVM OOM when using field collapse component
Posted by Joe Calderon <ca...@gmail.com>.
heap space is 4gb set to grow up to 8gb, usage is normally ~1-2gb,
seems to happen within a few searches.
if its just me ill try to isolate it, it could be some other part of
my implementation
thx much
On Fri, Oct 2, 2009 at 1:18 AM, Martijn v Groningen
<ma...@gmail.com> wrote:
> No I have not encountered OOM exception yet with current field collapse patch.
> How large is your configured JVM heap space (-Xmx)? Field collapsing
> requires more memory then regular searches so. Does Solr run out of
> memory during the first search(es) or does it run out of memory after
> a while when it performed quite a few field collapse searches?
>
> I see that you are also using the collapse.includeCollapsedDocs.fl
> parameter for your search. This feature will require more memory then
> a normal field collapse search.
>
> I normally give the Solr instance a heap space of 1024M when having an
> index of a few million.
>
> Martijn
>
> 2009/10/2 Joe Calderon <ca...@gmail.com>:
>> i gotten two different out of memory errors while using the field
>> collapsing component, using the latest patch (2009-09-26) and the
>> latest nightly,
>>
>> has anyone else encountered similar problems? my collection is 5
>> million results but ive gotten the error collapsing as little as a few
>> thousand
>>
>> SEVERE: java.lang.OutOfMemoryError: Java heap space
>> at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:173)
>> at org.apache.lucene.util.OpenBitSet.ensureCapacityWords(OpenBitSet.java:749)
>> at org.apache.lucene.util.OpenBitSet.ensureCapacity(OpenBitSet.java:757)
>> at org.apache.lucene.util.OpenBitSet.expandingWordNum(OpenBitSet.java:292)
>> at org.apache.lucene.util.OpenBitSet.set(OpenBitSet.java:233)
>> at org.apache.solr.search.AbstractDocumentCollapser.addCollapsedDoc(AbstractDocumentCollapser.java:402)
>> at org.apache.solr.search.NonAdjacentDocumentCollapser.doCollapsing(NonAdjacentDocumentCollapser.java:115)
>> at org.apache.solr.search.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:208)
>> at org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:98)
>> at org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:66)
>> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148)
>> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387)
>> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
>> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> at org.mortbay.jetty.Server.handle(Server.java:326)
>> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
>>
>> SEVERE: java.lang.OutOfMemoryError: Java heap space
>> at org.apache.solr.util.DocSetScoreCollector.<init>(DocSetScoreCollector.java:44)
>> at org.apache.solr.search.NonAdjacentDocumentCollapser.doQuery(NonAdjacentDocumentCollapser.java:68)
>> at org.apache.solr.search.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:205)
>> at org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:98)
>> at org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:66)
>> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
>> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148)
>> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387)
>> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
>> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> at org.mortbay.jetty.Server.handle(Server.java:326)
>> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
>>
>
>
>
> --
> Met vriendelijke groet,
>
> Martijn van Groningen
>
Re: JVM OOM when using field collapse component
Posted by Martijn v Groningen <ma...@gmail.com>.
No I have not encountered OOM exception yet with current field collapse patch.
How large is your configured JVM heap space (-Xmx)? Field collapsing
requires more memory then regular searches so. Does Solr run out of
memory during the first search(es) or does it run out of memory after
a while when it performed quite a few field collapse searches?
I see that you are also using the collapse.includeCollapsedDocs.fl
parameter for your search. This feature will require more memory then
a normal field collapse search.
I normally give the Solr instance a heap space of 1024M when having an
index of a few million.
Martijn
2009/10/2 Joe Calderon <ca...@gmail.com>:
> i gotten two different out of memory errors while using the field
> collapsing component, using the latest patch (2009-09-26) and the
> latest nightly,
>
> has anyone else encountered similar problems? my collection is 5
> million results but ive gotten the error collapsing as little as a few
> thousand
>
> SEVERE: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:173)
> at org.apache.lucene.util.OpenBitSet.ensureCapacityWords(OpenBitSet.java:749)
> at org.apache.lucene.util.OpenBitSet.ensureCapacity(OpenBitSet.java:757)
> at org.apache.lucene.util.OpenBitSet.expandingWordNum(OpenBitSet.java:292)
> at org.apache.lucene.util.OpenBitSet.set(OpenBitSet.java:233)
> at org.apache.solr.search.AbstractDocumentCollapser.addCollapsedDoc(AbstractDocumentCollapser.java:402)
> at org.apache.solr.search.NonAdjacentDocumentCollapser.doCollapsing(NonAdjacentDocumentCollapser.java:115)
> at org.apache.solr.search.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:208)
> at org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:98)
> at org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:66)
> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
>
> SEVERE: java.lang.OutOfMemoryError: Java heap space
> at org.apache.solr.util.DocSetScoreCollector.<init>(DocSetScoreCollector.java:44)
> at org.apache.solr.search.NonAdjacentDocumentCollapser.doQuery(NonAdjacentDocumentCollapser.java:68)
> at org.apache.solr.search.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:205)
> at org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:98)
> at org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:66)
> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
>
--
Met vriendelijke groet,
Martijn van Groningen