Reg: Solr OOM

[BigData dev <bi...@gmail.com>]

Hi,
We are seeing Solr OOM exceptions, when we issue a query to solr collection.

The below is the stack trace we are seeing:
at org.apache.lucene.codecs.blocktree.IntersectTermsEnumFrame.load(Lorg/apache/lucene/util/BytesRef;)V (IntersectTermsEnumFrame.java:195)
 at org.apache.lucene.codecs.blocktree.IntersectTermsEnum.pushFrame(I)Lorg/apache/lucene/codecs/blocktree/IntersectTermsEnumFrame; (IntersectTermsEnum.java:211)
 at org.apache.lucene.codecs.blocktree.IntersectTermsEnum._next()Lorg/apache/lucene/util/BytesRef; (IntersectTermsEnum.java:665)
 at org.apache.lucene.codecs.blocktree.IntersectTermsEnum.next()Lorg/apache/lucene/util/BytesRef; (IntersectTermsEnum.java:500)
 at org.apache.lucene.index.ExitableDirectoryReader$ExitableTermsEnum.next()Lorg/apache/lucene/util/BytesRef; (ExitableDirectoryReader.java:185)
 at org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.collectTerms(Lorg/apache/lucene/index/LeafReaderContext;Lorg/apache/lucene/index/TermsEnum;Ljava/util/List;)Z (MultiTermQueryConstantScoreWrapper.java:120)
 at org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(Lorg/apache/lucene/index/LeafReaderContext;)Lorg/apache/lucene/search/MultiTermQueryConstantScoreWrapper$WeightOrDocIdSet; (MultiTermQueryConstantScoreWrapper.java:147)
 at org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.bulkScorer(Lorg/apache/lucene/index/LeafReaderContext;)Lorg/apache/lucene/search/BulkScorer; (MultiTermQueryConstantScoreWrapper.java:194)
 at org.apache.lucene.search.IndexSearcher.search(Ljava/util/List;Lorg/apache/lucene/search/Weight;Lorg/apache/lucene/search/Collector;)V (IndexSearcher.java:666)
 at org.apache.lucene.search.IndexSearcher.search(Lorg/apache/lucene/search/Query;Lorg/apache/lucene/search/Collector;)V (IndexSearcher.java:473)
 at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(Lorg/apache/solr/search/QueryResult;Lorg/apache/lucene/search/Query;Lorg/apache/lucene/search/Collector;Lorg/apache/solr/search/QueryCommand;Lorg/apache/solr/search/DelegatingCollector;)V (SolrIndexSearcher.java:242)
 at org.apache.solr.search.SolrIndexSearcher.getDocListNC(Lorg/apache/solr/search/QueryResult;Lorg/apache/solr/search/QueryCommand;)V (SolrIndexSearcher.java:1803)
 at org.apache.solr.search.SolrIndexSearcher.getDocListC(Lorg/apache/solr/search/QueryResult;Lorg/apache/solr/search/QueryCommand;)V (SolrIndexSearcher.java:1620)
 at org.apache.solr.search.SolrIndexSearcher.search(Lorg/apache/solr/search/QueryResult;Lorg/apache/solr/search/QueryCommand;)Lorg/apache/solr/search/QueryResult; (SolrIndexSearcher.java:617)
 at org.apache.solr.handler.component.QueryComponent.process(Lorg/apache/solr/handler/component/ResponseBuilder;)V (QueryComponent.java:531)
 at org.apache.solr.handler.component.SearchHandler.handleRequestBody(Lorg/apache/solr/request/SolrQueryRequest;Lorg/apache/solr/response/SolrQueryResponse;)V (SearchHandler.java:295)
 at org.apache.solr.handler.RequestHandlerBase.handleRequest(Lorg/apache/solr/request/SolrQueryRequest;Lorg/apache/solr/response/SolrQueryResponse;)V (RequestHandlerBase.java:153)
 at org.apache.solr.core.SolrCore.execute(Lorg/apache/solr/request/SolrRequestHandler;Lorg/apache/solr/request/SolrQueryRequest;Lorg/apache/solr/response/SolrQueryResponse;)V (SolrCore.java:2213)
 at org.apache.solr.servlet.HttpSolrCall.execute(Lorg/apache/solr/response/SolrQueryResponse;)V (HttpSolrCall.java:654)
 at org.apache.solr.servlet.HttpSolrCall.call()Lorg/apache/solr/servlet/SolrDispatchFilter$Action; (HttpSolrCall.java:460)
 at org.apache.solr.servlet.SolrDispatchFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;Z)V (SolrDispatchFilter.java:303)
 at org.apache.solr.servlet.SolrDispatchFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (SolrDispatchFilter.java:254)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (ServletHandler.java:1668)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ServletHandler.java:581)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ScopedHandler.java:143)
 at org.eclipse.jetty.security.SecurityHandler.handle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (SecurityHandler.java:548)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (SessionHandler.java:226)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ContextHandler.java:1160)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ServletHandler.java:511)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (SessionHandler.java:185)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ContextHandler.java:1092)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ScopedHandler.java:141)
 at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ContextHandlerCollection.java:213)
 at org.eclipse.jetty.server.handler.HandlerCollection.handle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (HandlerCollection.java:119)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (HandlerWrapper.java:134)
 at org.eclipse.jetty.server.Server.handle(Lorg/eclipse/jetty/server/HttpChannel;)V (Server.java:518)
 at org.eclipse.jetty.server.HttpChannel.handle()Z (HttpChannel.java:308)
 at org.eclipse.jetty.server.HttpConnection.onFillable()V (HttpConnection.java:244)
 at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded()V (AbstractConnection.java:273)
 at org.eclipse.jetty.io.FillInterest.fillable()V (FillInterest.java:95)
 at org.eclipse.jetty.io.SelectChannelEndPoint$2.run()V (SelectChannelEndPoint.java:93)
 at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun()V (ExecuteProduceConsume.java:246)
 at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run()V (ExecuteProduceConsume.java:156)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Ljava/lang/Runnable;)V (QueuedThreadPool.java:654)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run()V (QueuedThreadPool.ja


And from the head-dump analysis, we see the 2 major causes for OOM are:





For the first one, we are not sure of sudden spike in the memory, where as for second one from the jira(https://issues.apache.org/jira/browse/SOLR-12009) we got information that we need to enable docValues. 

Thank You @varun Thacker for information. If you have any more update on the first problem, could  you please provide some information.

Could you please help in this issue, if any one has faced similar issue or same experience with Solr OOM exceptions in this area?

Thanks,
Bharat

Re: Reg: Solr OOM

[Shawn Heisey <ap...@elyograg.org>]

When Varun suggested you contact the mailing list in SOLR-12009, he 
meant the solr-user mailing list.  This mailing list is for discussion 
about development of Lucene and Solr, not for support, bugs, etc.  Those 
kinds of discussions belong on the user list.

If this discussion requires more interaction, it should be moved to the 
user list, or possibly to the IRC channel.

On 2/20/2018 7:33 PM, BigData dev wrote:
> We are seeing Solr OOM exceptions, when we issue a query to solr 
> collection.
>
> The below is the stack trace we are seeing:

The stacktrace for OOME frequently has absolutely no relation to the 
part of the program that has filled the heap.  It is merely the part of 
the program that was executing at the moment that no more heap was 
available.  I couldn't tell if you were sharing the stacktrace from the 
logged exception or from the heap analysis.

If I'm reading the heap analysis correctly, it shows about 15GB of 
memory allocated by an array of IntersectTermsEnumFrameobjects. 
Unfortunately, this class has no javadocs, and I am not familiar enough 
with the Lucene API to know what it's used for.  The class visibility 
appears to be package, not public, so it seems to be an internal 
implementation detail, which is probably why there are no javadocs.

> And from the head-dump analysis, we see the 2 major causes for OOM are:

<snip>

> For the first one, we are not sure of sudden spike in the memory, 
> where as for second one from the 
> jira(https://issues.apache.org/jira/browse/SOLR-12009) we got 
> information that we need to enable docValues.

I'm guessing that one of two things is happening here, and it could be both:

1) Your index is so big that you're going to need a larger heap.
2) The types of queries you are sending to Solr are very memory-hungry.

For the second problem, if you can reduce the query complexity, you 
might not need as much heap.  Adding docValues as you were advised 
(which requires a complete reindex) might also help, depending on the 
nature of the queries.

For the first problem, more information is necessary. Here's a list of 
questions:

What is your max heap?  I'm guessing it is at least 18GB, based on the 
two highlighted entries from heap analysis.  It may be even larger, but 
I can't tell for sure.  18GB is a pretty big heap.

How much total memory is in the server?  What OS is it running?

Is there software other than Solr running on the machine?

How many documents are in the core?

How big is the core on disk?

If there are multiple cores in the Solr instance, I will need the 
previous two pieces of information for all of them.

Do you know how many queries per second Solr is handling? If there is 
ongoing indexing, do you know how many documents per second are being 
added/updated, and do you know how often commits that open a new 
searcher are happening?  Sharing solrconfig.xml would be a good 
proactive step, and we may need to see Solr's logfile.  Use a paste 
website or a file-sharing service for this. Attachments rarely make it 
to the list.

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org