NPE on auto-warming and out of memory issues

[briand <br...@gmail.com>]

We are experiencing OOM issues with a SOLR index that has about 12G of
indexed data with 2GB allocated to the JVM.   We first see these type of
messages in the log: 

Oct 18, 2007 10:25:00 AM org.apache.solr.core.SolrException log
SEVERE: Error during auto-warming of key:+(search_place_type:citi
search_place_type:attract search_place_type:airport
search_place_type:univers):java.lang
.OutOfMemoryError: Java heap space

Oct 18, 2007 10:25:07 AM org.apache.solr.core.SolrException log
SEVERE: Error during auto-warming of
key:+search_place_type:busi:java.lang.OutOfMemoryError: Java heap space

After a log of continuing messages like the ones above we'll see log series
of messages like this:

Oct 18, 2007 10:55:58 AM org.apache.solr.core.SolrException log
SEVERE: Error during auto-warming of
key:org.apache.solr.search.QueryResultKey@9f798e04:java.lang.NullPointerException
        at org.apache.lucene.index.Term.compareTo(Term.java:91)
        at
org.apache.lucene.index.TermInfosReader.getIndexOffset(TermInfosReader.java:112)
        at
org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:147)
        at
org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:342)
        at org.apache.lucene.index.MultiReader.docFreq(MultiReader.java:220)
        at
org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:87)
        at org.apache.lucene.search.Similarity.idf(Similarity.java:459)
        at
org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:44)
        at
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:145)
        at
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.<init>(DisjunctionMaxQuery.java:99)
        at
org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:161)
        at
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:187)
        at
org.apache.lucene.search.BooleanQuery$BooleanWeight2.<init>(BooleanQuery.java:342)
        at
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
        at org.apache.lucene.search.Query.weight(Query.java:95)
        at org.apache.lucene.search.Searcher.createWeight(Searcher.java:171)
        at org.apache.lucene.search.Searcher.search(Searcher.java:118)
        at org.apache.lucene.search.Searcher.search(Searcher.java:97)
        at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:888)
        at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:805)
        at
org.apache.solr.search.SolrIndexSearcher.access$100(SolrIndexSearcher.java:60)
        at
org.apache.solr.search.SolrIndexSearcher$2.regenerateItem(SolrIndexSearcher.java:251)
        at org.apache.solr.search.LRUCache.warm(LRUCache.java:193)
        at
org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:1385)
        at org.apache.solr.core.SolrCore$1.call(SolrCore.java:488)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
        at java.util.concurrent.FutureTask.run(FutureTask.java:123)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)

We have our configuration set for autowarmCount to:

<filterCache class="solr.LRUCache" size="512" initialSize="512"
autowarmCount="256"/>

<queryResultCache class="solr.LRUCache" size="512" initialSize="512"
autowarmCount="256"/>

<documentCache class="solr.LRUCache" size="512" initialSize="512"
autowarmCount="0"/>

It appears that autowarming has something to do with our OOM.   I've read
that if you set the autowarmCount to 0 or a lower value then the first
request after a commit may take some time.    Definitely willing to try
setting the autowarmCount to a lower value.   Does anyone have any other
ideas to help with this autowarming issue?   Thanks. 
-- 
View this message in context: http://www.nabble.com/NPE-on-auto-warming-and-out-of-memory-issues-tf4654164.html#a13297439
Sent from the Solr - User mailing list archive at Nabble.com.