Re: OOM at MultiSegmentReader.norms

[Otis Gospodnetic <ot...@yahoo.com>]

Vivek,
No need for any RAMBuffer stuff, just use the appropriate ramBufferSizeMB value in solrconfig.xml.  This will only help with indexing, though!


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----
> From: vivek sar <vi...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Monday, March 30, 2009 12:53:21 PM
> Subject: Re: OOM at MultiSegmentReader.norms
> 
> Thanks Otis and Mike.
> 
> I'm indexing total of 9 fields, with 5 having norms turned on. I think
> I may not need it and will try use the omitNorms for them.
> 
> How do I make use of RAMBuffer in Solr? I couldn't find anything on
> this on the Wiki - any pointer?
> 
> Thanks,
> -vivek
> 
> On Sat, Mar 28, 2009 at 1:09 AM, Michael McCandless
> wrote:
> > Still, 1024M ought to be enough to load one field's norms (how many
> > fields have norms?).  If you do things requiring FieldCache that'll
> > also consume RAM.
> >
> > It's also possible you're hitting this bug (false OOME) in Sun's JRE:
> >
> >  http://issues.apache.org/jira/browse/LUCENE-1566
> >
> > Feel free to go vote for it!
> >
> > Mike
> >
> > On Fri, Mar 27, 2009 at 10:11 PM, Otis Gospodnetic
> > wrote:
> >>
> >> That's a tiny heap.  Part of it is used for indexing, too.  And the fact that 
> your heap is so small shows you are not really making use of that nice 
> ramBufferSizeMB setting. :)
> >>
> >> Also, use omitNorms="true" for fields that don't need norms (if their types 
> don't already do that).
> >>
> >> Otis
> >> --
> >> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >>
> >>
> >>
> >> ----- Original Message ----
> >>> From: vivek sar 
> >>> To: solr-user@lucene.apache.org
> >>> Sent: Friday, March 27, 2009 6:15:59 PM
> >>> Subject: OOM at MultiSegmentReader.norms
> >>>
> >>> Hi,
> >>>
> >>>    I've index of size 50G (around 100 million documents) and growing -
> >>> around 2000 records (1 rec = 500 byes) are being written every second
> >>> continuously. If I make any search on this index I get OOM. I'm using
> >>> default cache settings (512,512,256) in the solrconfig.xml. The search
> >>> is using the admin interface (returning 10 rows) with no sorting,
> >>> faceting or highlighting. Max heap size is 1024m.
> >>>
> >>> Mar 27, 2009 9:13:41 PM org.apache.solr.common.SolrException log
> >>> SEVERE: java.lang.OutOfMemoryError: Java heap space
> >>>         at
> >>> 
> org.apache.lucene.index.MultiSegmentReader.norms(MultiSegmentReader.java:335)
> >>>         at
> >>> org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:69)
> >>>         at 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:132)
> >>>         at org.apache.lucene.search.Searcher.search(Searcher.java:126)
> >>>         at org.apache.lucene.search.Searcher.search(Searcher.java:105)
> >>>         at
> >>> 
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:966)
> >>>         at
> >>> 
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838)
> >>>         at
> >>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
> >>>         at
> >>> 
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
> >>>         at
> >>> 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
> >>>         at
> >>> 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> >>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
> >>>         at
> >>> 
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
> >>>         at
> >>> 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
> >>>         at
> >>> 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> >>>         at
> >>> 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> >>>         at
> >>> 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> >>>         at
> >>> 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> >>>         at
> >>> 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
> >>>         at
> >>> 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> >>>         at
> >>> 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> >>>
> >>> What could be the problem?
> >>>
> >>> Thanks,
> >>> -vivek
> >>
> >>
> >