RE: OutofMemory on Highlightling

["Gargate, Siddharth" <sg...@ptc.com>]

Also adding solr dev group to this mail.

http://www.nabble.com/OutofMemory-on-Highlightling-td23075331.html

I am not able to figure out the issue

-----Original Message-----
From: Gargate, Siddharth [mailto:sgargate@ptc.com] 
Sent: Friday, April 24, 2009 10:46 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling

I am not sure whether lazy loading should help solve this problem. I
have set enableLazyFieldLoading to true but it is not helping.

I went through the code and observed that
DefaultSolrHighlighter.doHighlighting is reading all the documents and
the fields for highlighting (In my case, 1 MB stored field is read for
all documents). 

Also I am confused over the following code in SolrIndexSearcher.doc()
method

if(!enableLazyFieldLoading || fields == null) {
      d = searcher.getIndexReader().document(i);
    } else {
      d = searcher.getIndexReader().document(i, 
             new SetNonLazyFieldSelector(fields));
    }

Are we setting the fields as NonLazy even if lazy loading is enabled?

Thanks,
Siddharth

-----Original Message-----
From: Gargate, Siddharth [mailto:sgargate@ptc.com] 
Sent: Wednesday, April 22, 2009 11:12 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling

Here is the stack trace

SEVERE: java.lang.OutOfMemoryError: Java heap space
        at
java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133)
        at java.lang.StringCoding.decode(StringCoding.java:173)
        at java.lang.String.<init>(String.java:444)
        at
org.apache.lucene.store.IndexInput.readString(IndexInput.java:125)
        at
org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:390)
        at
org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:230)
        at
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:892)
        at
org.apache.lucene.index.MultiSegmentReader.document(MultiSegmentReader.j
ava:277)
        at
org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:176
)
        at
org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:457)
        at
org.apache.solr.search.SolrIndexSearcher.readDocs(SolrIndexSearcher.java
:482)
        at
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultS
olrHighlighter.java:253)
        at
org.apache.solr.handler.component.HighlightComponent.process(HighlightCo
mponent.java:84)
        at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(Search
Handler.java:195)
        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB
ase.java:131)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
        at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja
va:303)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j
ava:232)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:235)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:206)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
e.java:233)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
e.java:191)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
:128)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
:102)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
java:109)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:2
86)
        at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:84
5)
        at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(
Http11Protocol.java:583)
        at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:619)



-----Original Message-----
From: Gargate, Siddharth [mailto:sgargate@ptc.com] 
Sent: Wednesday, April 22, 2009 9:29 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling

I tried disabling the documentCache but still the same issue. 

<documentCache
      class="solr.LRUCache"
      size="0"
      initialSize="0"
      autowarmCount="0"/>



-----Original Message-----
From: Koji Sekiguchi [mailto:koji@r.email.ne.jp] 
Sent: Monday, April 20, 2009 4:38 PM
To: solr-user@lucene.apache.org
Subject: Re: OutofMemory on Highlightling

Gargate, Siddharth wrote:
> Anybody facing the same issue? Following is my configuration
> ...
> <field name="content" type="text" indexed="true" stored="false"
> multiValued="true"/>
> <field name="teaser" type="text" indexed="false" stored="true"/>
> <copyField source="content" dest="teaser" maxChars="1000000" />
> ...
>
> ...
> <requestHandler name="standard" class="solr.SearchHandler"
> default="true">
>      <lst name="defaults">
>        <str name="echoParams">explicit</str>
>        
>        <int name="rows">500</int>
> 	   <str name="hl">true</str>
> 		<str name="fl">id,score</str>
> 	    <str name="hl.fl">teaser</str>
> 		<str name="hl.alternateField">teaser</str>
> 		<int name="hl.fragsize">200</int>
> 		<int name="hl.maxAlternateFieldLength">200</int>
> 		<int name="hl.maxAnalyzedChars">500</int>
>      </lst>
>   </requestHandler>
> ...
>
> Search works fine if I disable highlighting and it brings 500 results.
> But if I enable hightlighting and set the no. of rows to just 20 I get
> OOME.
>
>   
How about switching documentCache off?

Koji