Re: DIH OutOfMemoryError?

[Lance Norskog <go...@gmail.com>]

Make sure streaming is on.
Try using autoCommit in solrconfig.xml. This will push documents out
of memory onto disk at a regular interval.

On Thu, Mar 31, 2011 at 8:51 AM, Markus Jelsma
<ma...@openindex.io> wrote:
> Try splitting the files into smaller chunks. It'll help.
>
>> Hi,
>>
>> I'm trying to index a big XML file (800Mo) using DIH, but i'm getting an
>> OutOfMemoryError!
>>
>> I've got 2048mo of RAM on this server, obviously it's not enough... How
>> much RAM is recomended for indexing big files?
>>
>> Thanks for your help
>>
>>
>> Here is the error from DIH mode verbose:
>>
>> <str name="EXCEPTION">java.lang.ClassCastException:
>> java.lang.OutOfMemoryError cannot be cast to java.lang.Exception
>>      at
>> org.apache.solr.handler.dataimport.DebugLogger.log(DebugLogger.java:139)
>>      at
>> org.apache.solr.handler.dataimport.SolrWriter.log(SolrWriter.java:237)
>>      at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java
>> :422) at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java
>> :383) at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:24
>> 2) at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180)
>>      at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.j
>> ava:331) at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:38
>> 9) at
>> org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(Data
>> ImportHandler.java:203) at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase
>> .java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>>      at
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:
>> 338) at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java
>> :241) at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio
>> nFilterChain.java:235) at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC
>> hain.java:206) at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j
>> ava:233) at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j
>> ava:191) at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12
>> 7) at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10
>> 2) at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav
>> a:109) at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>>      at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
>>      at
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Htt
>> p11Protocol.java:588) at
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>      at java.lang.Thread.run(Thread.java:636)
>> </str>
>



-- 
Lance Norskog
goksron@gmail.com

Re: DIH OutOfMemoryError?

[stockii <st...@googlemail.com>]

Make sure streaming is on. 
--> how to check ? 

-----
------------------------------- System ----------------------------------------

One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, 
1 Core with 31 Million Documents other Cores < 100.000

- Solr1 for Search-Requests - commit every Minute  - 5GB Xmx
- Solr2 for Update-Request  - delta every Minute - 4GB Xmx
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-OutOfMemoryError-tp2759013p2811270.html
Sent from the Solr - User mailing list archive at Nabble.com.