You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by jyn7 <jy...@gmail.com> on 2011/06/17 03:14:25 UTC
SOlR -- Out of Memory exception
We just started using SOLR. I am trying to load a single file with 20 million
records into SOLR using the CSV uploader. I keep getting and out of Memory
after loading 7 million records. Here is the config:
<autoCommit>
<maxDocs>10000</maxDocs>
<maxTime>60000</maxTime>
I also encountered a LockObtainFailedException
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out:
NativeFSLock@D:\work\solr\.\data\index\write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at
org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1097)
So I changed the lockType to SIngle, now again I am getting an Out of
Memory Exception. I also increased the JVM heap space to 2048M but still
getting an Out of Memory.
--
View this message in context: http://lucene.472066.n3.nabble.com/SOlR-Out-of-Memory-exception-tp3074636p3074636.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOlR -- Out of Memory exception
Posted by jyn7 <jy...@gmail.com>.
I commented the autocommit option and tried uploading the file (a smaller
file now 5 million records) and I hit an oom again:
Jun 17, 2011 2:32:59 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
--
View this message in context: http://lucene.472066.n3.nabble.com/SOlR-Out-of-Memory-exception-tp3074636p3077812.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOlR -- Out of Memory exception
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Fri, Jun 17, 2011 at 1:30 AM, pravesh <su...@yahoo.com> wrote:
> If you are sending whole CSV in a single HTTP request using curl, why not
> consider sending it in smaller chunks?
Smaller chunks should not matter - Solr streams from the input (i.e.
the whole thing is not buffered in memory).
It could be related to autoCommit. Commits may be stacking up faster
than can be handled. I'd recommend getting rid of autocommit if
possible, or at a minimum get rid of the maxDocs based autocommit.
Incremental updates can use commitWithin to guarantee a
time-of-visibility, and bulk updates like this CSV upload normally
shouldn't commit until the end.
-Yonik
http://www.lucidimagination.com
Re: SOlR -- Out of Memory exception
Posted by jyn7 <jy...@gmail.com>.
I did that , but when I split them into 5 mill records, the first file went
through fine, when I started processing the second file SOLR hit an OOM
again:
org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.<init>(FreqProxTermsWriterPerField.java:184)
at
org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.newInstance(FreqProxTermsWriterPerField.java:194)
at
org.apache.lucene.index.ParallelPostingsArray.grow(ParallelPostingsArray.java:48)
at
org.apache.lucene.index.TermsHashPerField.growParallelPostingsArray(TermsHashPerField.java:137)
at
org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:440)
at
org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:169)
at
org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:248)
--
View this message in context: http://lucene.472066.n3.nabble.com/SOlR-Out-of-Memory-exception-tp3074636p3076610.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOlR -- Out of Memory exception
Posted by pravesh <su...@yahoo.com>.
If you are sending whole CSV in a single HTTP request using curl, why not
consider sending it in smaller chunks?
--
View this message in context: http://lucene.472066.n3.nabble.com/SOlR-Out-of-Memory-exception-tp3074636p3075091.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOlR -- Out of Memory exception
Posted by jyn7 <jy...@gmail.com>.
Yes Eric, after changing the lock type to Single, I got an OOM after loading
5.5 million records. I am using the curl command to upload the csv.
--
View this message in context: http://lucene.472066.n3.nabble.com/SOlR-Out-of-Memory-exception-tp3074636p3074765.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOlR -- Out of Memory exception
Posted by Erick Erickson <er...@gmail.com>.
Hmmmm, are you still getting your OOM after 7M records? Or some larger
number? And how are you using the CSV uploader?
Best
Erick
On Thu, Jun 16, 2011 at 9:14 PM, jyn7 <jy...@gmail.com> wrote:
> We just started using SOLR. I am trying to load a single file with 20 million
> records into SOLR using the CSV uploader. I keep getting and out of Memory
> after loading 7 million records. Here is the config:
>
> <autoCommit>
> <maxDocs>10000</maxDocs>
> <maxTime>60000</maxTime>
> I also encountered a LockObtainFailedException
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out:
> NativeFSLock@D:\work\solr\.\data\index\write.lock
> at org.apache.lucene.store.Lock.obtain(Lock.java:84)
> at
> org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1097)
>
> So I changed the lockType to SIngle, now again I am getting an Out of
> Memory Exception. I also increased the JVM heap space to 2048M but still
> getting an Out of Memory.
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/SOlR-Out-of-Memory-exception-tp3074636p3074636.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>