[jira] [Issue Comment Deleted] (SOLR-4356) SOLR 4.1 Out Of Memory error After commit of a few thousand Solr Docs

["Harish Verma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harish Verma updated SOLR-4356:
-------------------------------

    Comment: was deleted

(was: screenshots of Memory Dump)
    
> SOLR 4.1 Out Of Memory error After commit of a few thousand Solr Docs
> ---------------------------------------------------------------------
>
>                 Key: SOLR-4356
>                 URL: https://issues.apache.org/jira/browse/SOLR-4356
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java, Schema and Analysis, Tests
>    Affects Versions: 4.1
>         Environment: OS = Ubuntu 12.04
> Sun JAVA 7
> Max Java Heap Space = 2GB
> Apache Tomcat 7
> Hardware = {Intel core i3, 2GB RAM}
> Average no of fields in a Solr Doc = 100
>            Reporter: Harish Verma
>              Labels: performance, test
>             Fix For: 4.1.1
>
>         Attachments: memorydump1.png, memorydump2.png
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> we are testing solr 4.1 running inside tomcat 7 and java 7 with  following options
> JAVA_OPTS="-Xms256m -Xmx2048m -XX:MaxPermSize=1024m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+ParallelRefProcEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ubuntu/OOM_HeapDump"
> our source code looks like following:
> /**** START *****/
> int noOfSolrDocumentsInBatch = 0;
> for(int i=0 ; i<5000 ; i++) {
>     SolrInputDocument solrInputDocument = getNextSolrInputDocument();
>     server.add(solrInputDocument);
>     noOfSolrDocumentsInBatch += 1;
>     if(noOfSolrDocumentsInBatch == 10) {
>         server.commit();
>         noOfSolrDocumentsInBatch = 0;
>     }
> }
> /**** END *****/
> the method "getNextSolrInputDocument()" generates a solr document with 100 fields (average). Around 50 of the fields are of "text_general" type.
> Some of the "test_general" fields consist of approx 1000 words rest consists of few words. Ouf of total fields there are around 35-40 multivalued fields (not of type "text_general").
> We are indexing all the fields but storing only 8 fields. Out of these 8 fields two are string type, five are long and one is boolean. So our index size is only 394 MB. But the RAM occupied at time of OOM is around 2.5 GB. Why the memory is so high even though the index size is small?
> What is being stored in the memory? Our understanding is that after every commit documents are flushed to the disk.So nothing should remain in RAM after commit.
> We are using the following settings:
> server.commit() set waitForSearcher=true and waitForFlush=true
> solrConfig.xml has following properties set:
> directoryFactory = solr.MMapDirectoryFactory
> maxWarmingSearchers = 1
> text_general data type is being used as supplied in the schema.xml with the solr setup.
> maxIndexingThreads = 8(default)
> <autoCommit><maxTime>15000</maxTime><openSearcher>false</openSearcher></autoCommit>
> We get Java heap Out Of Memory Error after commiting around 3990 solr documents.Some of the snapshots of memory dump from profiler are attached.
> can somebody please suggest what should we do to minimize/optimize the memory consumption in our case with the reasons?
> also suggest what should be optimal values and reason for following parameters of solrConfig.xml 
> useColdSearcher - true/false?
>     maxwarmingsearchers- number
>     spellcheck-on/off?
>     omitNorms=true/false?
>     omitTermFreqAndPositions?
>     mergefactor? we are using default value 10
>     java garbage collection tuning parameters ?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org