DataImport Handler Out of Memory

[Deeksha Sharma <DS...@Flexera.com>]

I am trying to create indexes using dataimport handler (Solr 5.2.1). Data is in mysql db and the number of records are more than 3.5 million. My solr server stops due to OOM (out of memory error). I tried starting solr by giving 12GB of RAM but still no luck.


Also, I see that Solr fetches all the documents in 1 request. Is there a way to configure Solr to stream the data from DB or any other solution somewhere may have tried?


Note: When my records are nearly 2 Million, I am able to create indexes by giving Solr 10GB of RAM.


Your help is appreciated.



Thanks

Deeksha

RE: DataImport Handler Out of Memory

["Allison, Timothy B." <ta...@mitre.org>]

https://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F


-----Original Message-----
From: Deeksha Sharma [mailto:DSharma@Flexera.com] 
Sent: Wednesday, September 27, 2017 1:40 PM
To: solr-user@lucene.apache.org
Subject: DataImport Handler Out of Memory

I am trying to create indexes using dataimport handler (Solr 5.2.1). Data is in mysql db and the number of records are more than 3.5 million. My solr server stops due to OOM (out of memory error). I tried starting solr by giving 12GB of RAM but still no luck.


Also, I see that Solr fetches all the documents in 1 request. Is there a way to configure Solr to stream the data from DB or any other solution somewhere may have tried?


Note: When my records are nearly 2 Million, I am able to create indexes by giving Solr 10GB of RAM.


Your help is appreciated.



Thanks

Deeksha