You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by vijeshnair <vi...@gmail.com> on 2013/01/21 08:04:51 UTC
Data import handler start bulging the memory after completing 1
million
<http://lucene.472066.n3.nabble.com/file/n4034949/ScreenShot034.jpg>
You may refer this snapshot to get an understanding of the resource
consumption. I am trying to index a total number of 13 million documents
from MySQL to SOLR. First 1 million document's got completed very smoothly
in the first 2 minutes, later it started bulging the RAM, and it never gets
released in between. I have tried all the known tricks and tactics, still
failing to rectify this issue. I am using SOLR 4.0, using DIH to import from
MySQL 5.5 DB. Any help will be much appreciated, and I am trying to find any
loop hole in my schema and config files.
--
View this message in context: http://lucene.472066.n3.nabble.com/Data-import-handler-start-bulging-the-memory-after-completing-1-million-tp4034949.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Data import handler start bulging the memory after completing 1
million
Posted by vijeshnair <vi...@gmail.com>.
Well, here is the update. I have managed to run the full import for whole
12.2 million documents, now I have the full catalog in SOLR. The issue now
is the number of segments, which is pretty huge, so I wanted to optimize it.
But the memory which SOLR occupied during the indexing process is still not
freed up, hence I took a heap dump of it. It's been noticed that 75% of the
heap space is being occupied by byte[], it's actually pretty huge. What
could be the reason here? am I missing some omit norms setting here, so that
norms for some of the fields unnecessarily getting preserved in the memory?
Please advise, so that I can rectify the issue. Please find the snapshot for
the heap dump, object lists.
<http://lucene.472066.n3.nabble.com/file/n4035021/ScreenShot035.jpg>
--
View this message in context: http://lucene.472066.n3.nabble.com/Data-import-handler-start-bulging-the-memory-after-completing-1-million-tp4034949p4035021.html
Sent from the Solr - User mailing list archive at Nabble.com.