You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Marc Sturlese <ma...@gmail.com> on 2009/04/01 11:49:43 UTC
performance tests with DataImportHandler and full-import
Hey there,
I am doing performance tests with full-import command from
DataImportHandler. I have configured 20 cores with 1 Gig index each (about a
milion docs per index). If I start doing full-imports indexing from a mysql
table SEQUENCIALY with cron jobs as frequently as possible it will work fine
for about 17 or 20 full-import jobs in total. After that it starts taking
much much longer for each full import... (from 30 min at the begining to 4
hours for the las full-import before crashing) until I get an
OutOfMemoryError java Heap Space.
Apr 1 10:30:47 indexer-03 solr: 63534480 [Thread-536] ERROR
org.apache.solr.handler.dataimport.DataImporter - Full Import failed
Apr 1 10:34:04 indexer-03 solr: 63581229 [http-8080-Processor92] ERROR
org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
Java heap space
Apr 1 10:34:23 indexer-03 solr: 63562764 [Thread-546] WARN
org.apache.solr.handler.dataimport.DocBuilder - GC overhead limit exceeded
Apr 1 10:36:11 indexer-03 solr: 63739903 [http-8080-Processor99] ERROR
org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
Java heap space
Apr 1 10:36:11 indexer-03 solr: 63876821 [http-8080-Processor95] ERROR
org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
Java heap space
Apr 1 10:36:20 indexer-03 solr: 63787787 [Thread-546] ERROR
org.apache.solr.handler.dataimport.DataImporter - Full Import failed
Apr 1 10:40:02 indexer-03 solr: 64073790 [http-8080-Processor100] ERROR
org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
Java heap space
Apr 1 10:40:45 indexer-03 solr: 63991575 [http-8080-Processor96] ERROR
org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
Java heap space
Even I do indexations sequencially I have tryed using a the
ConcurrentGarvageCollector: -XX:+UseConcMarkSweepGC but nothing seems to
change (it's quite logic as I don't index concurrently)
I am runing on
Debian 2.6.26-1-amd64
mysql for database
java version:
java version "1.6.0_12"
Java(TM) SE Runtime Environment (build 1.6.0_12-b04)
Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode)
Any idea why this could be happening?
--
View this message in context: http://www.nabble.com/performance-tests-with-DataImportHandler-and-full-import-tp22823145p22823145.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: performance tests with DataImportHandler and full-import
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
I guess Solr itself is hogging more memory.
May be you can try reloading the core before each import.
On Wed, Apr 1, 2009 at 3:19 PM, Marc Sturlese <ma...@gmail.com> wrote:
>
> Hey there,
> I am doing performance tests with full-import command from
> DataImportHandler. I have configured 20 cores with 1 Gig index each (about a
> milion docs per index). If I start doing full-imports indexing from a mysql
> table SEQUENCIALY with cron jobs as frequently as possible it will work fine
> for about 17 or 20 full-import jobs in total. After that it starts taking
> much much longer for each full import... (from 30 min at the begining to 4
> hours for the las full-import before crashing) until I get an
> OutOfMemoryError java Heap Space.
>
> Apr 1 10:30:47 indexer-03 solr: 63534480 [Thread-536] ERROR
> org.apache.solr.handler.dataimport.DataImporter - Full Import failed
> Apr 1 10:34:04 indexer-03 solr: 63581229 [http-8080-Processor92] ERROR
> org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
> Java heap space
> Apr 1 10:34:23 indexer-03 solr: 63562764 [Thread-546] WARN
> org.apache.solr.handler.dataimport.DocBuilder - GC overhead limit exceeded
> Apr 1 10:36:11 indexer-03 solr: 63739903 [http-8080-Processor99] ERROR
> org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
> Java heap space
> Apr 1 10:36:11 indexer-03 solr: 63876821 [http-8080-Processor95] ERROR
> org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
> Java heap space
> Apr 1 10:36:20 indexer-03 solr: 63787787 [Thread-546] ERROR
> org.apache.solr.handler.dataimport.DataImporter - Full Import failed
> Apr 1 10:40:02 indexer-03 solr: 64073790 [http-8080-Processor100] ERROR
> org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
> Java heap space
> Apr 1 10:40:45 indexer-03 solr: 63991575 [http-8080-Processor96] ERROR
> org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
> Java heap space
>
> Even I do indexations sequencially I have tryed using a the
> ConcurrentGarvageCollector: -XX:+UseConcMarkSweepGC but nothing seems to
> change (it's quite logic as I don't index concurrently)
> I am runing on
> Debian 2.6.26-1-amd64
> mysql for database
> java version:
> java version "1.6.0_12"
> Java(TM) SE Runtime Environment (build 1.6.0_12-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode)
>
> Any idea why this could be happening?
>
>
>
>
> --
> View this message in context: http://www.nabble.com/performance-tests-with-DataImportHandler-and-full-import-tp22823145p22823145.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul