SOLRJ commit question - SOLR 4.3.1

[Learner <bb...@gmail.com>]

Hi,

I am adding around 100 million records to SOLR using SOLRJ. I am not
performing commit operation until I add all the documents to SOLR. I see
that my program adds the docs very fast (1 million per minute) for around 18
million documents which is expected result but after 18 million records
index time keeps climbing higher and higher..

I am using ConcurrentUpdateSolrServer for adding the docs,

I have set the queueSize="200", thread count of 8.
I am adding document batch of size 250 (I add the document array once size
reaches 250).

My system configuration:
24 cores
32GB RAM
110 GB HD

Am I doing anything wrong in terms of specifying the queue size / document
batch size?

Can someone please help me fix this index issue?





--
View this message in context: http://lucene.472066.n3.nabble.com/SOLRJ-commit-question-SOLR-4-3-1-tp4072868.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLRJ commit question - SOLR 4.3.1

[Erick Erickson <er...@gmail.com>]

The queue size is high, but I doubt that's your issue. Here's what
I'd do:
1> check the Solr server. Is it CPU bound? I/O bound? You have to
identify where the resources are being spent before you get to
implementing a solution.

Are you using SolrCloud? If so, not committing until all 100M docs is
going to grow your transaction log. Specifically it'll store all 100M docs
in the tlog rather than just the last N. Try setting your autocommit to
some relatively short interval (say 1 minute, or even 15 seconds) with
openSearcher=false.

Otherwise, we need some more detail on what's going on exactly, the
solr log may help here.

Best
Erick

On Mon, Jun 24, 2013 at 3:49 PM, Learner <bb...@gmail.com> wrote:
> Hi,
>
> I am adding around 100 million records to SOLR using SOLRJ. I am not
> performing commit operation until I add all the documents to SOLR. I see
> that my program adds the docs very fast (1 million per minute) for around 18
> million documents which is expected result but after 18 million records
> index time keeps climbing higher and higher..
>
> I am using ConcurrentUpdateSolrServer for adding the docs,
>
> I have set the queueSize="200", thread count of 8.
> I am adding document batch of size 250 (I add the document array once size
> reaches 250).
>
> My system configuration:
> 24 cores
> 32GB RAM
> 110 GB HD
>
> Am I doing anything wrong in terms of specifying the queue size / document
> batch size?
>
> Can someone please help me fix this index issue?
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/SOLRJ-commit-question-SOLR-4-3-1-tp4072868.html
> Sent from the Solr - User mailing list archive at Nabble.com.