SolrCloud batch updates

["michael.boom" <my...@yahoo.com>]

I'm currently using a SolrCoud setup and I index my data using a couple of
in-house indexing clients.
The clients process some files and post json messages containing added
documents in batches.
Initially my batch size was 100k docs and the post request took about 20-30
secs.
I switched to 10k batches and now the updates are much faster but also more
in number.

My commit settings are :
- autocommit - 45s / 100k docs, openSearcher=false 
- softAutoCommit - every 3 minutes

I'm trying to figure out which one is preferable - bigger batches, rare or
smaller batches, often? And why?
Which are the background operations that take place after posting docs? 
At which point does the replication kick in - after commit or after update?






-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-batch-updates-tp4098463.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud batch updates

[Anshum Gupta <an...@anshumgupta.net>]

Hi Michael,

Here's a good post by Erick Erickson about understanding commits and
transaction logs in Solr.
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

About the replication, as soon as you post an update, here's what happens:
1. The update gets routed to the correct leader
2. The leader writes it to it's transaction log
3. Leader forwards the updates to the replicas.
4. When the replicas respond in positive about the update being successful,
the leader returns a success message for the update.

Hope that helps.


On Wed, Oct 30, 2013 at 9:06 PM, michael.boom <my...@yahoo.com> wrote:

> I'm currently using a SolrCoud setup and I index my data using a couple of
> in-house indexing clients.
> The clients process some files and post json messages containing added
> documents in batches.
> Initially my batch size was 100k docs and the post request took about 20-30
> secs.
> I switched to 10k batches and now the updates are much faster but also more
> in number.
>
> My commit settings are :
> - autocommit - 45s / 100k docs, openSearcher=false
> - softAutoCommit - every 3 minutes
>
> I'm trying to figure out which one is preferable - bigger batches, rare or
> smaller batches, often? And why?
> Which are the background operations that take place after posting docs?
> At which point does the replication kick in - after commit or after update?
>
>
>
>
>
>
> -----
> Thanks,
> Michael
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/SolrCloud-batch-updates-tp4098463.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 

Anshum Gupta
http://www.anshumgupta.net