Sharding and Replication clarification

[Asif <ta...@gmail.com>]

Hi,

I had questions on implementation of Sharding and Replication features of
Solr/Cloud.

1. I noticed that when sharding is enabled for a collection - individual
requests are sent to each node serving as a shard.

2. Replication too follows above strategy of sending individual documents
to the nodes serving as a replica.

I am working with a system that requires massive number of writes - I have
noticed that due to above reason - the cloud eventually starts to fail
(Even though I am using a ensemble).

I do understand the reason behind individual updates - but why not batch
them up or give a option to batch N updates in either of the above case - I
did come across a presentation that talked about batching 10 updates for
replication at least, but I do not think this is the case.
- Asif

Re: Sharding and Replication clarification

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Wed, Jun 19, 2013 at 11:12 PM, Asif <ta...@gmail.com> wrote:
> Hi,
>
> I had questions on implementation of Sharding and Replication features of
> Solr/Cloud.
>
> 1. I noticed that when sharding is enabled for a collection - individual
> requests are sent to each node serving as a shard.

Yes, search requests are distributed to a member of each logical
shard. If you know the shard that you want to search you can specify a
shards=shard1,shard2 parameter to limit searches to those shards only.

>
> 2. Replication too follows above strategy of sending individual documents
> to the nodes serving as a replica.

Yes, full documents are replicated in SolrCloud for normal updates
instead of index fragments as used to be the case in non-cloud
replication. The old replication method is still used for recovery.

>
> I am working with a system that requires massive number of writes - I have
> noticed that due to above reason - the cloud eventually starts to fail
> (Even though I am using a ensemble).
>
> I do understand the reason behind individual updates - but why not batch
> them up or give a option to batch N updates in either of the above case - I
> did come across a presentation that talked about batching 10 updates for
> replication at least, but I do not think this is the case.
> - Asif

If you send a batch update request then the requests to replicas will
be batched. I think the default is that 10 documents per replica node
are batched together. But if you send one document at a time then the
replication will also happen one document at a time.


--
Regards,
Shalin Shekhar Mangar.