Does sharding improve or degrade performance?

[Piyush Kunal <pi...@myntra.com>]

We did the following change:

1. Previously we had 1 shard and 32 replicas for 1.2million documents of
size 5 GB.
2. We changed it to 4 shards and 8 replicas for 1.2 million documents of
size 5GB

We have a combined RPM of around 20k rpm for solr.

But unfortunately we saw a degrade in performance with RTs going insanely
high when we moved to setup 2.

What could be probable reasons and how it can be fixed?

Re: Does sharding improve or degrade performance?

[Shawn Heisey <ap...@elyograg.org>]

On 12/12/2016 1:14 PM, Piyush Kunal wrote:
> We did the following change:
>
> 1. Previously we had 1 shard and 32 replicas for 1.2million documents of
> size 5 GB.
> 2. We changed it to 4 shards and 8 replicas for 1.2 million documents of
> size 5GB

How many machines and shards per machine were you running in both
situations?  For either setup, I would recommend at least 32 machines,
where each one handles exactly one shard replica.  For the latter setup,
you may need even more machines, so there are more replicas.

> We have a combined RPM of around 20k rpm for solr.

Twenty thousand queries per minute is over 300 per second.  This is a
very high query rate, which is going to require many replicas.  Your
replica count has gone down significantly with the change you made.

> But unfortunately we saw a degrade in performance with RTs going insanely
> high when we moved to setup 2.

With such a high query rate, I'm not really surprised that this caused
the performance to go down, even if you actually do have 32 machines. 
Distributed queries are a two-phase process where the coordinating node
sends individual queries to each shard to find out how to sort the
sub-results into one final result, and then sends a second query to
relevant shards to request the individual documents it needs for the
result.  The total number of individual queries goes up significantly.

Before Solr was doing one query for one result.  Now it is doing between
five and nine queries for one result (the initial query from your client
to the coordinating node, the first query to each of the four shards,
and then a possible second query to each shard).  If the number of
search hits is more than zero, it will be at least six queries.  This is
why one shard is preferred for high query rates if you can fit the whole
index into one shard.  Five gigabytes is a pretty small Solr index.

Sharding is most effective when the query rate is low, because Solr can
take advantage of idle CPUs.  It makes it possible to have a much larger
index.  A high query rate means that there are no idle CPUs.

Thanks,
Shawn

Re: Does sharding improve or degrade performance?

[Erick Erickson <er...@gmail.com>]

Sharding adds inevitable overhead. Particularly
each request, rather than being serviced on a
single replica has to send out a first request
to each replica, get the ID and sort criteria back,
then send out a second request to get the actual docs.

Especially if you're asking for a lot of rows this can get
very expensive. And you're now serving your queries
on 1/4 of the machines. In the first setup, an incoming
request was completely serviced on 1 node. Now you're
requiring 4 nodes to participate.

Sharding is always a second choice and always has
some overhead. As long as your QTimes are
acceptable, you should stick with only a single replica.

Best,
Erick

On Mon, Dec 12, 2016 at 12:14 PM, Piyush Kunal <pi...@myntra.com> wrote:
> All our shards and replicas reside on different machines with 16GB RAM and
> 4 cores.
>
> On Tue, Dec 13, 2016 at 1:44 AM, Piyush Kunal <pi...@myntra.com>
> wrote:
>
>> We did the following change:
>>
>> 1. Previously we had 1 shard and 32 replicas for 1.2million documents of
>> size 5 GB.
>> 2. We changed it to 4 shards and 8 replicas for 1.2 million documents of
>> size 5GB
>>
>> We have a combined RPM of around 20k rpm for solr.
>>
>> But unfortunately we saw a degrade in performance with RTs going insanely
>> high when we moved to setup 2.
>>
>> What could be probable reasons and how it can be fixed?
>>

Re: Does sharding improve or degrade performance?

[Piyush Kunal <pi...@myntra.com>]

All our shards and replicas reside on different machines with 16GB RAM and
4 cores.

On Tue, Dec 13, 2016 at 1:44 AM, Piyush Kunal <pi...@myntra.com>
wrote:

> We did the following change:
>
> 1. Previously we had 1 shard and 32 replicas for 1.2million documents of
> size 5 GB.
> 2. We changed it to 4 shards and 8 replicas for 1.2 million documents of
> size 5GB
>
> We have a combined RPM of around 20k rpm for solr.
>
> But unfortunately we saw a degrade in performance with RTs going insanely
> high when we moved to setup 2.
>
> What could be probable reasons and how it can be fixed?
>