You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Saksham Gupta <sa...@indiamart.com.INVALID> on 2023/09/13 08:32:29 UTC
Optimal Sharding Strategy for Solr Cloud v8.10
Hi All,
I have been trying to reduce the response time of solr cloud(v8.10, 8
nodes). To achieve this, I have tried increasing the number of shards of
solr cloud which can help reduce data size on each shard thereby reducing
response time.
I have encountered a few questions regarding sharding strategy:
1. How to decide the ideal number of shards? Is there a minimum or maximum
number of shards which should be used?
2. What is the minimum size of a shard after which reducing the size
further won't have any effect on the response time (as time taken by other
factors like data aggregation will compensate for that) ?
3. Is there some maximum limit to the size of data that should be kept in a
shard?
As of now we have 8 shards each on a separate node with ~25 gb of
data(15-16 million docs) present on each shard. Please advise me of the
standard approaches to define the number of shards and shard size. Thanks
in advance.
Re: Optimal Sharding Strategy for Solr Cloud v8.10
Posted by Jan Høydahl <ja...@cominvent.com>.
Yes, if your average query touches just too many documents (such as huge OR queries) and has some processing that needs to touch each hit (scoring, result transformation, highlighting mm), then simply splitting the elephant with shards may help. Or if you ask for 100 facets and your facets are slow, you could perhaps use facet.threads to speed up that part. Or if you use grouping you could try collapse. Etc etc. We need to know more about your data, queries and use case to answer what the cure might be.
Jan
> 13. sep. 2023 kl. 16:22 skrev Walter Underwood <wu...@wunderwood.org>:
>
> This is all great advice.
>
> There is no optimal number of shards. I’ve run clusters with 4 shards, we currently have one cluster with 96 shards and one with 320 shards. The next one we build out will probably not be sharded.
>
> With long queries, I’ve usually seen a roughly linear speedup with sharding. Double the shards, halve the response time.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>> On Sep 13, 2023, at 4:48 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>>
>> Hi,
>>
>> There are no hard rules wrt sharding, it often comes down to measuring and experimenting for your workload.
>>
>> There are other things to consider than shard size. Why are the queries slow? How many rows do you ask for? Do you use faceting? Grouping?
>> You have 25Gb of data on each of the 8 nodes/shards. Now, how much RAM does each node have, and how much RAM did you allocate to Solr/Java?
>> A common mistake is to allocate too much ram/heap to Solr to you don't get any virtual memory caching in Linux.
>> Say you have 32Gb of physical RAM on the nodes. Then do not give 30 of those to Solr. Instead give 8Gb to Solr and let 24Gb be available for disk caching.
>>
>> Other things to consider is to look at whether your queries can be optimized by rewriting them to more efficient equivalents. Sometimes, Solr-level caches can also help.
>>
>> Wrt shards efficiency: If you already have 8 shards, it is not much more expensive to go to 16, but you increase the risk of a single failure affecting your requests...
>>
>> Jan
>>
>>> 13. sep. 2023 kl. 10:32 skrev Saksham Gupta <sa...@indiamart.com.INVALID>:
>>>
>>> Hi All,
>>>
>>> I have been trying to reduce the response time of solr cloud(v8.10, 8
>>> nodes). To achieve this, I have tried increasing the number of shards of
>>> solr cloud which can help reduce data size on each shard thereby reducing
>>> response time.
>>>
>>>
>>> I have encountered a few questions regarding sharding strategy:
>>>
>>> 1. How to decide the ideal number of shards? Is there a minimum or maximum
>>> number of shards which should be used?
>>>
>>> 2. What is the minimum size of a shard after which reducing the size
>>> further won't have any effect on the response time (as time taken by other
>>> factors like data aggregation will compensate for that) ?
>>>
>>> 3. Is there some maximum limit to the size of data that should be kept in a
>>> shard?
>>>
>>>
>>> As of now we have 8 shards each on a separate node with ~25 gb of
>>> data(15-16 million docs) present on each shard. Please advise me of the
>>> standard approaches to define the number of shards and shard size. Thanks
>>> in advance.
>>
>
Re: Optimal Sharding Strategy for Solr Cloud v8.10
Posted by Wei <we...@gmail.com>.
@Walter,
Just curious for 320 shards and 2.5 billion docs, how is the query
throughput? The internal distributed traffic would be huge due to the fan
out.
Regards,
Wei
On Thu, Sep 14, 2023 at 7:36 AM Walter Underwood <wu...@wunderwood.org>
wrote:
> I would have thought it would be a huge hassle, but it isn’t. We build a
> cluster with ArgoCD and Kubernetes deployments. The nodes all report to
> DataDog. The only real bother is that it is hard to go directly to the
> admin UI on a specific node, something doesn’t work right with the
> hostnames and permissions from outside.
>
> Oh, and we run blue/green clusters, so there are two of these beasts.
>
> The graphical display in the admin UI is pretty impressive. Here is a view
> of part of it.
>
>
> https://www.dropbox.com/scl/fi/99xfgek24qocowhft6q7b/Screenshot-2023-09-14-at-7.27.16-AM.png?rlkey=nmjyrl9z0n92lgidfei45vq4q&dl=0
>
> The collection currently has about 2.5 billion documents. When I worked at
> Infoseek, our index of the entire web was 12 million documents.
>
> This is at LexisNexis.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
> > On Sep 13, 2023, at 11:47 PM, Bernd Fehling <
> bernd.fehling@uni-bielefeld.de> wrote:
> >
> > @Walter,
> >
> > how on earth are you monitoring all vital Solr Cloud Parameters for 320
> shards?
> >
> > Regards,
> > Bernd
> >
> >
> > Am 13.09.23 um 16:22 schrieb Walter Underwood:
> >> This is all great advice.
> >> There is no optimal number of shards. I’ve run clusters with 4 shards,
> we currently have one cluster with 96 shards and one with 320 shards. The
> next one we build out will probably not be sharded.
> >> With long queries, I’ve usually seen a roughly linear speedup with
> sharding. Double the shards, halve the response time.
> >> wunder
> >> Walter Underwood
> >> wunder@wunderwood.org
> >> http://observer.wunderwood.org/ (my blog)
> >>> On Sep 13, 2023, at 4:48 AM, Jan Høydahl <ja...@cominvent.com>
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> There are no hard rules wrt sharding, it often comes down to measuring
> and experimenting for your workload.
> >>>
> >>> There are other things to consider than shard size. Why are the
> queries slow? How many rows do you ask for? Do you use faceting? Grouping?
> >>> You have 25Gb of data on each of the 8 nodes/shards. Now, how much RAM
> does each node have, and how much RAM did you allocate to Solr/Java?
> >>> A common mistake is to allocate too much ram/heap to Solr to you don't
> get any virtual memory caching in Linux.
> >>> Say you have 32Gb of physical RAM on the nodes. Then do not give 30 of
> those to Solr. Instead give 8Gb to Solr and let 24Gb be available for disk
> caching.
> >>>
> >>> Other things to consider is to look at whether your queries can be
> optimized by rewriting them to more efficient equivalents. Sometimes,
> Solr-level caches can also help.
> >>>
> >>> Wrt shards efficiency: If you already have 8 shards, it is not much
> more expensive to go to 16, but you increase the risk of a single failure
> affecting your requests...
> >>>
> >>> Jan
> >>>
> >>>> 13. sep. 2023 kl. 10:32 skrev Saksham Gupta <
> saksham.gupta@indiamart.com.INVALID>:
> >>>>
> >>>> Hi All,
> >>>>
> >>>> I have been trying to reduce the response time of solr cloud(v8.10, 8
> >>>> nodes). To achieve this, I have tried increasing the number of shards
> of
> >>>> solr cloud which can help reduce data size on each shard thereby
> reducing
> >>>> response time.
> >>>>
> >>>>
> >>>> I have encountered a few questions regarding sharding strategy:
> >>>>
> >>>> 1. How to decide the ideal number of shards? Is there a minimum or
> maximum
> >>>> number of shards which should be used?
> >>>>
> >>>> 2. What is the minimum size of a shard after which reducing the size
> >>>> further won't have any effect on the response time (as time taken by
> other
> >>>> factors like data aggregation will compensate for that) ?
> >>>>
> >>>> 3. Is there some maximum limit to the size of data that should be
> kept in a
> >>>> shard?
> >>>>
> >>>>
> >>>> As of now we have 8 shards each on a separate node with ~25 gb of
> >>>> data(15-16 million docs) present on each shard. Please advise me of
> the
> >>>> standard approaches to define the number of shards and shard size.
> Thanks
> >>>> in advance.
> >>>
> >
> > --
> > *************************************************************
> > Bernd Fehling Bielefeld University Library
> > Dipl.-Inform. (FH) LibTec - Library Technology
> > Universitätsstr. 25 and Knowledge Management
> > 33615 Bielefeld
> > Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
> > https://www.ub.uni-bielefeld.de/~befehl/
> >
> > BASE - Bielefeld Academic Search Engine - www.base-search.net
> > *************************************************************
>
>
Re: Optimal Sharding Strategy for Solr Cloud v8.10
Posted by Walter Underwood <wu...@wunderwood.org>.
I would have thought it would be a huge hassle, but it isn’t. We build a cluster with ArgoCD and Kubernetes deployments. The nodes all report to DataDog. The only real bother is that it is hard to go directly to the admin UI on a specific node, something doesn’t work right with the hostnames and permissions from outside.
Oh, and we run blue/green clusters, so there are two of these beasts.
The graphical display in the admin UI is pretty impressive. Here is a view of part of it.
https://www.dropbox.com/scl/fi/99xfgek24qocowhft6q7b/Screenshot-2023-09-14-at-7.27.16-AM.png?rlkey=nmjyrl9z0n92lgidfei45vq4q&dl=0
The collection currently has about 2.5 billion documents. When I worked at Infoseek, our index of the entire web was 12 million documents.
This is at LexisNexis.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Sep 13, 2023, at 11:47 PM, Bernd Fehling <be...@uni-bielefeld.de> wrote:
>
> @Walter,
>
> how on earth are you monitoring all vital Solr Cloud Parameters for 320 shards?
>
> Regards,
> Bernd
>
>
> Am 13.09.23 um 16:22 schrieb Walter Underwood:
>> This is all great advice.
>> There is no optimal number of shards. I’ve run clusters with 4 shards, we currently have one cluster with 96 shards and one with 320 shards. The next one we build out will probably not be sharded.
>> With long queries, I’ve usually seen a roughly linear speedup with sharding. Double the shards, halve the response time.
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>> On Sep 13, 2023, at 4:48 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>>>
>>> Hi,
>>>
>>> There are no hard rules wrt sharding, it often comes down to measuring and experimenting for your workload.
>>>
>>> There are other things to consider than shard size. Why are the queries slow? How many rows do you ask for? Do you use faceting? Grouping?
>>> You have 25Gb of data on each of the 8 nodes/shards. Now, how much RAM does each node have, and how much RAM did you allocate to Solr/Java?
>>> A common mistake is to allocate too much ram/heap to Solr to you don't get any virtual memory caching in Linux.
>>> Say you have 32Gb of physical RAM on the nodes. Then do not give 30 of those to Solr. Instead give 8Gb to Solr and let 24Gb be available for disk caching.
>>>
>>> Other things to consider is to look at whether your queries can be optimized by rewriting them to more efficient equivalents. Sometimes, Solr-level caches can also help.
>>>
>>> Wrt shards efficiency: If you already have 8 shards, it is not much more expensive to go to 16, but you increase the risk of a single failure affecting your requests...
>>>
>>> Jan
>>>
>>>> 13. sep. 2023 kl. 10:32 skrev Saksham Gupta <sa...@indiamart.com.INVALID>:
>>>>
>>>> Hi All,
>>>>
>>>> I have been trying to reduce the response time of solr cloud(v8.10, 8
>>>> nodes). To achieve this, I have tried increasing the number of shards of
>>>> solr cloud which can help reduce data size on each shard thereby reducing
>>>> response time.
>>>>
>>>>
>>>> I have encountered a few questions regarding sharding strategy:
>>>>
>>>> 1. How to decide the ideal number of shards? Is there a minimum or maximum
>>>> number of shards which should be used?
>>>>
>>>> 2. What is the minimum size of a shard after which reducing the size
>>>> further won't have any effect on the response time (as time taken by other
>>>> factors like data aggregation will compensate for that) ?
>>>>
>>>> 3. Is there some maximum limit to the size of data that should be kept in a
>>>> shard?
>>>>
>>>>
>>>> As of now we have 8 shards each on a separate node with ~25 gb of
>>>> data(15-16 million docs) present on each shard. Please advise me of the
>>>> standard approaches to define the number of shards and shard size. Thanks
>>>> in advance.
>>>
>
> --
> *************************************************************
> Bernd Fehling Bielefeld University Library
> Dipl.-Inform. (FH) LibTec - Library Technology
> Universitätsstr. 25 and Knowledge Management
> 33615 Bielefeld
> Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
> https://www.ub.uni-bielefeld.de/~befehl/
>
> BASE - Bielefeld Academic Search Engine - www.base-search.net
> *************************************************************
Re: Optimal Sharding Strategy for Solr Cloud v8.10
Posted by Bernd Fehling <be...@uni-bielefeld.de>.
@Walter,
how on earth are you monitoring all vital Solr Cloud Parameters for 320 shards?
Regards,
Bernd
Am 13.09.23 um 16:22 schrieb Walter Underwood:
> This is all great advice.
>
> There is no optimal number of shards. I’ve run clusters with 4 shards, we currently have one cluster with 96 shards and one with 320 shards. The next one we build out will probably not be sharded.
>
> With long queries, I’ve usually seen a roughly linear speedup with sharding. Double the shards, halve the response time.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>> On Sep 13, 2023, at 4:48 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>>
>> Hi,
>>
>> There are no hard rules wrt sharding, it often comes down to measuring and experimenting for your workload.
>>
>> There are other things to consider than shard size. Why are the queries slow? How many rows do you ask for? Do you use faceting? Grouping?
>> You have 25Gb of data on each of the 8 nodes/shards. Now, how much RAM does each node have, and how much RAM did you allocate to Solr/Java?
>> A common mistake is to allocate too much ram/heap to Solr to you don't get any virtual memory caching in Linux.
>> Say you have 32Gb of physical RAM on the nodes. Then do not give 30 of those to Solr. Instead give 8Gb to Solr and let 24Gb be available for disk caching.
>>
>> Other things to consider is to look at whether your queries can be optimized by rewriting them to more efficient equivalents. Sometimes, Solr-level caches can also help.
>>
>> Wrt shards efficiency: If you already have 8 shards, it is not much more expensive to go to 16, but you increase the risk of a single failure affecting your requests...
>>
>> Jan
>>
>>> 13. sep. 2023 kl. 10:32 skrev Saksham Gupta <sa...@indiamart.com.INVALID>:
>>>
>>> Hi All,
>>>
>>> I have been trying to reduce the response time of solr cloud(v8.10, 8
>>> nodes). To achieve this, I have tried increasing the number of shards of
>>> solr cloud which can help reduce data size on each shard thereby reducing
>>> response time.
>>>
>>>
>>> I have encountered a few questions regarding sharding strategy:
>>>
>>> 1. How to decide the ideal number of shards? Is there a minimum or maximum
>>> number of shards which should be used?
>>>
>>> 2. What is the minimum size of a shard after which reducing the size
>>> further won't have any effect on the response time (as time taken by other
>>> factors like data aggregation will compensate for that) ?
>>>
>>> 3. Is there some maximum limit to the size of data that should be kept in a
>>> shard?
>>>
>>>
>>> As of now we have 8 shards each on a separate node with ~25 gb of
>>> data(15-16 million docs) present on each shard. Please advise me of the
>>> standard approaches to define the number of shards and shard size. Thanks
>>> in advance.
>>
>
>
--
*************************************************************
Bernd Fehling Bielefeld University Library
Dipl.-Inform. (FH) LibTec - Library Technology
Universitätsstr. 25 and Knowledge Management
33615 Bielefeld
Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
https://www.ub.uni-bielefeld.de/~befehl/
BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************
Re: Optimal Sharding Strategy for Solr Cloud v8.10
Posted by Walter Underwood <wu...@wunderwood.org>.
This is all great advice.
There is no optimal number of shards. I’ve run clusters with 4 shards, we currently have one cluster with 96 shards and one with 320 shards. The next one we build out will probably not be sharded.
With long queries, I’ve usually seen a roughly linear speedup with sharding. Double the shards, halve the response time.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Sep 13, 2023, at 4:48 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>
> Hi,
>
> There are no hard rules wrt sharding, it often comes down to measuring and experimenting for your workload.
>
> There are other things to consider than shard size. Why are the queries slow? How many rows do you ask for? Do you use faceting? Grouping?
> You have 25Gb of data on each of the 8 nodes/shards. Now, how much RAM does each node have, and how much RAM did you allocate to Solr/Java?
> A common mistake is to allocate too much ram/heap to Solr to you don't get any virtual memory caching in Linux.
> Say you have 32Gb of physical RAM on the nodes. Then do not give 30 of those to Solr. Instead give 8Gb to Solr and let 24Gb be available for disk caching.
>
> Other things to consider is to look at whether your queries can be optimized by rewriting them to more efficient equivalents. Sometimes, Solr-level caches can also help.
>
> Wrt shards efficiency: If you already have 8 shards, it is not much more expensive to go to 16, but you increase the risk of a single failure affecting your requests...
>
> Jan
>
>> 13. sep. 2023 kl. 10:32 skrev Saksham Gupta <sa...@indiamart.com.INVALID>:
>>
>> Hi All,
>>
>> I have been trying to reduce the response time of solr cloud(v8.10, 8
>> nodes). To achieve this, I have tried increasing the number of shards of
>> solr cloud which can help reduce data size on each shard thereby reducing
>> response time.
>>
>>
>> I have encountered a few questions regarding sharding strategy:
>>
>> 1. How to decide the ideal number of shards? Is there a minimum or maximum
>> number of shards which should be used?
>>
>> 2. What is the minimum size of a shard after which reducing the size
>> further won't have any effect on the response time (as time taken by other
>> factors like data aggregation will compensate for that) ?
>>
>> 3. Is there some maximum limit to the size of data that should be kept in a
>> shard?
>>
>>
>> As of now we have 8 shards each on a separate node with ~25 gb of
>> data(15-16 million docs) present on each shard. Please advise me of the
>> standard approaches to define the number of shards and shard size. Thanks
>> in advance.
>
Re: Optimal Sharding Strategy for Solr Cloud v8.10
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
There are no hard rules wrt sharding, it often comes down to measuring and experimenting for your workload.
There are other things to consider than shard size. Why are the queries slow? How many rows do you ask for? Do you use faceting? Grouping?
You have 25Gb of data on each of the 8 nodes/shards. Now, how much RAM does each node have, and how much RAM did you allocate to Solr/Java?
A common mistake is to allocate too much ram/heap to Solr to you don't get any virtual memory caching in Linux.
Say you have 32Gb of physical RAM on the nodes. Then do not give 30 of those to Solr. Instead give 8Gb to Solr and let 24Gb be available for disk caching.
Other things to consider is to look at whether your queries can be optimized by rewriting them to more efficient equivalents. Sometimes, Solr-level caches can also help.
Wrt shards efficiency: If you already have 8 shards, it is not much more expensive to go to 16, but you increase the risk of a single failure affecting your requests...
Jan
> 13. sep. 2023 kl. 10:32 skrev Saksham Gupta <sa...@indiamart.com.INVALID>:
>
> Hi All,
>
> I have been trying to reduce the response time of solr cloud(v8.10, 8
> nodes). To achieve this, I have tried increasing the number of shards of
> solr cloud which can help reduce data size on each shard thereby reducing
> response time.
>
>
> I have encountered a few questions regarding sharding strategy:
>
> 1. How to decide the ideal number of shards? Is there a minimum or maximum
> number of shards which should be used?
>
> 2. What is the minimum size of a shard after which reducing the size
> further won't have any effect on the response time (as time taken by other
> factors like data aggregation will compensate for that) ?
>
> 3. Is there some maximum limit to the size of data that should be kept in a
> shard?
>
>
> As of now we have 8 shards each on a separate node with ~25 gb of
> data(15-16 million docs) present on each shard. Please advise me of the
> standard approaches to define the number of shards and shard size. Thanks
> in advance.
Re: Optimal Sharding Strategy for Solr Cloud v8.10
Posted by ufuk yılmaz <uy...@vivaldi.net.INVALID>.
My two cents:
1) Try to have shards small enough to fit in memory so entire index can be cached, less disk access=more speed. So #of shards depend on the memory available on your nodes. If you still need more read throughput add more replicas
2) At some point network chatter would be too much due to the #of shards
3) 2 billion documents per shard (if this didn’t change in recent versions)
-ufuk
—
> On 13 Sep 2023, at 11:32, Saksham Gupta <sa...@indiamart.com.invalid> wrote:
>
> Hi All,
>
> I have been trying to reduce the response time of solr cloud(v8.10, 8
> nodes). To achieve this, I have tried increasing the number of shards of
> solr cloud which can help reduce data size on each shard thereby reducing
> response time.
>
>
> I have encountered a few questions regarding sharding strategy:
>
> 1. How to decide the ideal number of shards? Is there a minimum or maximum
> number of shards which should be used?
>
> 2. What is the minimum size of a shard after which reducing the size
> further won't have any effect on the response time (as time taken by other
> factors like data aggregation will compensate for that) ?
>
> 3. Is there some maximum limit to the size of data that should be kept in a
> shard?
>
>
> As of now we have 8 shards each on a separate node with ~25 gb of
> data(15-16 million docs) present on each shard. Please advise me of the
> standard approaches to define the number of shards and shard size. Thanks
> in advance.