You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Andrzej Bialecki (Jira)" <ji...@apache.org> on 2021/03/09 15:29:00 UTC
[jira] [Resolved] (SOLR-12730) Implement staggered SPLITSHARD
requests in IndexSizeTrigger
[ https://issues.apache.org/jira/browse/SOLR-12730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki resolved SOLR-12730.
-------------------------------------
Fix Version/s: (was: master (9.0))
Resolution: Fixed
This has been fixed a long time ago (and subsequently removed from 9.0).
> Implement staggered SPLITSHARD requests in IndexSizeTrigger
> -----------------------------------------------------------
>
> Key: SOLR-12730
> URL: https://issues.apache.org/jira/browse/SOLR-12730
> Project: Solr
> Issue Type: Improvement
> Components: AutoScaling
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Priority: Major
> Fix For: 8.1
>
>
> Simulated large scale tests uncovered an interesting scenario that occurs also in real clusters where {{IndexSizeTrigger}} is used for controlling the maximum shard size.
> As index size grows and the number of shards grows, if document assignment is more or less even then at equal intervals (on a {{log2}} scale) there will be an avalanche of SPLITSHARD operations, because all shards will reach the critical size at approximately the same time.
> A hundred or more split shard operations running in parallel may severely affect the cluster performance.
> One possible approach to reduce the likelihood of this situation is to split shards not exactly in half but rather fudge the proportions around 60/40% in a random sequence, so that the resulting sub-sub-sub…shards would reach the thresholds at different times. This would require modifications to the SPLITSHARD command to allow this randomization.
> Another approach would be to simply limit the maximum number of parallel split shard operations. However, this would slow down the process of reaching the balance (increase lag) and possibly violate other operational constraints due to some shards waiting too long for the split and significantly exceeding their max size.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org