[jira] [Commented] (SOLR-17373) Shard splitByPrefix should not do so if it would be too imbalanced/inefficient

["David Smiley (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-17373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17867017#comment-17867017 ] 

David Smiley commented on SOLR-17373:
-------------------------------------

Also or separately, I think the computed prefix histogram should be filtered so as to ensure that each prefix has at least one non-deleted doc.  This should be fairly cheap and simple, and addresses the particularly egregious scenario we encountered.

> Shard splitByPrefix should not do so if it would be too imbalanced/inefficient
> ------------------------------------------------------------------------------
>
>                 Key: SOLR-17373
>                 URL: https://issues.apache.org/jira/browse/SOLR-17373
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: SolrCloud
>            Reporter: David Smiley
>            Priority: Major
>
> Shard split "splitByPrefix" exists to reduce the number of shards that a typical prefix is in, thus reducing query fanout distributed search (assuming the route param is used), and it can isolate indexing activity as well.  Sometimes this can result in a very imbalanced (in-efficient) shard split that may even quickly lead to another split back-to-back!  (imagine splitting off less than 1%).  Here we propose that if the split would only split off < 20% of docs or so, then it's too inefficient.  Instead, split the middle of the largest key prefix.
> Note: it's also been observed that a prefix might be so extremely low represented that it's likely those docs are marked deleted as part of a previous shard split (if "link" split method).  Thus this inefficiency can have a cascading badness effect.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org