[jira] [Comment Edited] (SOLR-15055) Re-implement 'withCollection' and 'maxShardsPerNode'

["Andrzej Bialecki (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262615#comment-17262615 ] 

Andrzej Bialecki edited comment on SOLR-15055 at 1/11/21, 12:47 PM:
--------------------------------------------------------------------

So, let's take a step back, and start with use-cases and requirements.

Cross-collection joins is the main use-case because it benefits greatly from co-location of replicas for the primary (search) and secondary (join) collections.

 1. when creating the primary collection (or adding new replicas to it) the placement plugin must take this into account and enforce the co-location of new primary replicas if possible.

* in 8x the framework would automatically create secondary replicas as necessary to satisfy this requirement (at the cost of code complexity).
* now at minimum the placement plugin should fail if it's not possible to satisfy this constraint with the current cluster layout, because then some primary replicas would have to use secondary replicas from other nodes and this would greatly (and unevenly) increase the query latency.
* if the placement of the primary collection fails because this constraint cannot be satisfied (there are too few secondary replicas) then the operator must manually add as many secondary replicas as needed. This is the weakness of this minimal approach - it would be better to somehow automate it to eliminate the manual intervention.

 2. removal of secondary replicas should be prevented if they are actively used under this constraint by any primary replicas on a specific node - allowing this would cause performance issues as described above.
 3. removal of primary replicas should cause the secondary replicas to be removed as well, if they are no longer in use on a specific node - but ensuring that at least N replicas of the secondary collection remain (to prevent data loss).


was (Author: ab):
So, let's take a step back, and start with use-cases and requirements:
* cross-collection joins is the main use-case because it benefits greatly from co-location of replicas for the primary (search) and secondary (join) collections.
* when creating the primary collection (or adding new replicas to it) the placement plugin must take this into account and enforce the co-location of new primary replicas if possible.
** in 8x the framework would automatically create secondary replicas as necessary to satisfy this requirement (at the cost of code complexity).
** now at minimum the placement plugin should fail if it's not possible to satisfy this constraint with the current cluster layout, because then some primary replicas would have to use secondary replicas from other nodes and this would greatly (and unevenly) increase the query latency.
** if the placement of the primary collection fails because this constraint cannot be satisfied (there are too few secondary replicas) then the operator must manually add as many secondary replicas as needed. This is the weakness of this minimal approach - it would be better to somehow automate it to eliminate the manual intervention. 
* removal of secondary replicas should be prevented if they are actively used under this constraint by any primary replicas on a specific node - allowing this would cause performance issues as described above.
* removal of primary replicas should cause the secondary replicas to be removed as well, if they are no longer in use on a specific node - but ensuring that at least N replicas of the secondary collection remain (to prevent data loss).

> Re-implement 'withCollection' and 'maxShardsPerNode'
> ----------------------------------------------------
>
>                 Key: SOLR-15055
>                 URL: https://issues.apache.org/jira/browse/SOLR-15055
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Andrzej Bialecki
>            Assignee: Andrzej Bialecki
>            Priority: Major
>          Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> Solr 8x replica placement provided two settings that are very useful in certain scenarios:
> * {{withCollection}} constraint specified that replicas should be placed on the same nodes where replicas of another collection are located. In the 8x implementation this was limited in practice to co-locating single-shard secondary collections used for joins or other lookups from the main collection (which could be multi-sharded).
> * {{maxShardsPerNode}} - this constraint specified the maximum number of replicas per shard that can be placed on the same node. In most scenarios this was set to 1 in order to ensure fault-tolerance (ie. at most 1 replica of any given shard would be placed on any given node). Changing this constraint to values > 1 would reduce fault-tolerance but may be desired in test setups or as a temporary relief measure.
>  
> Both these constraints are collection-specific so they should be configured e.g. as collection properties.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org