[jira] [Comment Edited] (SOLR-6491) Umbrella JIRA for managing the leader assignments

["Erick Erickson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136004#comment-14136004 ] 

Erick Erickson edited comment on SOLR-6491 at 9/16/14 7:25 PM:
---------------------------------------------------------------

bq: A node may be preferred for sometime and after sometime it can change
True. That's where the SOLR-6513 comes in. Remember also that this is a _preference_, not an absolute map.

bq: After a new leader election happens overseer does a quick check on all the nodes where the shard has replicas
One situation we're talking about here is one in which the pathology has already happened. For illustration, consider the following:

I have 10 shards and 10 nodes. I have 100 replicas, i.e. each node has one replica for every shard.
All the nodes are shut down.
I start 1 node. All 10 leaders are elected on that node.
Now I bring up my other 9 machines.

No leader election goes on here as the other 9 nodes come up. So "After a new leader election happens overseer does a quick check on all the nodes where the shard has replicas, and find out if there is a more suitable node for that shard in the cluster" doesn't apply since there's no leader election going on. I suppose we could just fire a "balance leaders" command, but that kind of pre-supposes a symmetric cluster.

I'm reluctant to try to do everything automatically. Consider an asymmetric cluster, both in terms of hardware and collection characteristics. You have machines of varying power hosting (perhaps many) collections with different usage patterns. I don't see how to replace the operations folks' knowledge of all this with an auto-assignment process; the "and find out if there is a more suitable node" above. Suitable by what criteria? To do this really well, we'd need to have a language to define capabilities for both collections and nodes, and a heuristic to factor all that in auto-leader election.

If we do take a stab at figuring this all out automatically, it seems that the preferredLeader mechanism could be the way this is realized; auto-assign the roles, then trigger the re-election process. With room for the ops people to tweak it as they see fit.

Now all _that_ said, maybe I'm over-thinking the problem. It might be "good enough" to just not bother with the preferredLeader role and have a "rebalance leaders now" command. And trust the operations people to distribute their collections appropriately. By "appropriately", I'm thinking of assigning the collections across the cluster such that the leaders will be acceptably distributed if we just balance them within a collection.

BTW, I think the idea of more intelligently electing leaders is a good one, and this in no way means we shouldn't do that too. I'm concerned more intelligence there is only part of the total solution though.


was (Author: erickerickson):
bq: A node may be preferred for sometime and after sometime it can change
True. That's where the SOLR-6513 comes in. Remember also that this is a _preference_, not an absolute map.

bq: After a new leader election happens overseer does a quick check on all the nodes where the shard has replicas
One situation we're talking about here is one in which the pathology has already happened. For illustration, consider the following:

I have 10 shards and 10 nodes. I have 100 replicas, i.e. each node has one replica for every shard.
All the nodes are shut down.
I start 1 node. All 10 leaders are elected on that node.
Now I bring up my other 9 machines.

No leader election goes on here as the other 9 nodes come up. So "After a new leader election happens overseer does a quick check on all the nodes where the shard has replicas, and find out if there is a more suitable node for that shard in the cluster" doesn't apply since there's no leader election going on. I suppose we could just fire a "balance leaders" command, but that kind of pre-supposes a symmetric cluster.

I'm reluctant to try to do everything automatically. Consider an asymmetric cluster, both in terms of hardware and collection characteristics. You have machines of varying power hosting (perhaps many) collections with different usage patterns. I don't see how to replace the operations folks' knowledge of all this with an auto-assignment process; the "and find out if there is a more suitable node" above. Suitable by what criteria? To do this really well, we'd need to have a language to define capabilities for both collections and nodes, and a heuristic to factor all that in auto-leader election.

If we do take a stab at figuring this all out automatically, it seems that the preferredLeader mechanism could be the way this is realized; auto-assign the roles, then trigger the re-election process. With room for the ops people to tweak it as they see fit.

Now all _that_ said, maybe I'm over-thinking the problem. It might be "good enough" to just not bother with the preferredLeader role and have a "rebalance leaders now" command. And trust the operations people to distribute their collections appropriately. By "appropriately", I'm thinking of assigning the collections across the cluster such that the leaders will be acceptably distributed if we just balance them within a collection.

> Umbrella JIRA for managing the leader assignments
> -------------------------------------------------
>
>                 Key: SOLR-6491
>                 URL: https://issues.apache.org/jira/browse/SOLR-6491
>             Project: Solr
>          Issue Type: Improvement
>    Affects Versions: 4.11, 5.0
>            Reporter: Erick Erickson
>            Assignee: Erick Erickson
>
> Leaders can currently get out of balance due to the sequence of how nodes are brought up in a cluster. For very good reasons shard leadership cannot be permanently assigned.
> However, it seems reasonable that a sys admin could optionally specify that a particular node be the _preferred_ leader for a particular collection/shard. During leader election, preference would be given to any node so marked when electing any leader.
> So the proposal here is to add another role for preferredLeader to the collections API, something like
> ADDROLE?role=preferredLeader&collection=collection_name&shard=shardId
> Second, it would be good to have a new collections API call like ELECTPREFERREDLEADERS?collection=collection_name
> (I really hate that name so far, but you see the idea). That command would (asynchronously?) make an attempt to transfer leadership for each shard in a collection to the leader labeled as the preferred leader by the new ADDROLE role.
> I'm going to start working on this, any suggestions welcome!
> This will subsume several other JIRAs, I'll link them momentarily.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org