You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by David Smiley <ds...@apache.org> on 2023/12/18 20:19:29 UTC
Re: "Tell" a leader to give up leadership
Does anyone else have a use-case of *prioritizing* "leaders" or some
singleton role to have control of the preference of which node does
something? There are different ways of looking at this; maybe I'm not
expressing it in a way that resonates with others. Maybe another way to
think of this is, given a list of potential leaders, how can they negotiate
who is the leader, given some business logic to say who it is that "should"
be leader?
I work on Apache Solr and can relay some indications of how it's being
done, which is in a couple ways. Lets consider the "Overseer", which is a
cluster singleton. Optionally, some Solr nodes may be designated as being
preferred nodes for the Overseer. Such nodes "join at head" in the ZK
election recipe so that they are next in line for leadership. A thread on
the Overseer periodically checks to see if another node would be a better
node to run the Overseer. If this occurs, it tells itself to Quit. It
does this via a sending a message to itself via a ZK queue (a pre-existing
queue for how the Overseer works) -- I don't get the point at this since
it's in-process but it's not important.
I would prefer to use Curator for the Overseer election, but this node
preference feature seems to prevent that. If this is solved, Solr could
then possibly also use it to elect "replicas" of a "shard", which has a
similar feature but some additional aspects.
~ David Smiley
On Mon, Oct 9, 2023 at 4:14 PM Enrico Olivelli <eo...@gmail.com> wrote:
> bump
>
> Enrico
>
> Il giorno lun 27 feb 2023 alle ore 23:31 David Smiley
> <ds...@apache.org> ha scritto:
> >
> > Using the LeaderSelector recipe, how might some external process (with
> ZK access) get the elected leader to give up its leadership? Of course
> there might be some other communication channels like HTTP but this
> question disregards those options. Also disregard a ZK queue recipe as a
> communication medium to the leader, which would work but may be
> heavy-handed. So I'm wondering if there's a relatively simple /
> lightweight option.
> >
> > My higher level need is to prioritize potential leaders based on some
> globally known metadata (e.g. a file in ZK for example) where all
> candidates agree on which candidates are the most preferred leaders. If a
> candidate starts its LeaderSelector and also recognizes that he "should" be
> the leader, I want him to negotiate this with the current leader. Perhaps
> I should be thinking about this in some other way; I'm not sure. I'd be
> happy to ask this in a new thread if asked.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
>
Re: "Tell" a leader to give up leadership
Posted by Enrico Olivelli <eo...@gmail.com>.
Il giorno gio 11 gen 2024 alle ore 04:14 David Smiley
<ds...@apache.org> ha scritto:
>
> It's just a best-effort preference, not a guarantee. Some uses:
> * convenience for operating the system, reasoning about operational signals / logs etc. is easier if you know where something usually runs
> * balancing effort across a cluster. This node might effectively be reserved for the Overseer so it's overhead doesn't interfere with things happening on other nodes.
> * more easily controlling rolling restart behavior to restart the Overseer last instead of it moving "N" times where N is the number of nodes in a rolling restart (I've seen this happen). It's disruptive / inefficient.
>
> A possible outcome here is that *maybe*, this recipe isn't well suited to the task I describe; maybe there is another recipe / technique people have used.
In my experience I have seen that usually you prefer to have
homogeneous machines to run a given role, or in case of a preference
towards a specific machine, that was implemented this way:
- that node id is always the leader
- there is some watcher system (automated) that finds if the leader is down
- if it is down then it designates another id to become the new
"preferred leader"
Having competing peers but with someone who is "more powerful" is
somehow hard to figure out, because there is a conflict between "we
are all peers" and "I am the preferred"
It's only my 2 cents here, maybe others have different opinions/experiences
Enrico
>
> On Wed, Jan 10, 2024 at 8:43 AM Enrico Olivelli <eo...@gmail.com> wrote:
>>
>> David,
>> Sorry for the late reply, I missed your message.
>>
>> Il Lun 18 Dic 2023, 21:19 David Smiley <ds...@apache.org> ha scritto:
>>>
>>> Does anyone else have a use-case of prioritizing "leaders" or some singleton role to have control of the preference of which node does something? There are different ways of looking at this; maybe I'm not expressing it in a way that resonates with others. Maybe another way to think of this is, given a list of potential leaders, how can they negotiate who is the leader, given some business logic to say who it is that "should" be leader?
>>>
>>>
>>> I work on Apache Solr and can relay some indications of how it's being done, which is in a couple ways. Lets consider the "Overseer", which is a cluster singleton. Optionally, some Solr nodes may be designated as being preferred nodes for the Overseer. Such nodes "join at head" in the ZK election recipe so that they are next in line for leadership. A thread on the Overseer periodically checks to see if another node would be a better node to run the Overseer. If this occurs, it tells itself to Quit. It does this via a sending a message to itself via a ZK queue (a pre-existing queue for how the Overseer works) -- I don't get the point at this since it's in-process but it's not important.
>>>
>>> I would prefer to use Curator for the Overseer election, but this node preference feature seems to prevent that. If this is solved, Solr could then possibly also use it to elect "replicas" of a "shard", which has a similar feature but some additional aspects.
>>>
>>
>> I am not sure about how much we can put this into a "recipe" in Curator.
>> Basically you have to code your own logic to enforce who becomes the leader.
>>
>> One question is: why should one node be more preferable than another node for becoming the leader ?
>>
>> What happens if a node that is not the preferable leader is elected and then a better leader joins ?
>>
>> Cheers
>> Enrico
>>
>>>
>>> ~ David Smiley
>>>
>>> On Mon, Oct 9, 2023 at 4:14 PM Enrico Olivelli <eo...@gmail.com> wrote:
>>>>
>>>> bump
>>>>
>>>> Enrico
>>>>
>>>> Il giorno lun 27 feb 2023 alle ore 23:31 David Smiley
>>>> <ds...@apache.org> ha scritto:
>>>> >
>>>> > Using the LeaderSelector recipe, how might some external process (with ZK access) get the elected leader to give up its leadership? Of course there might be some other communication channels like HTTP but this question disregards those options. Also disregard a ZK queue recipe as a communication medium to the leader, which would work but may be heavy-handed. So I'm wondering if there's a relatively simple / lightweight option.
>>>> >
>>>> > My higher level need is to prioritize potential leaders based on some globally known metadata (e.g. a file in ZK for example) where all candidates agree on which candidates are the most preferred leaders. If a candidate starts its LeaderSelector and also recognizes that he "should" be the leader, I want him to negotiate this with the current leader. Perhaps I should be thinking about this in some other way; I'm not sure. I'd be happy to ask this in a new thread if asked.
>>>> >
>>>> > ~ David Smiley
>>>> > Apache Lucene/Solr Search Developer
>>>> > http://www.linkedin.com/in/davidwsmiley
Re: "Tell" a leader to give up leadership
Posted by David Smiley <ds...@apache.org>.
It's just a best-effort preference, not a guarantee. Some uses:
* convenience for operating the system, reasoning about operational signals
/ logs etc. is easier if you know where something usually runs
* balancing effort across a cluster. This node might effectively be
reserved for the Overseer so it's overhead doesn't interfere with things
happening on other nodes.
* more easily controlling rolling restart behavior to restart the Overseer
last instead of it moving "N" times where N is the number of nodes in a
rolling restart (I've seen this happen). It's disruptive / inefficient.
A possible outcome here is that *maybe*, this recipe isn't well suited to
the task I describe; maybe there is another recipe / technique people have
used.
On Wed, Jan 10, 2024 at 8:43 AM Enrico Olivelli <eo...@gmail.com> wrote:
> David,
> Sorry for the late reply, I missed your message.
>
> Il Lun 18 Dic 2023, 21:19 David Smiley <ds...@apache.org> ha scritto:
>
>> Does anyone else have a use-case of *prioritizing* "leaders" or some
>> singleton role to have control of the preference of which node does
>> something? There are different ways of looking at this; maybe I'm not
>> expressing it in a way that resonates with others. Maybe another way to
>> think of this is, given a list of potential leaders, how can they negotiate
>> who is the leader, given some business logic to say who it is that "should"
>> be leader?
>>
>
>> I work on Apache Solr and can relay some indications of how it's being
>> done, which is in a couple ways. Lets consider the "Overseer", which is a
>> cluster singleton. Optionally, some Solr nodes may be designated as being
>> preferred nodes for the Overseer. Such nodes "join at head" in the ZK
>> election recipe so that they are next in line for leadership. A thread on
>> the Overseer periodically checks to see if another node would be a better
>> node to run the Overseer. If this occurs, it tells itself to Quit. It
>> does this via a sending a message to itself via a ZK queue (a pre-existing
>> queue for how the Overseer works) -- I don't get the point at this since
>> it's in-process but it's not important.
>>
>> I would prefer to use Curator for the Overseer election, but this node
>> preference feature seems to prevent that. If this is solved, Solr could
>> then possibly also use it to elect "replicas" of a "shard", which has a
>> similar feature but some additional aspects.
>>
>>
> I am not sure about how much we can put this into a "recipe" in Curator.
> Basically you have to code your own logic to enforce who becomes the
> leader.
>
> One question is: why should one node be more preferable than another node
> for becoming the leader ?
>
> What happens if a node that is not the preferable leader is elected and
> then a better leader joins ?
>
> Cheers
> Enrico
>
>
>> ~ David Smiley
>>
>> On Mon, Oct 9, 2023 at 4:14 PM Enrico Olivelli <eo...@gmail.com>
>> wrote:
>>
>>> bump
>>>
>>> Enrico
>>>
>>> Il giorno lun 27 feb 2023 alle ore 23:31 David Smiley
>>> <ds...@apache.org> ha scritto:
>>> >
>>> > Using the LeaderSelector recipe, how might some external process (with
>>> ZK access) get the elected leader to give up its leadership? Of course
>>> there might be some other communication channels like HTTP but this
>>> question disregards those options. Also disregard a ZK queue recipe as a
>>> communication medium to the leader, which would work but may be
>>> heavy-handed. So I'm wondering if there's a relatively simple /
>>> lightweight option.
>>> >
>>> > My higher level need is to prioritize potential leaders based on some
>>> globally known metadata (e.g. a file in ZK for example) where all
>>> candidates agree on which candidates are the most preferred leaders. If a
>>> candidate starts its LeaderSelector and also recognizes that he "should" be
>>> the leader, I want him to negotiate this with the current leader. Perhaps
>>> I should be thinking about this in some other way; I'm not sure. I'd be
>>> happy to ask this in a new thread if asked.
>>> >
>>> > ~ David Smiley
>>> > Apache Lucene/Solr Search Developer
>>> > http://www.linkedin.com/in/davidwsmiley
>>>
>>
Re: "Tell" a leader to give up leadership
Posted by Enrico Olivelli <eo...@gmail.com>.
David,
Sorry for the late reply, I missed your message.
Il Lun 18 Dic 2023, 21:19 David Smiley <ds...@apache.org> ha scritto:
> Does anyone else have a use-case of *prioritizing* "leaders" or some
> singleton role to have control of the preference of which node does
> something? There are different ways of looking at this; maybe I'm not
> expressing it in a way that resonates with others. Maybe another way to
> think of this is, given a list of potential leaders, how can they negotiate
> who is the leader, given some business logic to say who it is that "should"
> be leader?
>
> I work on Apache Solr and can relay some indications of how it's being
> done, which is in a couple ways. Lets consider the "Overseer", which is a
> cluster singleton. Optionally, some Solr nodes may be designated as being
> preferred nodes for the Overseer. Such nodes "join at head" in the ZK
> election recipe so that they are next in line for leadership. A thread on
> the Overseer periodically checks to see if another node would be a better
> node to run the Overseer. If this occurs, it tells itself to Quit. It
> does this via a sending a message to itself via a ZK queue (a pre-existing
> queue for how the Overseer works) -- I don't get the point at this since
> it's in-process but it's not important.
>
> I would prefer to use Curator for the Overseer election, but this node
> preference feature seems to prevent that. If this is solved, Solr could
> then possibly also use it to elect "replicas" of a "shard", which has a
> similar feature but some additional aspects.
>
>
I am not sure about how much we can put this into a "recipe" in Curator.
Basically you have to code your own logic to enforce who becomes the leader.
One question is: why should one node be more preferable than another node
for becoming the leader ?
What happens if a node that is not the preferable leader is elected and
then a better leader joins ?
Cheers
Enrico
> ~ David Smiley
>
> On Mon, Oct 9, 2023 at 4:14 PM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
>> bump
>>
>> Enrico
>>
>> Il giorno lun 27 feb 2023 alle ore 23:31 David Smiley
>> <ds...@apache.org> ha scritto:
>> >
>> > Using the LeaderSelector recipe, how might some external process (with
>> ZK access) get the elected leader to give up its leadership? Of course
>> there might be some other communication channels like HTTP but this
>> question disregards those options. Also disregard a ZK queue recipe as a
>> communication medium to the leader, which would work but may be
>> heavy-handed. So I'm wondering if there's a relatively simple /
>> lightweight option.
>> >
>> > My higher level need is to prioritize potential leaders based on some
>> globally known metadata (e.g. a file in ZK for example) where all
>> candidates agree on which candidates are the most preferred leaders. If a
>> candidate starts its LeaderSelector and also recognizes that he "should" be
>> the leader, I want him to negotiate this with the current leader. Perhaps
>> I should be thinking about this in some other way; I'm not sure. I'd be
>> happy to ask this in a new thread if asked.
>> >
>> > ~ David Smiley
>> > Apache Lucene/Solr Search Developer
>> > http://www.linkedin.com/in/davidwsmiley
>>
>