You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by Ignasi Rius <ir...@gmail.com> on 2013/07/10 12:18:18 UTC
N+1 redundancy with Curator
Hi all,
I am a newbie to Zookeeper and Curator. After a couple of weeks
reading, testing and playing with it, I am quite convinced that it is
a great solution for our cluster management needs.
I would like to ask you some advice on how could I implement the
following behaviour based on Curator recipes:
We provide a service which is split/distributed among N nodes, and we
would like to implement a N+1 redundancy strategy. That is N active
nodes plus 1 spare node which can "behave" as any of the others.
Indeed, when each node starts, it should take the following role:
- a) Behave like a Spare Node: iif there are already "N" nodes working.
- b) Behave like node (M+1): if (M < N) (assuming M is the number of
current workers).
Then, in case that any of the N working nodes dies, the Spare one
should behave like the one that died.
Summarizing, as an exemple with 3 hosts (1,2 and 3) this is a
possible sequence of events:
- Host 1 starts and acts as Node 1.
- Host 2 starts and acts as Node 2.
- Host 3 starts and acts as a Spare Node.
- Host 2 dies.
- Host 3 takes over and acts as Node 2.
My question is about what do you guys think would be the best
"practice" for implementing such scenario?
I thought that the simplest thing to do was using an
InterProcessSemaphoreV2 (N), so if a node can acquire the semaphore,
it means that it is not the "Spare" one. However, I don't see a way in
Curator to find out "which node I am" in case I can acquire de
semaphore.
On the contrary, LeaderLatch seems to handle participant IDs, etc...
but there is only "one leader" at a time (obviusly).
Thanks for your time. Any ideas are welcome.
Best,
Ignasi
Re: N+1 redundancy with Curator
Posted by Ignasi Rius <ir...@gmail.com>.
Thanks!
So far it's working like a charm... let's see what happens when we
move it to production.
2013/7/11 Jordan Zimmerman <jo...@jordanzimmerman.com>:
> InterProcessSemaphoreV2.getParticipantNodes() would have ordinal numbers on each node - it's part of the recipe. But, use whichever method works best for you.
>
> -Jordan
>
> On Jul 11, 2013, at 8:26 AM, Ignasi Rius <ir...@gmail.com> wrote:
>
>> Hi Jordan, Eric, thanks for your advices.
>>
>> InterProcessSemaphoreV2.getParticipantNodes() is fine, but as Jordan
>> says, I would like that each participant knows who it is ordinally.
>> The reason is that I would like to use this number as an identifier
>> for the "role" I want that node to fulfill.
>>
>> Eric's suggestion will do the job and looks pretty easy to implement
>> and mantain. I will store a "list of roles" somewhere, and once a node
>> acquires de semaphore, it will try to create each "role node" until it
>> succeeds. That's it.
>>
>> Thanks!
>>
>> 2013/7/10 Eric Tschetter <ec...@gmail.com>:
>>> The other option is to have a specific znode for each "role" that you want
>>> fulfilled. On startup, you have each process race to claim one of those
>>> znodes by creating an ephemeral node. If it manages to create its ephemeral
>>> node, then it acts as that role. If it gets a NodeExistsException, then it
>>> tries another role.
>>>
>>> This is similar to electing a leader for each role. It does cause some
>>> stampeding when a node goes down, though. There also is no recipe for this
>>> currently.
>>>
>>> --Eric
>>>
>>>
>>> On Wednesday, July 10, 2013, Jordan Zimmerman wrote:
>>>>
>>>> I would use InterProcessSemaphoreV2. Does each node need to know which
>>>> participant it is ordinally? If so, I'm curious why. In any event, you can
>>>> use setNodeData() to set an identifier for each participant. Just like
>>>> LeaderLatch InterProcessSemaphoreV2 has a getParticipantNodes() method.
>>>>
>>>> -Jordan
>>>>
>>>> On Jul 10, 2013, at 3:18 AM, Ignasi Rius <ir...@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I am a newbie to Zookeeper and Curator. After a couple of weeks
>>>>> reading, testing and playing with it, I am quite convinced that it is
>>>>> a great solution for our cluster management needs.
>>>>>
>>>>> I would like to ask you some advice on how could I implement the
>>>>> following behaviour based on Curator recipes:
>>>>>
>>>>> We provide a service which is split/distributed among N nodes, and we
>>>>> would like to implement a N+1 redundancy strategy. That is N active
>>>>> nodes plus 1 spare node which can "behave" as any of the others.
>>>>>
>>>>> Indeed, when each node starts, it should take the following role:
>>>>> - a) Behave like a Spare Node: iif there are already "N" nodes working.
>>>>> - b) Behave like node (M+1): if (M < N) (assuming M is the number of
>>>>> current workers).
>>>>>
>>>>> Then, in case that any of the N working nodes dies, the Spare one
>>>>> should behave like the one that died.
>>>>>
>>>>>
>>>>> Summarizing, as an exemple with 3 hosts (1,2 and 3) this is a
>>>>> possible sequence of events:
>>>>>
>>>>> - Host 1 starts and acts as Node 1.
>>>>> - Host 2 starts and acts as Node 2.
>>>>> - Host 3 starts and acts as a Spare Node.
>>>>> - Host 2 dies.
>>>>> - Host 3 takes over and acts as Node 2.
>>>>>
>>>>>
>>>>> My question is about what do you guys think would be the best
>>>>> "practice" for implementing such scenario?
>>>>>
>>>>> I thought that the simplest thing to do was using an
>>>>> InterProcessSemaphoreV2 (N), so if a node can acquire the semaphore,
>>>>> it means that it is not the "Spare" one. However, I don't see a way in
>>>>> Curator to find out "which node I am" in case I can acquire de
>>>>> semaphore.
>>>>>
>>>>> On the contrary, LeaderLatch seems to handle participant IDs, etc...
>>>>> but there is only "one leader" at a time (obviusly).
>>>>>
>>>>> Thanks for your time. Any ideas are welcome.
>>>>>
>>>>> Best,
>>>>> Ignasi
>>>>
>>>
>
Re: N+1 redundancy with Curator
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
InterProcessSemaphoreV2.getParticipantNodes() would have ordinal numbers on each node - it's part of the recipe. But, use whichever method works best for you.
-Jordan
On Jul 11, 2013, at 8:26 AM, Ignasi Rius <ir...@gmail.com> wrote:
> Hi Jordan, Eric, thanks for your advices.
>
> InterProcessSemaphoreV2.getParticipantNodes() is fine, but as Jordan
> says, I would like that each participant knows who it is ordinally.
> The reason is that I would like to use this number as an identifier
> for the "role" I want that node to fulfill.
>
> Eric's suggestion will do the job and looks pretty easy to implement
> and mantain. I will store a "list of roles" somewhere, and once a node
> acquires de semaphore, it will try to create each "role node" until it
> succeeds. That's it.
>
> Thanks!
>
> 2013/7/10 Eric Tschetter <ec...@gmail.com>:
>> The other option is to have a specific znode for each "role" that you want
>> fulfilled. On startup, you have each process race to claim one of those
>> znodes by creating an ephemeral node. If it manages to create its ephemeral
>> node, then it acts as that role. If it gets a NodeExistsException, then it
>> tries another role.
>>
>> This is similar to electing a leader for each role. It does cause some
>> stampeding when a node goes down, though. There also is no recipe for this
>> currently.
>>
>> --Eric
>>
>>
>> On Wednesday, July 10, 2013, Jordan Zimmerman wrote:
>>>
>>> I would use InterProcessSemaphoreV2. Does each node need to know which
>>> participant it is ordinally? If so, I'm curious why. In any event, you can
>>> use setNodeData() to set an identifier for each participant. Just like
>>> LeaderLatch InterProcessSemaphoreV2 has a getParticipantNodes() method.
>>>
>>> -Jordan
>>>
>>> On Jul 10, 2013, at 3:18 AM, Ignasi Rius <ir...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I am a newbie to Zookeeper and Curator. After a couple of weeks
>>>> reading, testing and playing with it, I am quite convinced that it is
>>>> a great solution for our cluster management needs.
>>>>
>>>> I would like to ask you some advice on how could I implement the
>>>> following behaviour based on Curator recipes:
>>>>
>>>> We provide a service which is split/distributed among N nodes, and we
>>>> would like to implement a N+1 redundancy strategy. That is N active
>>>> nodes plus 1 spare node which can "behave" as any of the others.
>>>>
>>>> Indeed, when each node starts, it should take the following role:
>>>> - a) Behave like a Spare Node: iif there are already "N" nodes working.
>>>> - b) Behave like node (M+1): if (M < N) (assuming M is the number of
>>>> current workers).
>>>>
>>>> Then, in case that any of the N working nodes dies, the Spare one
>>>> should behave like the one that died.
>>>>
>>>>
>>>> Summarizing, as an exemple with 3 hosts (1,2 and 3) this is a
>>>> possible sequence of events:
>>>>
>>>> - Host 1 starts and acts as Node 1.
>>>> - Host 2 starts and acts as Node 2.
>>>> - Host 3 starts and acts as a Spare Node.
>>>> - Host 2 dies.
>>>> - Host 3 takes over and acts as Node 2.
>>>>
>>>>
>>>> My question is about what do you guys think would be the best
>>>> "practice" for implementing such scenario?
>>>>
>>>> I thought that the simplest thing to do was using an
>>>> InterProcessSemaphoreV2 (N), so if a node can acquire the semaphore,
>>>> it means that it is not the "Spare" one. However, I don't see a way in
>>>> Curator to find out "which node I am" in case I can acquire de
>>>> semaphore.
>>>>
>>>> On the contrary, LeaderLatch seems to handle participant IDs, etc...
>>>> but there is only "one leader" at a time (obviusly).
>>>>
>>>> Thanks for your time. Any ideas are welcome.
>>>>
>>>> Best,
>>>> Ignasi
>>>
>>
Re: N+1 redundancy with Curator
Posted by Ignasi Rius <ir...@gmail.com>.
Hi Jordan, Eric, thanks for your advices.
InterProcessSemaphoreV2.getParticipantNodes() is fine, but as Jordan
says, I would like that each participant knows who it is ordinally.
The reason is that I would like to use this number as an identifier
for the "role" I want that node to fulfill.
Eric's suggestion will do the job and looks pretty easy to implement
and mantain. I will store a "list of roles" somewhere, and once a node
acquires de semaphore, it will try to create each "role node" until it
succeeds. That's it.
Thanks!
2013/7/10 Eric Tschetter <ec...@gmail.com>:
> The other option is to have a specific znode for each "role" that you want
> fulfilled. On startup, you have each process race to claim one of those
> znodes by creating an ephemeral node. If it manages to create its ephemeral
> node, then it acts as that role. If it gets a NodeExistsException, then it
> tries another role.
>
> This is similar to electing a leader for each role. It does cause some
> stampeding when a node goes down, though. There also is no recipe for this
> currently.
>
> --Eric
>
>
> On Wednesday, July 10, 2013, Jordan Zimmerman wrote:
>>
>> I would use InterProcessSemaphoreV2. Does each node need to know which
>> participant it is ordinally? If so, I'm curious why. In any event, you can
>> use setNodeData() to set an identifier for each participant. Just like
>> LeaderLatch InterProcessSemaphoreV2 has a getParticipantNodes() method.
>>
>> -Jordan
>>
>> On Jul 10, 2013, at 3:18 AM, Ignasi Rius <ir...@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > I am a newbie to Zookeeper and Curator. After a couple of weeks
>> > reading, testing and playing with it, I am quite convinced that it is
>> > a great solution for our cluster management needs.
>> >
>> > I would like to ask you some advice on how could I implement the
>> > following behaviour based on Curator recipes:
>> >
>> > We provide a service which is split/distributed among N nodes, and we
>> > would like to implement a N+1 redundancy strategy. That is N active
>> > nodes plus 1 spare node which can "behave" as any of the others.
>> >
>> > Indeed, when each node starts, it should take the following role:
>> > - a) Behave like a Spare Node: iif there are already "N" nodes working.
>> > - b) Behave like node (M+1): if (M < N) (assuming M is the number of
>> > current workers).
>> >
>> > Then, in case that any of the N working nodes dies, the Spare one
>> > should behave like the one that died.
>> >
>> >
>> > Summarizing, as an exemple with 3 hosts (1,2 and 3) this is a
>> > possible sequence of events:
>> >
>> > - Host 1 starts and acts as Node 1.
>> > - Host 2 starts and acts as Node 2.
>> > - Host 3 starts and acts as a Spare Node.
>> > - Host 2 dies.
>> > - Host 3 takes over and acts as Node 2.
>> >
>> >
>> > My question is about what do you guys think would be the best
>> > "practice" for implementing such scenario?
>> >
>> > I thought that the simplest thing to do was using an
>> > InterProcessSemaphoreV2 (N), so if a node can acquire the semaphore,
>> > it means that it is not the "Spare" one. However, I don't see a way in
>> > Curator to find out "which node I am" in case I can acquire de
>> > semaphore.
>> >
>> > On the contrary, LeaderLatch seems to handle participant IDs, etc...
>> > but there is only "one leader" at a time (obviusly).
>> >
>> > Thanks for your time. Any ideas are welcome.
>> >
>> > Best,
>> > Ignasi
>>
>
Re: N+1 redundancy with Curator
Posted by Eric Tschetter <ec...@gmail.com>.
The other option is to have a specific znode for each "role" that you want
fulfilled. On startup, you have each process race to claim one of those
znodes by creating an ephemeral node. If it manages to create its
ephemeral node, then it acts as that role. If it gets a
NodeExistsException, then it tries another role.
This is similar to electing a leader for each role. It does cause some
stampeding when a node goes down, though. There also is no recipe for this
currently.
--Eric
On Wednesday, July 10, 2013, Jordan Zimmerman wrote:
> I would use InterProcessSemaphoreV2. Does each node need to know which
> participant it is ordinally? If so, I'm curious why. In any event, you can
> use setNodeData() to set an identifier for each participant. Just like
> LeaderLatch InterProcessSemaphoreV2 has a getParticipantNodes() method.
>
> -Jordan
>
> On Jul 10, 2013, at 3:18 AM, Ignasi Rius <iriusvsn@gmail.com<javascript:;>>
> wrote:
>
> > Hi all,
> >
> > I am a newbie to Zookeeper and Curator. After a couple of weeks
> > reading, testing and playing with it, I am quite convinced that it is
> > a great solution for our cluster management needs.
> >
> > I would like to ask you some advice on how could I implement the
> > following behaviour based on Curator recipes:
> >
> > We provide a service which is split/distributed among N nodes, and we
> > would like to implement a N+1 redundancy strategy. That is N active
> > nodes plus 1 spare node which can "behave" as any of the others.
> >
> > Indeed, when each node starts, it should take the following role:
> > - a) Behave like a Spare Node: iif there are already "N" nodes working.
> > - b) Behave like node (M+1): if (M < N) (assuming M is the number of
> > current workers).
> >
> > Then, in case that any of the N working nodes dies, the Spare one
> > should behave like the one that died.
> >
> >
> > Summarizing, as an exemple with 3 hosts (1,2 and 3) this is a
> > possible sequence of events:
> >
> > - Host 1 starts and acts as Node 1.
> > - Host 2 starts and acts as Node 2.
> > - Host 3 starts and acts as a Spare Node.
> > - Host 2 dies.
> > - Host 3 takes over and acts as Node 2.
> >
> >
> > My question is about what do you guys think would be the best
> > "practice" for implementing such scenario?
> >
> > I thought that the simplest thing to do was using an
> > InterProcessSemaphoreV2 (N), so if a node can acquire the semaphore,
> > it means that it is not the "Spare" one. However, I don't see a way in
> > Curator to find out "which node I am" in case I can acquire de
> > semaphore.
> >
> > On the contrary, LeaderLatch seems to handle participant IDs, etc...
> > but there is only "one leader" at a time (obviusly).
> >
> > Thanks for your time. Any ideas are welcome.
> >
> > Best,
> > Ignasi
>
>
Re: N+1 redundancy with Curator
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
I would use InterProcessSemaphoreV2. Does each node need to know which participant it is ordinally? If so, I'm curious why. In any event, you can use setNodeData() to set an identifier for each participant. Just like LeaderLatch InterProcessSemaphoreV2 has a getParticipantNodes() method.
-Jordan
On Jul 10, 2013, at 3:18 AM, Ignasi Rius <ir...@gmail.com> wrote:
> Hi all,
>
> I am a newbie to Zookeeper and Curator. After a couple of weeks
> reading, testing and playing with it, I am quite convinced that it is
> a great solution for our cluster management needs.
>
> I would like to ask you some advice on how could I implement the
> following behaviour based on Curator recipes:
>
> We provide a service which is split/distributed among N nodes, and we
> would like to implement a N+1 redundancy strategy. That is N active
> nodes plus 1 spare node which can "behave" as any of the others.
>
> Indeed, when each node starts, it should take the following role:
> - a) Behave like a Spare Node: iif there are already "N" nodes working.
> - b) Behave like node (M+1): if (M < N) (assuming M is the number of
> current workers).
>
> Then, in case that any of the N working nodes dies, the Spare one
> should behave like the one that died.
>
>
> Summarizing, as an exemple with 3 hosts (1,2 and 3) this is a
> possible sequence of events:
>
> - Host 1 starts and acts as Node 1.
> - Host 2 starts and acts as Node 2.
> - Host 3 starts and acts as a Spare Node.
> - Host 2 dies.
> - Host 3 takes over and acts as Node 2.
>
>
> My question is about what do you guys think would be the best
> "practice" for implementing such scenario?
>
> I thought that the simplest thing to do was using an
> InterProcessSemaphoreV2 (N), so if a node can acquire the semaphore,
> it means that it is not the "Spare" one. However, I don't see a way in
> Curator to find out "which node I am" in case I can acquire de
> semaphore.
>
> On the contrary, LeaderLatch seems to handle participant IDs, etc...
> but there is only "one leader" at a time (obviusly).
>
> Thanks for your time. Any ideas are welcome.
>
> Best,
> Ignasi