You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Prabhjot Bharaj <pr...@gmail.com> on 2015/09/02 10:47:41 UTC
Distributed Task Queue with client node failovers
Hello Folks,
I am trying to design a distributed queue with Zookeeper. The use case is
like this:-
A set of 't' tasks to be executed by 'n' nodes in round-robin fashion. But,
if any node goes down, the other nodes should take the task.
For this purpose, I'm reading about zookeeper Queue recipe. The Queue
recipe relies on client watcher.
Initially, all clients run their respective tasks. But, after one
iteration, lets say one client node goes down.
In this case, that client node itself is unavailable to get a Watch Event,
can the task be transferred to another node ?
The worst case will be -
All the client nodes, but one, are down. Can this one node still pick up
all tasks sequentially and execute them ?
Thanks,
Prabhjot
Re: Distributed Task Queue with client node failovers
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Please have a look at Nirmata Workflow (which I wrote). Given your description, it does what you want. Even if you don’t use it, you can review it for an example of how to do what you want: http://nirmataoss.github.io/workflow/
-Jordan
On September 2, 2015 at 5:14:47 AM, Prabhjot Bharaj (prabhbharaj@gmail.com) wrote:
Hi Adam,
I had not gone through the contents of the link you provided
But, after reading the Tech Note 4, I can say that my use case is not on
that scale.
Regards,
Prabhjot
On Wed, Sep 2, 2015 at 4:09 PM, Adam Milne-Smith <ad...@milne-smith.co.uk>
wrote:
> Not an answer to your question but I just wanted to check you've read
> https://cwiki.apache.org/confluence/display/CURATOR/TN4
>
> On 2 Sep 2015 09:47, Prabhjot Bharaj <pr...@gmail.com> wrote:
> >
> > Hello Folks,
> >
> > I am trying to design a distributed queue with Zookeeper. The use case is
> > like this:-
> >
> > A set of 't' tasks to be executed by 'n' nodes in round-robin fashion.
> But,
> > if any node goes down, the other nodes should take the task.
> >
> > For this purpose, I'm reading about zookeeper Queue recipe. The Queue
> > recipe relies on client watcher.
> >
> > Initially, all clients run their respective tasks. But, after one
> > iteration, lets say one client node goes down.
> > In this case, that client node itself is unavailable to get a Watch
> Event,
> > can the task be transferred to another node ?
> >
> > The worst case will be -
> > All the client nodes, but one, are down. Can this one node still pick up
> > all tasks sequentially and execute them ?
> >
> > Thanks,
> > Prabhjot
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"
Re: Distributed Task Queue with client node failovers
Posted by Prabhjot Bharaj <pr...@gmail.com>.
Hi Adam,
I had not gone through the contents of the link you provided
But, after reading the Tech Note 4, I can say that my use case is not on
that scale.
Regards,
Prabhjot
On Wed, Sep 2, 2015 at 4:09 PM, Adam Milne-Smith <ad...@milne-smith.co.uk>
wrote:
> Not an answer to your question but I just wanted to check you've read
> https://cwiki.apache.org/confluence/display/CURATOR/TN4
>
> On 2 Sep 2015 09:47, Prabhjot Bharaj <pr...@gmail.com> wrote:
> >
> > Hello Folks,
> >
> > I am trying to design a distributed queue with Zookeeper. The use case is
> > like this:-
> >
> > A set of 't' tasks to be executed by 'n' nodes in round-robin fashion.
> But,
> > if any node goes down, the other nodes should take the task.
> >
> > For this purpose, I'm reading about zookeeper Queue recipe. The Queue
> > recipe relies on client watcher.
> >
> > Initially, all clients run their respective tasks. But, after one
> > iteration, lets say one client node goes down.
> > In this case, that client node itself is unavailable to get a Watch
> Event,
> > can the task be transferred to another node ?
> >
> > The worst case will be -
> > All the client nodes, but one, are down. Can this one node still pick up
> > all tasks sequentially and execute them ?
> >
> > Thanks,
> > Prabhjot
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"
Re: Distributed Task Queue with client node failovers
Posted by Adam Milne-Smith <ad...@milne-smith.co.uk>.
Not an answer to your question but I just wanted to check you've read https://cwiki.apache.org/confluence/display/CURATOR/TN4
On 2 Sep 2015 09:47, Prabhjot Bharaj <pr...@gmail.com> wrote:
>
> Hello Folks,
>
> I am trying to design a distributed queue with Zookeeper. The use case is
> like this:-
>
> A set of 't' tasks to be executed by 'n' nodes in round-robin fashion. But,
> if any node goes down, the other nodes should take the task.
>
> For this purpose, I'm reading about zookeeper Queue recipe. The Queue
> recipe relies on client watcher.
>
> Initially, all clients run their respective tasks. But, after one
> iteration, lets say one client node goes down.
> In this case, that client node itself is unavailable to get a Watch Event,
> can the task be transferred to another node ?
>
> The worst case will be -
> All the client nodes, but one, are down. Can this one node still pick up
> all tasks sequentially and execute them ?
>
> Thanks,
> Prabhjot