You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@zookeeper.apache.org by Check Peck <co...@gmail.com> on 2015/09/05 00:17:55 UTC

multiple machines working on different task

I have a simple requirement as I am using Zookeeper with Curator. I have a
znode like this "/test/clients/" and some other application will create a
znode under "/test/clients/" with some data in it and those are each task
that my other three machines has to perform so it will be like this:-

This will be a znode "/test/clients/clientid_123" with some data in it,
let's say Hello World 1
This will be a znode "/test/clients/clientid_124" with some data in it.
let's say Hello World 2
This will be a znode "/test/clients/clientid_125" with some data in it.
let's say Hello World 3
This will be a znode "/test/clients/clientid_126" with some data in it.
let's say Hello World 4

Now I have three machines in which I will be running a service which will
use Curator library. I want all those three machine to pick up some
clientid task from "/test/clients/" znode.

For example : if machineA is working on "clientid_123", then I don't want
machineB or machineC working on "clientid_123". A single machine can work
on multiple clientid task but no two machines should pick same client id
task.

How can I do this with zookeeper? I can have all three machines keep a
watch on "/test/clients/" znode and as soon as anything gets added up in
this znode then I don't know how to make sure only one is picking client id
task. I can use leadership but in leadership, only single machine will be
leader and that single machine will work on all the client id task and
that's what I dont want.

I want all the machines working on some client id task.

Re: multiple machines working on different task

Posted by Edward Ribeiro <ed...@gmail.com>.

Hi Check Peck,


Sorry for the late reply, but you can also take a look at Ordasity (
https://github.com/boundary/ordasity ) , that seems to be exactly design to
solve your kind of problem. Ordasity is written in Scala, but I *guess* you
should have no major problem reading the code and extracting its design for
your own scenario. As Jordan said: many ways of solving this issue.

Regards and good luck!
Edward

On Fri, Sep 4, 2015 at 7:29 PM, Check Peck <co...@gmail.com> wrote:

> Thanks Jordan.
>
> Can you briefly explain how I can leverage Nirmata Workflow here? I was
> going through this example
> http://nirmataoss.github.io/workflow/example.html
> but I was not able to understand how can I use it for my use case?
>
> And if you have some other solutions as well which I can use here then let
> me know as well.
>
>
> On Fri, Sep 4, 2015 at 3:20 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com
> > wrote:
>
> > There are many ways to accomplish this. However, I wrote something that
> > may solve your problem. Have a look at Nirmata Workflow:
> > http://nirmataoss.github.io/workflow/
> >
> > Even if you don’t use it directly, you can use the code as a sample of
> how
> > to do what you want.
> >
> > -Jordan
> >
> > On September 4, 2015 at 3:18:23 PM, Check Peck (comptechgeeky@gmail.com)
> > wrote:
> >
> > I have a simple requirement as I am using Zookeeper with Curator. I have
> a
> > znode like this "/test/clients/" and some other application will create a
> > znode under "/test/clients/" with some data in it and those are each task
> > that my other three machines has to perform so it will be like this:-
> >
> > This will be a znode "/test/clients/clientid_123" with some data in it,
> > let's say Hello World 1
> > This will be a znode "/test/clients/clientid_124" with some data in it.
> > let's say Hello World 2
> > This will be a znode "/test/clients/clientid_125" with some data in it.
> > let's say Hello World 3
> > This will be a znode "/test/clients/clientid_126" with some data in it.
> > let's say Hello World 4
> >
> > Now I have three machines in which I will be running a service which will
> > use Curator library. I want all those three machine to pick up some
> > clientid task from "/test/clients/" znode.
> >
> > For example : if machineA is working on "clientid_123", then I don't want
> > machineB or machineC working on "clientid_123". A single machine can work
> > on multiple clientid task but no two machines should pick same client id
> > task.
> >
> > How can I do this with zookeeper? I can have all three machines keep a
> > watch on "/test/clients/" znode and as soon as anything gets added up in
> > this znode then I don't know how to make sure only one is picking client
> > id
> > task. I can use leadership but in leadership, only single machine will be
> > leader and that single machine will work on all the client id task and
> > that's what I dont want.
> >
> > I want all the machines working on some client id task.
> >
> >
>

Re: multiple machines working on different task

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

Nirmata Workflow is a distributed scheduler. From what I can read, that’s what you’ve described. Have I missed something?

-Jordan

On September 4, 2015 at 5:29:33 PM, Check Peck (comptechgeeky@gmail.com) wrote:

Thanks Jordan.

Can you briefly explain how I can leverage Nirmata Workflow here? I was going through this example http://nirmataoss.github.io/workflow/example.html but I was not able to understand how can I use it for my use case?

And if you have some other solutions as well which I can use here then let me know as well.

On Fri, Sep 4, 2015 at 3:20 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
There are many ways to accomplish this. However, I wrote something that may solve your problem. Have a look at Nirmata Workflow: http://nirmataoss.github.io/workflow/

Even if you don’t use it directly, you can use the code as a sample of how to do what you want.

-Jordan

On September 4, 2015 at 3:18:23 PM, Check Peck (comptechgeeky@gmail.com) wrote:

I have a simple requirement as I am using Zookeeper with Curator. I have a
znode like this "/test/clients/" and some other application will create a
znode under "/test/clients/" with some data in it and those are each task
that my other three machines has to perform so it will be like this:-

This will be a znode "/test/clients/clientid_123" with some data in it,
let's say Hello World 1
This will be a znode "/test/clients/clientid_124" with some data in it.
let's say Hello World 2
This will be a znode "/test/clients/clientid_125" with some data in it.
let's say Hello World 3
This will be a znode "/test/clients/clientid_126" with some data in it.
let's say Hello World 4

Now I have three machines in which I will be running a service which will
use Curator library. I want all those three machine to pick up some
clientid task from "/test/clients/" znode.

For example : if machineA is working on "clientid_123", then I don't want
machineB or machineC working on "clientid_123". A single machine can work
on multiple clientid task but no two machines should pick same client id
task.

How can I do this with zookeeper? I can have all three machines keep a
watch on "/test/clients/" znode and as soon as anything gets added up in
this znode then I don't know how to make sure only one is picking client id
task. I can use leadership but in leadership, only single machine will be
leader and that single machine will work on all the client id task and
that's what I dont want.

I want all the machines working on some client id task.

Re: multiple machines working on different task

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

Nirmata Workflow is a distributed scheduler. From what I can read, that’s what you’ve described. Have I missed something?

-Jordan

On September 4, 2015 at 5:29:33 PM, Check Peck (comptechgeeky@gmail.com) wrote:

Thanks Jordan.

And if you have some other solutions as well which I can use here then let me know as well.

Even if you don’t use it directly, you can use the code as a sample of how to do what you want.

-Jordan

On September 4, 2015 at 3:18:23 PM, Check Peck (comptechgeeky@gmail.com) wrote:

Now I have three machines in which I will be running a service which will
use Curator library. I want all those three machine to pick up some
clientid task from "/test/clients/" znode.

I want all the machines working on some client id task.

Re: multiple machines working on different task

Posted by Check Peck <co...@gmail.com>.

Thanks Jordan.

Can you briefly explain how I can leverage Nirmata Workflow here? I was
going through this example http://nirmataoss.github.io/workflow/example.html
but I was not able to understand how can I use it for my use case?

And if you have some other solutions as well which I can use here then let
me know as well.


On Fri, Sep 4, 2015 at 3:20 PM, Jordan Zimmerman <jordan@jordanzimmerman.com
> wrote:

> There are many ways to accomplish this. However, I wrote something that
> may solve your problem. Have a look at Nirmata Workflow:
> http://nirmataoss.github.io/workflow/
>
> Even if you don’t use it directly, you can use the code as a sample of how
> to do what you want.
>
> -Jordan
>
> On September 4, 2015 at 3:18:23 PM, Check Peck (comptechgeeky@gmail.com)
> wrote:
>
> I have a simple requirement as I am using Zookeeper with Curator. I have a
> znode like this "/test/clients/" and some other application will create a
> znode under "/test/clients/" with some data in it and those are each task
> that my other three machines has to perform so it will be like this:-
>
> This will be a znode "/test/clients/clientid_123" with some data in it,
> let's say Hello World 1
> This will be a znode "/test/clients/clientid_124" with some data in it.
> let's say Hello World 2
> This will be a znode "/test/clients/clientid_125" with some data in it.
> let's say Hello World 3
> This will be a znode "/test/clients/clientid_126" with some data in it.
> let's say Hello World 4
>
> Now I have three machines in which I will be running a service which will
> use Curator library. I want all those three machine to pick up some
> clientid task from "/test/clients/" znode.
>
> For example : if machineA is working on "clientid_123", then I don't want
> machineB or machineC working on "clientid_123". A single machine can work
> on multiple clientid task but no two machines should pick same client id
> task.
>
> How can I do this with zookeeper? I can have all three machines keep a
> watch on "/test/clients/" znode and as soon as anything gets added up in
> this znode then I don't know how to make sure only one is picking client
> id
> task. I can use leadership but in leadership, only single machine will be
> leader and that single machine will work on all the client id task and
> that's what I dont want.
>
> I want all the machines working on some client id task.
>
>

Re: multiple machines working on different task

Posted by Check Peck <co...@gmail.com>.

Thanks Jordan.

Can you briefly explain how I can leverage Nirmata Workflow here? I was
going through this example http://nirmataoss.github.io/workflow/example.html
but I was not able to understand how can I use it for my use case?

And if you have some other solutions as well which I can use here then let
me know as well.


On Fri, Sep 4, 2015 at 3:20 PM, Jordan Zimmerman <jordan@jordanzimmerman.com
> wrote:

> There are many ways to accomplish this. However, I wrote something that
> may solve your problem. Have a look at Nirmata Workflow:
> http://nirmataoss.github.io/workflow/
>
> Even if you don’t use it directly, you can use the code as a sample of how
> to do what you want.
>
> -Jordan
>
> On September 4, 2015 at 3:18:23 PM, Check Peck (comptechgeeky@gmail.com)
> wrote:
>
> I have a simple requirement as I am using Zookeeper with Curator. I have a
> znode like this "/test/clients/" and some other application will create a
> znode under "/test/clients/" with some data in it and those are each task
> that my other three machines has to perform so it will be like this:-
>
> This will be a znode "/test/clients/clientid_123" with some data in it,
> let's say Hello World 1
> This will be a znode "/test/clients/clientid_124" with some data in it.
> let's say Hello World 2
> This will be a znode "/test/clients/clientid_125" with some data in it.
> let's say Hello World 3
> This will be a znode "/test/clients/clientid_126" with some data in it.
> let's say Hello World 4
>
> Now I have three machines in which I will be running a service which will
> use Curator library. I want all those three machine to pick up some
> clientid task from "/test/clients/" znode.
>
> For example : if machineA is working on "clientid_123", then I don't want
> machineB or machineC working on "clientid_123". A single machine can work
> on multiple clientid task but no two machines should pick same client id
> task.
>
> How can I do this with zookeeper? I can have all three machines keep a
> watch on "/test/clients/" znode and as soon as anything gets added up in
> this znode then I don't know how to make sure only one is picking client
> id
> task. I can use leadership but in leadership, only single machine will be
> leader and that single machine will work on all the client id task and
> that's what I dont want.
>
> I want all the machines working on some client id task.
>
>

Re: multiple machines working on different task

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

There are many ways to accomplish this. However, I wrote something that may solve your problem. Have a look at Nirmata Workflow: http://nirmataoss.github.io/workflow/

Even if you don’t use it directly, you can use the code as a sample of how to do what you want.

-Jordan

On September 4, 2015 at 3:18:23 PM, Check Peck (comptechgeeky@gmail.com) wrote:

Now I have three machines in which I will be running a service which will
use Curator library. I want all those three machine to pick up some
clientid task from "/test/clients/" znode.

I want all the machines working on some client id task.

Re: multiple machines working on different task

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

There are many ways to accomplish this. However, I wrote something that may solve your problem. Have a look at Nirmata Workflow: http://nirmataoss.github.io/workflow/

Even if you don’t use it directly, you can use the code as a sample of how to do what you want.

-Jordan

On September 4, 2015 at 3:18:23 PM, Check Peck (comptechgeeky@gmail.com) wrote:

Now I have three machines in which I will be running a service which will
use Curator library. I want all those three machine to pick up some
clientid task from "/test/clients/" znode.

I want all the machines working on some client id task.