You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Віталій Тимчишин <ti...@gmail.com> on 2011/02/11 18:51:37 UTC
ExecutorService over a zookeeper
Hello.
I am trying to implement distributed executor service over a zookeeper and
encountered a problem I am not sure how to deal with.
First of all, let me show you the way I am trying to implements this.
I create a number of prefix nodes for different messages belonging to the
same job: Job, Queued, Reserved, Result
When someone want to submit a job, he creates EPHEMERAL_SEQUENTIAL node
under Job. The name returned will be job name, it will be used under every
other prefix. All created nodes are EPHEMERAL
Next sequence of OK processing operations looks like the next (S -
submitter, P - processor)
1) S creates marker under Queued prefix
2) P creates marker under Reserved prefix
3) S removes marker under Queued prefix
4) P sets data under Result prefix
5) S gets data and removes all the nodes
6) P (after Job is removed) also removes all the nodes left
This sheme supports both Processor problems (Reserved marker will disappear
and Submitter will restart from (1)) and Submitter problems (as soon as Job
dissappear, job is treated as cancelled by Processor).
The problem is that I need to use watch for data removal. And exists watch
for data removal may produce memory leaks if data was already removed on the
time of exists call - the watch will be registered, but never triggered. And
I don't see a way to clean-up watches on given path.
While writing this e-mail I'we got a thought - I can use getData to set up a
watch for removal. As far as I can see from client code, watch won't be set
unless item does exist.
Another problem is waiting for "Reserved" to appear - I need to set up
exists watch, but there may be case when task is cancelled before it was
taken by any processor. Should I make a fake "Reserved" item and then
instantly remove it just to trigger the watch (and make it go to GC)? If so,
can this be done as asynchronious calls without any callback?
--
Best regards,
Vitalii Tymchyshyn
Re: ExecutorService over a zookeeper
Posted by Mahadev Konar <ma...@apache.org>.
Please look at http://hadoop.apache.org/zookeeper/mailing_lists.html
on how to unsubscribe.
thanks
mahadev
2011/2/13 Aaron Crow <di...@yahoo.com>:
> unsubscribe
>
> 2011/2/13 Thomas Koch <th...@koch.ro>
>
>> Віталій Тимчишин:
>> > Hello.
>> >
>> > I am trying to implement distributed executor service over a zookeeper
>> and
>> > encountered a problem I am not sure how to deal with.
>> > <SNIP />
>> Hi,
>>
>> I've the impression, you're doing it too complicate. Why not have a znode
>> jobs/ and put all your jobs as persistent znodes in there.
>> So your job_queue could look like:
>>
>> jobs/
>> job1
>> job2
>> job3
>>
>> A consumer can then lock a job by putting an ephemeral node in one of the
>> job
>> znodes:
>>
>> jobs/
>> job1
>> job2/
>> .lock
>> job3
>>
>> Other consumers see that somebody is already working on job2. If the
>> consumer
>> should die, the ephemeral lock will vanish and another consumer could take
>> over.
>> You should not publish results in ZK but use some other communication
>> method.
>> ZK is really not meant as a messaging solution.
>> Maybe you'd be better served with a standard message queue (e.g. Apache
>> activeMQ, rabbitMQ) and use ZK only for coordination? If you have really
>> high
>> message load, you could have a look at Hedwig (google://zookeeper hedwig).
>>
>> Best regards,
>>
>> Thomas Koch, http://www.koch.ro
>>
>
Re: ExecutorService over a zookeeper
Posted by Aaron Crow <di...@yahoo.com>.
unsubscribe
2011/2/13 Thomas Koch <th...@koch.ro>
> Віталій Тимчишин:
> > Hello.
> >
> > I am trying to implement distributed executor service over a zookeeper
> and
> > encountered a problem I am not sure how to deal with.
> > <SNIP />
> Hi,
>
> I've the impression, you're doing it too complicate. Why not have a znode
> jobs/ and put all your jobs as persistent znodes in there.
> So your job_queue could look like:
>
> jobs/
> job1
> job2
> job3
>
> A consumer can then lock a job by putting an ephemeral node in one of the
> job
> znodes:
>
> jobs/
> job1
> job2/
> .lock
> job3
>
> Other consumers see that somebody is already working on job2. If the
> consumer
> should die, the ephemeral lock will vanish and another consumer could take
> over.
> You should not publish results in ZK but use some other communication
> method.
> ZK is really not meant as a messaging solution.
> Maybe you'd be better served with a standard message queue (e.g. Apache
> activeMQ, rabbitMQ) and use ZK only for coordination? If you have really
> high
> message load, you could have a look at Hedwig (google://zookeeper hedwig).
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro
>
Re: ExecutorService over a zookeeper
Posted by Віталій Тимчишин <ti...@gmail.com>.
15 лютого 2011 р. 12:10 Thomas Koch <th...@koch.ro> написав:
>
> Maybe the gearman project is more suitable to your needs?
> http://gearman.org
>
May be. But I already have zookeeper implementation working at the small
example. Will do massive test tomorrow.
For now I send create/delete pair for result/reserved nodes to ensure all
watches've fired on any task outcome to prevent leaks.
--
Best regards,
Vitalii Tymchyshyn
Re: ExecutorService over a zookeeper
Posted by Thomas Koch <th...@koch.ro>.
Віталій Тимчишин:
> 2011/2/14 Ted Dunning <te...@gmail.com>
>
> > Scalability.
> >
> > Zookeeper is limited to a few 10's of thousands of updates per second.
> >
> > Large updates would limit this even more.
> >
> > If that is enough, then great. If not, too bad.
>
> Yep, it's enough. Remeber, it's executor service. It has no sense to
> cluster a task that will take less then 100 millis in average. So, taking
> 10 updates per task, it should handle 10000s/10*0.1 = 100s parallel tasks.
> Its OK. It will handle a cluster of ~20 powerfull cluster nodes. If I will
> have more, I can easily have a set of independant servers with tasks
> distributed over them.
>
> BTW: Does this number change with new servers or replication makes it
> worse?
Maybe the gearman project is more suitable to your needs?
http://gearman.org
Thomas Koch, http://www.koch.ro
Re: ExecutorService over a zookeeper
Posted by Ted Dunning <te...@gmail.com>.
Replication makes the update rate worse (slightly).
2011/2/15 Віталій Тимчишин <ti...@gmail.com>
> BTW: Does this number change with new servers or replication makes it
> worse?
>
Re: ExecutorService over a zookeeper
Posted by Віталій Тимчишин <ti...@gmail.com>.
2011/2/14 Ted Dunning <te...@gmail.com>
> Scalability.
>
> Zookeeper is limited to a few 10's of thousands of updates per second.
> Large updates would limit this even more.
>
> If that is enough, then great. If not, too bad.
>
Yep, it's enough. Remeber, it's executor service. It has no sense to cluster
a task that will take less then 100 millis in average. So, taking 10 updates
per task, it should handle 10000s/10*0.1 = 100s parallel tasks. Its OK. It
will handle a cluster of ~20 powerfull cluster nodes. If I will have more, I
can easily have a set of independant servers with tasks distributed over
them.
BTW: Does this number change with new servers or replication makes it worse?
--
Best regards,
Vitalii Tymchyshyn
Re: ExecutorService over a zookeeper
Posted by Ted Dunning <te...@gmail.com>.
Scalability.
Zookeeper is limited to a few 10's of thousands of updates per second.
Large updates would limit this even more.
If that is enough, then great. If not, too bad.
Kafka should be able to handle 100 times that or more.
2011/2/14 Віталій Тимчишин <ti...@gmail.com>
> But why Zookeeper is not a messaging?
Re: ExecutorService over a zookeeper
Posted by Віталій Тимчишин <ti...@gmail.com>.
Sorry for second e-mail, did not answer the whole thing
You should not publish results in ZK but use some other communication
> method.
>
I do. But i need to tell that result are ready and available on this
address.
> ZK is really not meant as a messaging solution.
> Maybe you'd be better served with a standard message queue (e.g. Apache
> activeMQ, rabbitMQ) and use ZK only for coordination? If you have really
> high
> message load, you could have a look at Hedwig (google://zookeeper hedwig).
>
>
After few years working with activeMQ, I hate it. As other projects workers
in our company do. I simply hangs after some time with no apparent reason.
It does not like rollbacks. (Vitalii, please stop or this won't be a
Zookeeper e-mail :) ).
But why Zookeeper is not a messaging? Actually I was suggested Kafka in this
thread, and it looks like messaging solution. The problem is that JMS-like
messaging does not suit me. First of all, it does not alove a message to be
cancelled. Then, it gives no control over message processing (Producer may
want in future to control message requeue on Consumer failure for "bad"
message not to put down all Consumers one by one). Also it requires at least
two queues to handle responses (I've used Queue + Topic for Hazelcast) I
already has support for Hazelcast & Grid-Gain, but there are certain problem
with them. That's why I am trying Zookeeper now.
As for storage for both tasks & responses, we are using NFS now and may move
to HDFS if Zookeeper will be adopted successfully (or, better say, it's
orthogonal to clustering solution selection).
--
Best regards,
Vitalii Tymchyshyn
Re: ExecutorService over a zookeeper
Posted by Jake Mannix <ja...@gmail.com>.
Or, have the job znode list a host/port/interface(thrift/avro/etc) to call
back to, and skip the message queue altogether.
On Feb 13, 2011 12:06 PM, "Ted Dunning" <te...@gmail.com> wrote:
Consider also Kafka.
2011/2/13 Thomas Koch <th...@koch.ro>
> You should not publish results in ZK but use some other communication >
method. > ZK is really no...
Re: ExecutorService over a zookeeper
Posted by Ted Dunning <te...@gmail.com>.
Consider also Kafka.
2011/2/13 Thomas Koch <th...@koch.ro>
> You should not publish results in ZK but use some other communication
> method.
> ZK is really not meant as a messaging solution.
> Maybe you'd be better served with a standard message queue (e.g. Apache
> activeMQ, rabbitMQ) and use ZK only for coordination? If you have really
> high
> message load, you could have a look at Hedwig (google://zookeeper hedwig).
>
Re: ExecutorService over a zookeeper
Posted by Віталій Тимчишин <ti...@gmail.com>.
13 лютого 2011 р. 18:10 Thomas Koch <th...@koch.ro> написав:
> Віталій Тимчишин:
> > Hello.
> >
> > I am trying to implement distributed executor service over a zookeeper
> and
> > encountered a problem I am not sure how to deal with.
> > <SNIP />
> Hi,
>
> I've the impression, you're doing it too complicate. Why not have a znode
> jobs/ and put all your jobs as persistent znodes in there.
> So your job_queue could look like:
>
> jobs/
> job1
> job2
> job3
>
> A consumer can then lock a job by putting an ephemeral node in one of the
> job
> znodes:
>
> jobs/
> job1
> job2/
> .lock
> job3
>
>
1)This won't handle producer death as job can't be EPHEMERAL
2)The job list will have locked jobs every consumer must iterate over to
find non-locked one.
3) I have job result
So, instead of
jobs/
job1
.lock
.result
I have
jobs/
job1 (holding data)
locks/
job1
queue/
job1 (holding unlocked jobs)
result/
job1
This is actually 1 more node per job (queue) to handle (2)
But, my main Q is about watches leakes. Even in your scheme, adding a watch
on jobs/job1/.result may produce a leak if job is cancelled before it was
processed (so that .result never filled).
--
Best regards,
Vitalii Tymchyshyn
Re: ExecutorService over a zookeeper
Posted by Thomas Koch <th...@koch.ro>.
Віталій Тимчишин:
> Hello.
>
> I am trying to implement distributed executor service over a zookeeper and
> encountered a problem I am not sure how to deal with.
> <SNIP />
Hi,
I've the impression, you're doing it too complicate. Why not have a znode
jobs/ and put all your jobs as persistent znodes in there.
So your job_queue could look like:
jobs/
job1
job2
job3
A consumer can then lock a job by putting an ephemeral node in one of the job
znodes:
jobs/
job1
job2/
.lock
job3
Other consumers see that somebody is already working on job2. If the consumer
should die, the ephemeral lock will vanish and another consumer could take
over.
You should not publish results in ZK but use some other communication method.
ZK is really not meant as a messaging solution.
Maybe you'd be better served with a standard message queue (e.g. Apache
activeMQ, rabbitMQ) and use ZK only for coordination? If you have really high
message load, you could have a look at Hedwig (google://zookeeper hedwig).
Best regards,
Thomas Koch, http://www.koch.ro