Re: Using Zookeeper to distribute tasks

[David Rouchy <dr...@gmail.com>]

Hi all,

I'm studying ZooKeeper to see how we can use it in a project. I'm still a
little big puzzled by the usage of zookeeper for distributing tasks. The
usage of queue does not fully fit the bill. I was just wondering if it would
be interesting / realistic to use a kind of "election" to decide on which
znode you want to send the task?

David Rouchy

On Wed, Apr 28, 2010 at 7:39 AM, David Rouchy <dr...@gmail.com> wrote:

> Thank you for your quick answers. I'll have a look.
>
> David
>
>
> On Tue, Apr 27, 2010 at 10:14 AM, Thomas Koch <th...@koch.ro> wrote:
>
>> David Rouchy:
>> > Hi all,
>> >
>> > We are studying using ZooKeeper to manage configuration across multiple
>> > processes & server. What would be also interesting, as ZooKeeper know
>> the
>> > list of process running, would be to use it to distribute tasks.
>> >
>> > We have some long running tasks, so we used multiple servers to process
>> > multiple tasks in the same time. My idea would be to use multiple
>> ephemeral
>> > watcher connected to one znode. A process will send data to this node
>> when
>> >  a new task has to be launched. But here is my issue, I would like only
>> one
>> >  watcher to be triggered of this change (a random watcher in the list).
>> Is
>> >  there way to do such thing in ZooKeeper?
>> >
>> > Regards,
>> >
>> > David Rouchy
>>
>> Hi David,
>>
>> there's gearman[1], a client-server system to distribute tasks. I've
>> mentioned
>> it already two times on this list, so sorry for the repetition. Gearman
>> servers and clients are availabe in different implementations that use the
>> same protocol and are interchangeable AFAIK.
>> If somebody would build a gearman server using Zookeeper, there would be a
>> whole ecosystem for ZK for free to conquer!
>>
>> [1] http://gearman.org
>>
>> Regards,
>>
>> Thomas Koch, http://www.koch.ro
>>
>
>

Re: Using Zookeeper to distribute tasks

[Patrick Hunt <ph...@apache.org>]

That's the beauty of zk, it's provides primitives that allow you much 
flexibility in what/how you implement something. Perhaps you could 
sketch out a bit more detail on what you had in mind, requirements/goals 
and such? What I mean is, from a high level coordination perspective, 
then we could see how that might map down to zk operations.

Patrick

On 05/19/2010 03:29 AM, David Rouchy wrote:
> Hi all,
>
> I'm studying ZooKeeper to see how we can use it in a project. I'm still a
> little big puzzled by the usage of zookeeper for distributing tasks. The
> usage of queue does not fully fit the bill. I was just wondering if it would
> be interesting / realistic to use a kind of "election" to decide on which
> znode you want to send the task?
>
> David Rouchy
>
> On Wed, Apr 28, 2010 at 7:39 AM, David Rouchy<dr...@gmail.com>  wrote:
>
>> Thank you for your quick answers. I'll have a look.
>>
>> David
>>
>>
>> On Tue, Apr 27, 2010 at 10:14 AM, Thomas Koch<th...@koch.ro>  wrote:
>>
>>> David Rouchy:
>>>> Hi all,
>>>>
>>>> We are studying using ZooKeeper to manage configuration across multiple
>>>> processes&  server. What would be also interesting, as ZooKeeper know
>>> the
>>>> list of process running, would be to use it to distribute tasks.
>>>>
>>>> We have some long running tasks, so we used multiple servers to process
>>>> multiple tasks in the same time. My idea would be to use multiple
>>> ephemeral
>>>> watcher connected to one znode. A process will send data to this node
>>> when
>>>>   a new task has to be launched. But here is my issue, I would like only
>>> one
>>>>   watcher to be triggered of this change (a random watcher in the list).
>>> Is
>>>>   there way to do such thing in ZooKeeper?
>>>>
>>>> Regards,
>>>>
>>>> David Rouchy
>>>
>>> Hi David,
>>>
>>> there's gearman[1], a client-server system to distribute tasks. I've
>>> mentioned
>>> it already two times on this list, so sorry for the repetition. Gearman
>>> servers and clients are availabe in different implementations that use the
>>> same protocol and are interchangeable AFAIK.
>>> If somebody would build a gearman server using Zookeeper, there would be a
>>> whole ecosystem for ZK for free to conquer!
>>>
>>> [1] http://gearman.org
>>>
>>> Regards,
>>>
>>> Thomas Koch, http://www.koch.ro
>>>
>>
>>
>