is zookeeper suitable for my application?

[Li Li <fa...@gmail.com>]

hi all,
    I am a newbie of zookeeper and want to know whether zookeeper is
suitable for my application. If it does, then I will try to learn more
about it.
    My problem can be described as a producer-consumer problem.
    There is a thread that continue producing data files untill the
queue is full and then waiting consumer. Consumers can be divided into
2 categories -- Master and Slave. The data can not be removed from
queue unless it's consumed by master. The data will be removed
immediately if it consumed by all the consumers. But if the queue is
full, it will remove data that are consumed by Master. There will be
only one Master and many Slaves. The count of Slaves will change(it
seems zookeeper provide Ephemeral ZNode).
    it seems that znode's size should less than 1 mb. but I will need
save large data file in a znode. Is there any solution for this?
    thank you.

Re: is zookeeper suitable for my application?

[Milind Parikh <mi...@gmail.com>]

There are existing recipes that you will be able to modify for your needs.
Ymmv but look at the double barrier recipe. The key thing to design around
in high. Scalability based solution is the herding effect.
Perhaps look @ couchdb to store your files for znode?
Regards
Milind

On Oct 8, 2010 3:31 AM, "Li Li" <fa...@gmail.com> wrote:

hi all,
   I am a newbie of zookeeper and want to know whether zookeeper is
suitable for my application. If it does, then I will try to learn more
about it.
   My problem can be described as a producer-consumer problem.
   There is a thread that continue producing data files untill the
queue is full and then waiting consumer. Consumers can be divided into
2 categories -- Master and Slave. The data can not be removed from
queue unless it's consumed by master. The data will be removed
immediately if it consumed by all the consumers. But if the queue is
full, it will remove data that are consumed by Master. There will be
only one Master and many Slaves. The count of Slaves will change(it
seems zookeeper provide Ephemeral ZNode).
   it seems that znode's size should less than 1 mb. but I will need
save large data file in a znode. Is there any solution for this?
   thank you.

Re: is zookeeper suitable for my application?

[Ted Dunning <te...@gmail.com>]

ZK provides all of the coordination you need for this problem, but you
should store your data elsewhere.

Any key-data store with decent read-write speed will suffice.  Memcache
would be reasonable for that if
you don't need persistence in the presence of failure.  Voldemort would be
another alternative if you do
need persistence.

Couchdb is generally pretty slow as a key/value store and is probably not a
very good option for this.

On Fri, Oct 8, 2010 at 3:31 AM, Li Li <fa...@gmail.com> wrote:

>    it seems that znode's size should less than 1 mb. but I will need
> save large data file in a znode. Is there any solution for this?
>    thank you.
>