You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Diego Oliveira <lo...@gmail.com> on 2012/12/05 03:42:57 UTC
Curator and queue recipe warning
Hello folks,
I got to the Curator wiki page to see something about the distributed
queue recipe and got surprised with the big warning telling to not use
zookeeper as a distributed queue controller, the reason page described some
cases that I passed here... is this *TRUE*, if so, what option do we have.
The message:
*IMPORTANT* - We recommend that you do NOT use ZooKeeper for Queues. Please
see Tech Note 4 <https://github.com/Netflix/curator/wiki/Tech-Note-4> for
details. Links
https://github.com/Netflix/curator/wiki/Distributed-Queue
https://github.com/Netflix/curator/wiki/Tech-Note-4
--
Att.
Diego de Oliveira
System Architect
diego@diegooliveira.com
www.diegooliveira.com
Never argue with a fool -- people might not be able to tell the difference
Re: Curator and queue recipe warning
Posted by Diego Oliveira <lo...@gmail.com>.
Is there a number pointing what lightweight mean for zookeeper (like
messages/seconds or message/seconds/size)?
--
Diego
On Wed, Dec 5, 2012 at 1:36 AM, Camille Fournier <ca...@apache.org> wrote:
> That is generally accurate. ZooKeeper isn't written to be a distributed
> queue in the high-throughput messaging kind of sense. If you need something
> truly lightweight and simple to serve as a distributed queue for a few
> messages, it might work. But for anything that could see significant
> volume, there's better options out there.
> ZooKeeper is used to implement several systems that will serve this need.
> Depending on what you're looking for, you might check out:
> Kafka: http://incubator.apache.org/kafka/
> Kestrel: https://github.com/robey/kestrel
>
> Or many of the brokered messaging servers like ActiveMQ, etc.
>
> C
>
>
> On Tue, Dec 4, 2012 at 9:42 PM, Diego Oliveira <lo...@gmail.com> wrote:
>
> > Hello folks,
> >
> > I got to the Curator wiki page to see something about the distributed
> > queue recipe and got surprised with the big warning telling to not use
> > zookeeper as a distributed queue controller, the reason page described
> some
> > cases that I passed here... is this *TRUE*, if so, what option do we
> have.
> >
> > The message:
> > *IMPORTANT* - We recommend that you do NOT use ZooKeeper for Queues.
> Please
> > see Tech Note 4 <https://github.com/Netflix/curator/wiki/Tech-Note-4>
> for
> > details. Links
> >
> > https://github.com/Netflix/curator/wiki/Distributed-Queue
> > https://github.com/Netflix/curator/wiki/Tech-Note-4
> >
> > --
> > Att.
> > Diego de Oliveira
> > System Architect
> > diego@diegooliveira.com
> > www.diegooliveira.com
> > Never argue with a fool -- people might not be able to tell the
> difference
> >
>
--
Att.
Diego de Oliveira
System Architect
diego@diegooliveira.com
www.diegooliveira.com
Never argue with a fool -- people might not be able to tell the difference
Re: Curator and queue recipe warning
Posted by Camille Fournier <ca...@apache.org>.
That is generally accurate. ZooKeeper isn't written to be a distributed
queue in the high-throughput messaging kind of sense. If you need something
truly lightweight and simple to serve as a distributed queue for a few
messages, it might work. But for anything that could see significant
volume, there's better options out there.
ZooKeeper is used to implement several systems that will serve this need.
Depending on what you're looking for, you might check out:
Kafka: http://incubator.apache.org/kafka/
Kestrel: https://github.com/robey/kestrel
Or many of the brokered messaging servers like ActiveMQ, etc.
C
On Tue, Dec 4, 2012 at 9:42 PM, Diego Oliveira <lo...@gmail.com> wrote:
> Hello folks,
>
> I got to the Curator wiki page to see something about the distributed
> queue recipe and got surprised with the big warning telling to not use
> zookeeper as a distributed queue controller, the reason page described some
> cases that I passed here... is this *TRUE*, if so, what option do we have.
>
> The message:
> *IMPORTANT* - We recommend that you do NOT use ZooKeeper for Queues. Please
> see Tech Note 4 <https://github.com/Netflix/curator/wiki/Tech-Note-4> for
> details. Links
>
> https://github.com/Netflix/curator/wiki/Distributed-Queue
> https://github.com/Netflix/curator/wiki/Tech-Note-4
>
> --
> Att.
> Diego de Oliveira
> System Architect
> diego@diegooliveira.com
> www.diegooliveira.com
> Never argue with a fool -- people might not be able to tell the difference
>
Re: Curator and queue recipe warning
Posted by Diego Oliveira <lo...@gmail.com>.
I took a look last night and it sounds like the big problem is the
serialization mechanism. The transportation layer has a limit of 1M in
size, the getChildren method pass all the children name in that chunk, so
if you have a lot of children then you are in trouble.
On Wed, Dec 5, 2012 at 7:09 AM, Bryan Thompson <br...@systap.com> wrote:
> InterestingŠ. I had long ago requested clarification of the zookeeper
> limits. This is in keeping with my expectations. Good for relatively
> small numbers of children.
>
> For example, you would be safe trying to coordinate 1000 machines but not
> safe trying to coordinate the locations of 100,000+ shards over 1000
> machines.
>
> Bryan
>
> On 12/4/12 9:42 PM, "Diego Oliveira" <lo...@gmail.com> wrote:
>
> >Hello folks,
> >
> > I got to the Curator wiki page to see something about the distributed
> >queue recipe and got surprised with the big warning telling to not use
> >zookeeper as a distributed queue controller, the reason page described
> >some
> >cases that I passed here... is this *TRUE*, if so, what option do we have.
> >
> >The message:
> >*IMPORTANT* - We recommend that you do NOT use ZooKeeper for Queues.
> >Please
> >see Tech Note 4 <https://github.com/Netflix/curator/wiki/Tech-Note-4> for
> >details. Links
> >
> >https://github.com/Netflix/curator/wiki/Distributed-Queue
> >https://github.com/Netflix/curator/wiki/Tech-Note-4
> >
> >--
> >Att.
> >Diego de Oliveira
> >System Architect
> >diego@diegooliveira.com
> >www.diegooliveira.com
> >Never argue with a fool -- people might not be able to tell the difference
>
>
--
Att.
Diego de Oliveira
System Architect
diego@diegooliveira.com
www.diegooliveira.com
Never argue with a fool -- people might not be able to tell the difference
Re: Curator and queue recipe warning
Posted by Bryan Thompson <br...@systap.com>.
InterestingŠ. I had long ago requested clarification of the zookeeper
limits. This is in keeping with my expectations. Good for relatively
small numbers of children.
For example, you would be safe trying to coordinate 1000 machines but not
safe trying to coordinate the locations of 100,000+ shards over 1000
machines.
Bryan
On 12/4/12 9:42 PM, "Diego Oliveira" <lo...@gmail.com> wrote:
>Hello folks,
>
> I got to the Curator wiki page to see something about the distributed
>queue recipe and got surprised with the big warning telling to not use
>zookeeper as a distributed queue controller, the reason page described
>some
>cases that I passed here... is this *TRUE*, if so, what option do we have.
>
>The message:
>*IMPORTANT* - We recommend that you do NOT use ZooKeeper for Queues.
>Please
>see Tech Note 4 <https://github.com/Netflix/curator/wiki/Tech-Note-4> for
>details. Links
>
>https://github.com/Netflix/curator/wiki/Distributed-Queue
>https://github.com/Netflix/curator/wiki/Tech-Note-4
>
>--
>Att.
>Diego de Oliveira
>System Architect
>diego@diegooliveira.com
>www.diegooliveira.com
>Never argue with a fool -- people might not be able to tell the difference