You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by Sznajder ForMailingList <bs...@gmail.com> on 2013/11/24 17:51:52 UTC
Multiples queue consumers in a quorum reduces the throughput
Hi
A parallel problem to the problem I described earlier in this list is a
following:
The parameters are still the same:
- I have a producer who produces items in the queue (integer). The producer
adds some items when the queue is half of its initial size.
- The consumer just picks the item and write to a log file and goes to
sleep a while...
When I define a chorum of 2 machines - One producer, and one consumer, I
consume 1320 items in 5 minutes.
When I define a quorum of 3 machines - One producer and two consumers, I
consume 2693 items in 5 minutes (expected!)
When I define a quorum of 4 machines - One producer and three consumers, I
suddenly, consume only 1975 items in 5 minutes!! (Even less than the
precedent configuration!!)
I would thank you for your help/. This issue is relatively urgent for me..
Best regards
Benjamin
Re: Multiples queue consumers in a quorum reduces the throughput
Posted by Sznajder ForMailingList <bs...@gmail.com>.
Please ignore this duplicate mail
Benjamin
On Sun, Nov 24, 2013 at 6:51 PM, Sznajder ForMailingList <
bs4mailinglist@gmail.com> wrote:
> Hi
>
> A parallel problem to the problem I described earlier in this list is a
> following:
>
> The parameters are still the same:
>
> - I have a producer who produces items in the queue (integer). The
> producer adds some items when the queue is half of its initial size.
>
> - The consumer just picks the item and write to a log file and goes to
> sleep a while...
>
>
> When I define a chorum of 2 machines - One producer, and one consumer, I
> consume 1320 items in 5 minutes.
>
> When I define a quorum of 3 machines - One producer and two consumers, I
> consume 2693 items in 5 minutes (expected!)
>
> When I define a quorum of 4 machines - One producer and three consumers, I
> suddenly, consume only 1975 items in 5 minutes!! (Even less than the
> precedent configuration!!)
>
> I would thank you for your help/. This issue is relatively urgent for me..
>
> Best regards
> Benjamin
>
Re: Multiples queue consumers in a quorum reduces the throughput
Posted by Sznajder ForMailingList <bs...@gmail.com>.
Hi Jordan,
Thanks a lot for your help!
In the meanwhile, I think I found the problem. I open another thread for
this issue.
Best regards and thanks again.
Benjamin
On Tue, Nov 26, 2013 at 2:13 AM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:
> I’m not sure how I can help on this. ZooKeeper is not a good datastore
> system. ZooKeeper is a distributed coordination system. Each queue consumer
> must call getChildren() for the message path and attempt to process every
> message in the path. There’s no way around this given ZooKeeper’s current
> APIs. This is why the Curator docs say that “ZooKeeper is a very bad Queue”:
>
> https://cwiki.apache.org/confluence/display/CURATOR/TN4
>
> -Jordan
>
> On Nov 24, 2013, at 8:51 AM, Sznajder ForMailingList <
> bs4mailinglist@gmail.com> wrote:
>
> Hi
>
> A parallel problem to the problem I described earlier in this list is a
> following:
>
> The parameters are still the same:
>
> - I have a producer who produces items in the queue (integer). The
> producer adds some items when the queue is half of its initial size.
>
> - The consumer just picks the item and write to a log file and goes to
> sleep a while...
>
>
> When I define a chorum of 2 machines - One producer, and one consumer, I
> consume 1320 items in 5 minutes.
>
> When I define a quorum of 3 machines - One producer and two consumers, I
> consume 2693 items in 5 minutes (expected!)
>
> When I define a quorum of 4 machines - One producer and three consumers, I
> suddenly, consume only 1975 items in 5 minutes!! (Even less than the
> precedent configuration!!)
>
> I would thank you for your help/. This issue is relatively urgent for me..
>
> Best regards
> Benjamin
> <MyQueueConsumer.java><QueueProducer.java>
>
>
>
Re: Multiples queue consumers in a quorum reduces the throughput
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
I’m not sure how I can help on this. ZooKeeper is not a good datastore system. ZooKeeper is a distributed coordination system. Each queue consumer must call getChildren() for the message path and attempt to process every message in the path. There’s no way around this given ZooKeeper’s current APIs. This is why the Curator docs say that “ZooKeeper is a very bad Queue”:
https://cwiki.apache.org/confluence/display/CURATOR/TN4
-Jordan
On Nov 24, 2013, at 8:51 AM, Sznajder ForMailingList <bs...@gmail.com> wrote:
> Hi
>
> A parallel problem to the problem I described earlier in this list is a following:
>
> The parameters are still the same:
>
> - I have a producer who produces items in the queue (integer). The producer adds some items when the queue is half of its initial size.
>
> - The consumer just picks the item and write to a log file and goes to sleep a while...
>
>
> When I define a chorum of 2 machines - One producer, and one consumer, I consume 1320 items in 5 minutes.
>
> When I define a quorum of 3 machines - One producer and two consumers, I consume 2693 items in 5 minutes (expected!)
>
> When I define a quorum of 4 machines - One producer and three consumers, I suddenly, consume only 1975 items in 5 minutes!! (Even less than the precedent configuration!!)
>
> I would thank you for your help/. This issue is relatively urgent for me..
>
> Best regards
> Benjamin
> <MyQueueConsumer.java><QueueProducer.java>