You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by Sznajder ForMailingList <bs...@gmail.com> on 2013/11/28 13:48:57 UTC
Using lockPath() in QueueBuilder can hurt the throughput by 134%
Hi
Following my precedent questions on this mailing list, I continued to
investigate and found that using lockPath() in the QueueBuilder can hurt
the throughput of the consumption of the queue items by 134%.
In fact the lockPath() starts to seriously hurting the throughput when we
multiply the number of queue consumers.
For helping you to reproduce the problem, I wrote the attached code.
The main class gets one parameter : the connectString to the ZQ quorum. If
no parameters is provided, the class will launch the Curator TestingServer.
The producer starts by putting 100 items in the queue, then it monitors the
number of elements in the queue and measures the time spent to consume the
whole queue.
The consumers just consume the items without doing anything else, and then
sleep 500 msecs.
I tested the program with these different parameters:
1, 2, 4, 8, 16 and 32 consumers...
with setting the lockPath() and without setting the lockPath().
And I get the number described in the attached Excel file. You can see that
for 1 or 2 consumers, the hurt by using lockPath() isn't so significant,
but it can reach 134% when we use a larger number of consumers....
I put here the output:
[java] Without LOCK_PATH
[java] -=-=-=-=-=-= 1 consumers =-=-=-=-=-=-
[java] It took 29562 msecs for consuming 100 items
[java] -=-=-=-=-=-= 32 consumers =-=-=-=-=-=-
[java] It took 1559 msecs for consuming 100 items
[java] -=-=-=-=-=-= 16 consumers =-=-=-=-=-=-
[java] It took 2022 msecs for consuming 100 items
[java] -=-=-=-=-=-= 8 consumers =-=-=-=-=-=-
[java] It took 4045 msecs for consuming 100 items
[java] -=-=-=-=-=-= 4 consumers =-=-=-=-=-=-
[java] It took 7541 msecs for consuming 100 items
[java] -=-=-=-=-=-= 2 consumers =-=-=-=-=-=-
[java] It took 15539 msecs for consuming 100 items
[java]
[java]
[java]
[java] With LOCK_PATH
[java] -=-=-=-=-=-= 1 consumers =-=-=-=-=-=-
[java] It took 30562 msecs for consuming 100 items
[java] -=-=-=-=-=-= 32 consumers =-=-=-=-=-=-
[java] It took 3639 msecs for consuming 100 items
[java] -=-=-=-=-=-= 16 consumers =-=-=-=-=-=-
[java] It took 4610 msecs for consuming 100 items
[java] -=-=-=-=-=-= 8 consumers =-=-=-=-=-=-
[java] It took 6101 msecs for consuming 100 items
[java] -=-=-=-=-=-= 4 consumers =-=-=-=-=-=-
[java] It took 9073 msecs for consuming 100 items
[java] -=-=-=-=-=-= 2 consumers =-=-=-=-=-=-
[java] It took 16053 msecs for consuming 100 items
Re: Using lockPath() in QueueBuilder can hurt the throughput by 134%
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
This doesn’t surprise me. The Javadoc for the lockPath() method even says "There is a small performance penalty for this behavior however.”. I guess we should update that, but taking a lock for every message will incur a performance penalty.
-Jordan
On Nov 28, 2013, at 4:48 AM, Sznajder ForMailingList <bs...@gmail.com> wrote:
> Hi
>
> Following my precedent questions on this mailing list, I continued to investigate and found that using lockPath() in the QueueBuilder can hurt the throughput of the consumption of the queue items by 134%.
>
> In fact the lockPath() starts to seriously hurting the throughput when we multiply the number of queue consumers.
>
>
> For helping you to reproduce the problem, I wrote the attached code.
>
> The main class gets one parameter : the connectString to the ZQ quorum. If no parameters is provided, the class will launch the Curator TestingServer.
>
>
> The producer starts by putting 100 items in the queue, then it monitors the number of elements in the queue and measures the time spent to consume the whole queue.
>
> The consumers just consume the items without doing anything else, and then sleep 500 msecs.
>
> I tested the program with these different parameters:
> 1, 2, 4, 8, 16 and 32 consumers...
> with setting the lockPath() and without setting the lockPath().
>
>
>
> And I get the number described in the attached Excel file. You can see that for 1 or 2 consumers, the hurt by using lockPath() isn't so significant, but it can reach 134% when we use a larger number of consumers....
>
>
> I put here the output:
>
> [java] Without LOCK_PATH
> [java] -=-=-=-=-=-= 1 consumers =-=-=-=-=-=-
> [java] It took 29562 msecs for consuming 100 items
> [java] -=-=-=-=-=-= 32 consumers =-=-=-=-=-=-
> [java] It took 1559 msecs for consuming 100 items
> [java] -=-=-=-=-=-= 16 consumers =-=-=-=-=-=-
> [java] It took 2022 msecs for consuming 100 items
> [java] -=-=-=-=-=-= 8 consumers =-=-=-=-=-=-
> [java] It took 4045 msecs for consuming 100 items
> [java] -=-=-=-=-=-= 4 consumers =-=-=-=-=-=-
> [java] It took 7541 msecs for consuming 100 items
> [java] -=-=-=-=-=-= 2 consumers =-=-=-=-=-=-
> [java] It took 15539 msecs for consuming 100 items
> [java]
> [java]
> [java]
> [java] With LOCK_PATH
> [java] -=-=-=-=-=-= 1 consumers =-=-=-=-=-=-
> [java] It took 30562 msecs for consuming 100 items
> [java] -=-=-=-=-=-= 32 consumers =-=-=-=-=-=-
> [java] It took 3639 msecs for consuming 100 items
> [java] -=-=-=-=-=-= 16 consumers =-=-=-=-=-=-
> [java] It took 4610 msecs for consuming 100 items
> [java] -=-=-=-=-=-= 8 consumers =-=-=-=-=-=-
> [java] It took 6101 msecs for consuming 100 items
> [java] -=-=-=-=-=-= 4 consumers =-=-=-=-=-=-
> [java] It took 9073 msecs for consuming 100 items
> [java] -=-=-=-=-=-= 2 consumers =-=-=-=-=-=-
> [java] It took 16053 msecs for consuming 100 items
>
> <QueueConsumerProducer.java><SimpleQueueItem.java><Throughput.pdf>