You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Charith Wickramarachchi <ch...@gmail.com> on 2012/06/13 06:20:11 UTC
Sequential Id generation with Zookeeper - Tips to improve performance
Hi All ,
I'm evaluating Apache Zookeepers to use in my project where i have a
requirement to generate unique and sequential message ids(long) across a
cluster.
After some web search i found[1] and some other different suggestions. I
started with the solution suggested in [1] but even with single client i
only got a throughput around 20-30 ids/sec. But according to the
performance numbers pointed in the website i think i should be able get
better results ( i m expecting to get at least 1000 id/s as i m using this
for a MoM where for each message i'll be calling zk id generator.)
In this case I m using ZK version 3.3.4 running in my local machine with
default settings and client also running in the same machine. I m attaching
my id generator class.
Any tips to improve the performance of this is highly appreciated. Am i
doing some thing wrong ?
[1]
http://zookeeper-user.578899.n2.nabble.com/Unique-Id-Generation-td2688494.html
cheers,
Charith
--
Charith Dhanushka Wickramarachchi
http://charithwiki.blogspot.com/
Re: Sequential Id generation with Zookeeper - Tips to improve performance
Posted by Charith Wickramarachchi <ch...@gmail.com>.
Thanks Sam for the detailed reply. I'll try to comeup with a way partition
the counters.
i could find a library written for Zookeeper based id generation. [1]
After using it with some randomization to minimize the contention i could
get much higher throughput. (~400id/s).
So looks like this bottleneck is purely due to the approach i m
following(as you pointed out).
[1]http://sourceforge.net/projects/zkidgen/
thanks again for the help.
cheers,
Charith
On Wed, Jun 13, 2012 at 7:23 AM, Sam Tunnicliffe <sa...@beobal.com> wrote:
> Yes, the contention occurs in a multithreaded context when two or more
> clients attempt to increment the same counter concurrently. The first
> in wins and the following updates are rejected as the znode version no
> longer matches the one obtained from the getData call. A clients who's
> increment is rejected just goes around the loop again, we don't have
> any more sophisticated logic in place to ensure fairness.
>
> We don't do any special tuning on the server side and we're running in
> AWS on standard m1.large instances. It could be the case that we're
> only seeing 10s of updates/s per unique counter like you, but because
> our counters are partitioned and we have multiple concurrent clients,
> total throughput is increased.
>
> Cheers,
> Sam
>
> On 13 June 2012 11:52, Charith Wickramarachchi
> <ch...@gmail.com> wrote:
> > Thanks Sam for the reply.
> >
> > The contention happens when we to setData from muliple clients right ?
> (due
> > to locking). But here even when i try using a single client the
> throughput
> > is in 20-30 id/s range.
> > Did you do any tuning in the server side. ?
> >
> > regards,
> > Charith
> >
> > On Wed, Jun 13, 2012 at 6:17 AM, Sam Tunnicliffe <sa...@beobal.com> wrote:
> >
> >> Hi Charith,
> >>
> >> can you partition your counters somehow, rather than having a single
> >> global counter? This would let you use multiple znodes, reducing
> >> contention during the setData call.
> >> We use almost exactly the same approach as you in our zookeeper backed
> >> sequence server[1] but with sharded counters and haven't had any
> >> issues with throughput (sorry, I don't have perf numbers to hand, but
> >> peak load pushes a few hundred r/s through)
> >>
> >> Cheers,
> >> Sam
> >>
> >> [1] https://github.com/talis/H1
> >>
> >> On 13 June 2012 05:20, Charith Wickramarachchi
> >> <ch...@gmail.com> wrote:
> >> > Hi All ,
> >> >
> >> > I'm evaluating Apache Zookeepers to use in my project where i have a
> >> > requirement to generate unique and sequential message ids(long)
> across a
> >> > cluster.
> >> > After some web search i found[1] and some other different
> suggestions.
> >> I
> >> > started with the solution suggested in [1] but even with single
> client i
> >> > only got a throughput around 20-30 ids/sec. But according to the
> >> performance
> >> > numbers pointed in the website i think i should be able get better
> >> results (
> >> > i m expecting to get at least 1000 id/s as i m using this for a MoM
> where
> >> > for each message i'll be calling zk id generator.)
> >> >
> >> > In this case I m using ZK version 3.3.4 running in my local machine
> with
> >> > default settings and client also running in the same machine. I m
> >> attaching
> >> > my id generator class.
> >> >
> >> > Any tips to improve the performance of this is highly appreciated. Am
> i
> >> > doing some thing wrong ?
> >> >
> >> >
> >> > [1]
> >>
> http://zookeeper-user.578899.n2.nabble.com/Unique-Id-Generation-td2688494.html
> >> >
> >> > cheers,
> >> > Charith
> >> > --
> >> > Charith Dhanushka Wickramarachchi
> >> > http://charithwiki.blogspot.com/
> >> >
> >>
> >
> >
> >
> > --
> > Charith Dhanushka Wickramarachchi
> > http://charithwiki.blogspot.com/
>
--
Charith Dhanushka Wickramarachchi
http://charithwiki.blogspot.com/
Re: Sequential Id generation with Zookeeper - Tips to improve performance
Posted by Sam Tunnicliffe <sa...@beobal.com>.
Yes, the contention occurs in a multithreaded context when two or more
clients attempt to increment the same counter concurrently. The first
in wins and the following updates are rejected as the znode version no
longer matches the one obtained from the getData call. A clients who's
increment is rejected just goes around the loop again, we don't have
any more sophisticated logic in place to ensure fairness.
We don't do any special tuning on the server side and we're running in
AWS on standard m1.large instances. It could be the case that we're
only seeing 10s of updates/s per unique counter like you, but because
our counters are partitioned and we have multiple concurrent clients,
total throughput is increased.
Cheers,
Sam
On 13 June 2012 11:52, Charith Wickramarachchi
<ch...@gmail.com> wrote:
> Thanks Sam for the reply.
>
> The contention happens when we to setData from muliple clients right ? (due
> to locking). But here even when i try using a single client the throughput
> is in 20-30 id/s range.
> Did you do any tuning in the server side. ?
>
> regards,
> Charith
>
> On Wed, Jun 13, 2012 at 6:17 AM, Sam Tunnicliffe <sa...@beobal.com> wrote:
>
>> Hi Charith,
>>
>> can you partition your counters somehow, rather than having a single
>> global counter? This would let you use multiple znodes, reducing
>> contention during the setData call.
>> We use almost exactly the same approach as you in our zookeeper backed
>> sequence server[1] but with sharded counters and haven't had any
>> issues with throughput (sorry, I don't have perf numbers to hand, but
>> peak load pushes a few hundred r/s through)
>>
>> Cheers,
>> Sam
>>
>> [1] https://github.com/talis/H1
>>
>> On 13 June 2012 05:20, Charith Wickramarachchi
>> <ch...@gmail.com> wrote:
>> > Hi All ,
>> >
>> > I'm evaluating Apache Zookeepers to use in my project where i have a
>> > requirement to generate unique and sequential message ids(long) across a
>> > cluster.
>> > After some web search i found[1] and some other different suggestions.
>> I
>> > started with the solution suggested in [1] but even with single client i
>> > only got a throughput around 20-30 ids/sec. But according to the
>> performance
>> > numbers pointed in the website i think i should be able get better
>> results (
>> > i m expecting to get at least 1000 id/s as i m using this for a MoM where
>> > for each message i'll be calling zk id generator.)
>> >
>> > In this case I m using ZK version 3.3.4 running in my local machine with
>> > default settings and client also running in the same machine. I m
>> attaching
>> > my id generator class.
>> >
>> > Any tips to improve the performance of this is highly appreciated. Am i
>> > doing some thing wrong ?
>> >
>> >
>> > [1]
>> http://zookeeper-user.578899.n2.nabble.com/Unique-Id-Generation-td2688494.html
>> >
>> > cheers,
>> > Charith
>> > --
>> > Charith Dhanushka Wickramarachchi
>> > http://charithwiki.blogspot.com/
>> >
>>
>
>
>
> --
> Charith Dhanushka Wickramarachchi
> http://charithwiki.blogspot.com/
Re: Sequential Id generation with Zookeeper - Tips to improve performance
Posted by Charith Wickramarachchi <ch...@gmail.com>.
Thanks Sam for the reply.
The contention happens when we to setData from muliple clients right ? (due
to locking). But here even when i try using a single client the throughput
is in 20-30 id/s range.
Did you do any tuning in the server side. ?
regards,
Charith
On Wed, Jun 13, 2012 at 6:17 AM, Sam Tunnicliffe <sa...@beobal.com> wrote:
> Hi Charith,
>
> can you partition your counters somehow, rather than having a single
> global counter? This would let you use multiple znodes, reducing
> contention during the setData call.
> We use almost exactly the same approach as you in our zookeeper backed
> sequence server[1] but with sharded counters and haven't had any
> issues with throughput (sorry, I don't have perf numbers to hand, but
> peak load pushes a few hundred r/s through)
>
> Cheers,
> Sam
>
> [1] https://github.com/talis/H1
>
> On 13 June 2012 05:20, Charith Wickramarachchi
> <ch...@gmail.com> wrote:
> > Hi All ,
> >
> > I'm evaluating Apache Zookeepers to use in my project where i have a
> > requirement to generate unique and sequential message ids(long) across a
> > cluster.
> > After some web search i found[1] and some other different suggestions.
> I
> > started with the solution suggested in [1] but even with single client i
> > only got a throughput around 20-30 ids/sec. But according to the
> performance
> > numbers pointed in the website i think i should be able get better
> results (
> > i m expecting to get at least 1000 id/s as i m using this for a MoM where
> > for each message i'll be calling zk id generator.)
> >
> > In this case I m using ZK version 3.3.4 running in my local machine with
> > default settings and client also running in the same machine. I m
> attaching
> > my id generator class.
> >
> > Any tips to improve the performance of this is highly appreciated. Am i
> > doing some thing wrong ?
> >
> >
> > [1]
> http://zookeeper-user.578899.n2.nabble.com/Unique-Id-Generation-td2688494.html
> >
> > cheers,
> > Charith
> > --
> > Charith Dhanushka Wickramarachchi
> > http://charithwiki.blogspot.com/
> >
>
--
Charith Dhanushka Wickramarachchi
http://charithwiki.blogspot.com/
Re: Sequential Id generation with Zookeeper - Tips to improve performance
Posted by Sam Tunnicliffe <sa...@beobal.com>.
Hi Charith,
can you partition your counters somehow, rather than having a single
global counter? This would let you use multiple znodes, reducing
contention during the setData call.
We use almost exactly the same approach as you in our zookeeper backed
sequence server[1] but with sharded counters and haven't had any
issues with throughput (sorry, I don't have perf numbers to hand, but
peak load pushes a few hundred r/s through)
Cheers,
Sam
[1] https://github.com/talis/H1
On 13 June 2012 05:20, Charith Wickramarachchi
<ch...@gmail.com> wrote:
> Hi All ,
>
> I'm evaluating Apache Zookeepers to use in my project where i have a
> requirement to generate unique and sequential message ids(long) across a
> cluster.
> After some web search i found[1] and some other different suggestions. I
> started with the solution suggested in [1] but even with single client i
> only got a throughput around 20-30 ids/sec. But according to the performance
> numbers pointed in the website i think i should be able get better results (
> i m expecting to get at least 1000 id/s as i m using this for a MoM where
> for each message i'll be calling zk id generator.)
>
> In this case I m using ZK version 3.3.4 running in my local machine with
> default settings and client also running in the same machine. I m attaching
> my id generator class.
>
> Any tips to improve the performance of this is highly appreciated. Am i
> doing some thing wrong ?
>
>
> [1]http://zookeeper-user.578899.n2.nabble.com/Unique-Id-Generation-td2688494.html
>
> cheers,
> Charith
> --
> Charith Dhanushka Wickramarachchi
> http://charithwiki.blogspot.com/
>