You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Pramod Biligiri <pr...@gmail.com> on 2014/05/24 05:45:29 UTC
Request for some sample Zookeeper cluster sizes
Hi,
I am running some performance tests against Zookeeper changes that I'm
making. I would like to know a couple of real world Zookeeper deployment
sizes so that I can be reasonably sure that I'm testing the right thing.
If anyone is in a position to share some info from Zookeeper deployments
that they are aware of, it'd be very helpful. I'm also considering running
these benchmarks - https://github.com/brownsys/zookeeper-benchmark
Any experience with that?
Things I'd like to know: How many Zookeeper nodes in the ensemble, what
kind of hardware are they running on (esp. disk), how much memory is
allocated to the JVM?
If you can give me a characterization of the Zookeeper workload, that would
be the icing on the cake :)
Thanks,
Pramod
--
http://twitter.com/pramodbiligiri
RE: Request for some sample Zookeeper cluster sizes
Posted by FPJ <fp...@yahoo.com>.
I have placed similar requests in the past for such traces and never got a reply. Perhaps the zk-smoketests by phunt would be a good starting point for you.
-Flavio
> -----Original Message-----
> From: Pramod Biligiri [mailto:pramodbiligiri@gmail.com]
> Sent: 29 May 2014 11:59
> To: user@zookeeper.apache.org
> Subject: Re: Request for some sample Zookeeper cluster sizes
>
> Just wanted to place a reminder for this request, in case anyone is able to
> help out. We're about to start performance testing some features, and
> would find info about realistic cluster sizes and hardware configs very handy
> right now.
>
> Thanks,
> Pramod
>
>
> On Fri, May 23, 2014 at 8:45 PM, Pramod Biligiri <pr...@gmail.com>
> wrote:
>
> > Hi,
> > I am running some performance tests against Zookeeper changes that I'm
> > making. I would like to know a couple of real world Zookeeper
> > deployment sizes so that I can be reasonably sure that I'm testing the right
> thing.
> >
> > If anyone is in a position to share some info from Zookeeper
> > deployments that they are aware of, it'd be very helpful. I'm also
> > considering running these benchmarks -
> > https://github.com/brownsys/zookeeper-benchmark
> > Any experience with that?
> >
> > Things I'd like to know: How many Zookeeper nodes in the ensemble,
> > what kind of hardware are they running on (esp. disk), how much memory
> > is allocated to the JVM?
> >
> > If you can give me a characterization of the Zookeeper workload, that
> > would be the icing on the cake :)
> >
> > Thanks,
> > Pramod
> > --
> > http://twitter.com/pramodbiligiri
> >
Re: Request for some sample Zookeeper cluster sizes
Posted by Pramod Biligiri <pr...@gmail.com>.
Just wanted to place a reminder for this request, in case anyone is able to
help out. We're about to start performance testing some features, and would
find info about realistic cluster sizes and hardware configs very handy
right now.
Thanks,
Pramod
On Fri, May 23, 2014 at 8:45 PM, Pramod Biligiri <pr...@gmail.com>
wrote:
> Hi,
> I am running some performance tests against Zookeeper changes that I'm
> making. I would like to know a couple of real world Zookeeper deployment
> sizes so that I can be reasonably sure that I'm testing the right thing.
>
> If anyone is in a position to share some info from Zookeeper deployments
> that they are aware of, it'd be very helpful. I'm also considering running
> these benchmarks - https://github.com/brownsys/zookeeper-benchmark
> Any experience with that?
>
> Things I'd like to know: How many Zookeeper nodes in the ensemble, what
> kind of hardware are they running on (esp. disk), how much memory is
> allocated to the JVM?
>
> If you can give me a characterization of the Zookeeper workload, that
> would be the icing on the cake :)
>
> Thanks,
> Pramod
> --
> http://twitter.com/pramodbiligiri
>
Re: Request for some sample Zookeeper cluster sizes
Posted by Pramod Biligiri <pr...@gmail.com>.
Hi all,
Thanks for the info so far. It gives us a good starting point. I was able
to run Patrick Hunt's zk-latencies.py, should be handy in the future.
Pramod
On Thu, May 29, 2014 at 10:09 AM, ralph tice <ra...@gmail.com> wrote:
> We've found that c3.large instances in Amazon AWS work well for ZooKeeper
> instances. We run 5 nodes of ZK across 3 Availability Zones. Load is
> almost always zero, around 50-200kb/s network traffic in/out, ~500-750kb/s
> disk writes, ~135MB RAM used per node. JVM heap settings are default which
> works out to ~937.5MB on a c3.large.
>
> Each node is handling ~50 connections and doing about 100M sent/received a
> month. Our node count is pretty low though, only a few thousand. We use
> ZK for coordinating Apache Kafka as well as service discovery.
>
> Does anyone run ZK anywhere close to "hot", i.e. 60-80% machine
> utilization?
>
>
>
> On Thu, May 29, 2014 at 8:09 AM, James A. Robinson <
> jimr@highwire.stanford.edu> wrote:
>
> > Not really much information, but we're running a 5-node
> > zookeeper cluster, all the nodes are within the same
> > local network. We're using it to help coordinate Apache
> > Kafka and we're going to be using it to help coordinate
> > a bit of configuration for some web crawlers we'll be running
> > against sites we manage.
> >
> > Jim
> >
>
Re: Request for some sample Zookeeper cluster sizes
Posted by ralph tice <ra...@gmail.com>.
We've found that c3.large instances in Amazon AWS work well for ZooKeeper
instances. We run 5 nodes of ZK across 3 Availability Zones. Load is
almost always zero, around 50-200kb/s network traffic in/out, ~500-750kb/s
disk writes, ~135MB RAM used per node. JVM heap settings are default which
works out to ~937.5MB on a c3.large.
Each node is handling ~50 connections and doing about 100M sent/received a
month. Our node count is pretty low though, only a few thousand. We use
ZK for coordinating Apache Kafka as well as service discovery.
Does anyone run ZK anywhere close to "hot", i.e. 60-80% machine utilization?
On Thu, May 29, 2014 at 8:09 AM, James A. Robinson <
jimr@highwire.stanford.edu> wrote:
> Not really much information, but we're running a 5-node
> zookeeper cluster, all the nodes are within the same
> local network. We're using it to help coordinate Apache
> Kafka and we're going to be using it to help coordinate
> a bit of configuration for some web crawlers we'll be running
> against sites we manage.
>
> Jim
>
Re: Request for some sample Zookeeper cluster sizes
Posted by "James A. Robinson" <ji...@highwire.stanford.edu>.
Not really much information, but we're running a 5-node
zookeeper cluster, all the nodes are within the same
local network. We're using it to help coordinate Apache
Kafka and we're going to be using it to help coordinate
a bit of configuration for some web crawlers we'll be running
against sites we manage.
Jim