You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by rammohan ganapavarapu <ra...@gmail.com> on 2019/07/16 17:04:39 UTC
Zookeeper latency calculation
Hi,
I am trying to understand how zookeeper latency calculated, mntr command
always give avg_latency "0", can some one help how to calculate avg request
latency in zookeeper?
Thanks,
Ram
Re: Zookeeper latency calculation
Posted by Norbert Kalmar <nk...@cloudera.com.INVALID>.
It's integer, I can confirm.
On Wed, Jul 17, 2019 at 9:38 AM Andor Molnar <an...@apache.org> wrote:
> If I recall correctly avg_latency is an int, not float.I remember
> someone wanted to replace it with a float sometime recently.
> Correct me if I'm wrong.
> Andor
>
>
> -----Original Message-----From: Norbert Kalmar <
> nkalmar@cloudera.com.INVALID>Reply-To: user@zookeeper.apache.orgTo:
> user@zookeeper.apache.orgSubject: Re: Zookeeper latency
> calculationDate: Wed, 17 Jul 2019 09:27:35 +0200
> Hi Ram,
> ZooKeeper is very fast if deployed according to recommendations (nodes
> onthe same network, directly connected). It's possible it gives 0
> latency onavg, although usually it's a bit higher.I can recommend
> Patrick's smoke test if you wan't to test performance.Especially zk-
> smoketest and zk-latencies.py. It has a good readme:
> https://github.com/phunt/zk-smoketest
>
> But "mntr" command is pretty much the easiest tool out of the
> box.Especially if you are on 3.4.x
> Regards,Norbert
> On Tue, Jul 16, 2019 at 7:05 PM rammohan ganapavarapu <
> rammohanganap@gmail.com> wrote:
> > Hi,
> > I am trying to understand how zookeeper latency calculated, mntr
> > commandalways give avg_latency "0", can some one help how to
> > calculate avg requestlatency in zookeeper?
> >
> > Thanks,Ram
>
Re: Zookeeper latency calculation
Posted by Andor Molnar <an...@apache.org>.
If I recall correctly avg_latency is an int, not float.I remember
someone wanted to replace it with a float sometime recently.
Correct me if I'm wrong.
Andor
-----Original Message-----From: Norbert Kalmar <
nkalmar@cloudera.com.INVALID>Reply-To: user@zookeeper.apache.orgTo:
user@zookeeper.apache.orgSubject: Re: Zookeeper latency
calculationDate: Wed, 17 Jul 2019 09:27:35 +0200
Hi Ram,
ZooKeeper is very fast if deployed according to recommendations (nodes
onthe same network, directly connected). It's possible it gives 0
latency onavg, although usually it's a bit higher.I can recommend
Patrick's smoke test if you wan't to test performance.Especially zk-
smoketest and zk-latencies.py. It has a good readme:
https://github.com/phunt/zk-smoketest
But "mntr" command is pretty much the easiest tool out of the
box.Especially if you are on 3.4.x
Regards,Norbert
On Tue, Jul 16, 2019 at 7:05 PM rammohan ganapavarapu <
rammohanganap@gmail.com> wrote:
> Hi,
> I am trying to understand how zookeeper latency calculated, mntr
> commandalways give avg_latency "0", can some one help how to
> calculate avg requestlatency in zookeeper?
>
> Thanks,Ram
Re: Zookeeper latency calculation
Posted by Norbert Kalmar <nk...@cloudera.com.INVALID>.
Hi Ram,
ZooKeeper is very fast if deployed according to recommendations (nodes on
the same network, directly connected). It's possible it gives 0 latency on
avg, although usually it's a bit higher.
I can recommend Patrick's smoke test if you wan't to test performance.
Especially zk-smoketest and zk-latencies.py. It has a good readme:
https://github.com/phunt/zk-smoketest
But "mntr" command is pretty much the easiest tool out of the box.
Especially if you are on 3.4.x
Regards,
Norbert
On Tue, Jul 16, 2019 at 7:05 PM rammohan ganapavarapu <
rammohanganap@gmail.com> wrote:
> Hi,
>
> I am trying to understand how zookeeper latency calculated, mntr command
> always give avg_latency "0", can some one help how to calculate avg request
> latency in zookeeper?
>
>
> Thanks,
> Ram
>
Re: Zookeeper latency calculation
Posted by Patrick Hunt <ph...@apache.org>.
High max latency is typically due to gc, slow disk, or issues with vm
oversubscription. Turn on gc logging and verify. I've used strace in the
past to troubleshoot disk issues (fsync - although now a days it's in the
log).
Patrick
On Thu, Jul 18, 2019 at 8:40 AM rammohan ganapavarapu <
rammohanganap@gmail.com> wrote:
> I do have a busy system total packet processed by zk is like 60k/s (30k
> send and 30k received) and how about the max latencies? is ti always
> constant ? for me avg latency is 0 but max latency is showing 40s how is it
> possible?
>
>
>
>
> On Wed, Jul 17, 2019 at 8:46 PM Michael Han <ha...@apache.org> wrote:
>
> > Typing send too soon...
> >
> > >> always give avg_latency "0"
> >
> > The latency metrics depends on workloads. Try hit your cluster hard with
> > some artificially generated heavy read / write workloads, you will see
> the
> > number deviates from 0.
> >
> > >> If I recall correctly avg_latency is an int, not float
> >
> > This is now a float on master branch, and the change was made in
> > ZOOKEEPER-2641. I remember this because this actually breaks one of our
> > internal metrics system, where our system expects an int (the old type).
> >
> > On Wed, Jul 17, 2019 at 8:43 PM Michael Han <ha...@apache.org> wrote:
> >
> > > >> always give avg_latency "0"
> > >
> > > The latency metrics depends on workloads.
> > >
> > > On Wed, Jul 17, 2019 at 1:34 AM Enrico Olivelli <eo...@gmail.com>
> > > wrote:
> > >
> > >> Il mar 16 lug 2019, 19:05 rammohan ganapavarapu <
> > rammohanganap@gmail.com>
> > >> ha scritto:
> > >>
> > >> > Hi,
> > >> >
> > >> > I am trying to understand how zookeeper latency calculated, mntr
> > command
> > >> > always give avg_latency "0", can some one help how to calculate avg
> > >> request
> > >> > latency in zookeeper?
> > >> >
> > >>
> > >>
> > >> Are you also taking metrics on the client?
> > >> Maybe on the server side the value is biased by very fast ops
> > >>
> > >>
> > >> Enrico
> > >>
> > >> >
> > >> >
> > >> > Thanks,
> > >> > Ram
> > >> >
> > >>
> > >
> >
>
Re: Zookeeper latency calculation
Posted by rammohan ganapavarapu <ra...@gmail.com>.
I do have a busy system total packet processed by zk is like 60k/s (30k
send and 30k received) and how about the max latencies? is ti always
constant ? for me avg latency is 0 but max latency is showing 40s how is it
possible?
On Wed, Jul 17, 2019 at 8:46 PM Michael Han <ha...@apache.org> wrote:
> Typing send too soon...
>
> >> always give avg_latency "0"
>
> The latency metrics depends on workloads. Try hit your cluster hard with
> some artificially generated heavy read / write workloads, you will see the
> number deviates from 0.
>
> >> If I recall correctly avg_latency is an int, not float
>
> This is now a float on master branch, and the change was made in
> ZOOKEEPER-2641. I remember this because this actually breaks one of our
> internal metrics system, where our system expects an int (the old type).
>
> On Wed, Jul 17, 2019 at 8:43 PM Michael Han <ha...@apache.org> wrote:
>
> > >> always give avg_latency "0"
> >
> > The latency metrics depends on workloads.
> >
> > On Wed, Jul 17, 2019 at 1:34 AM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> >> Il mar 16 lug 2019, 19:05 rammohan ganapavarapu <
> rammohanganap@gmail.com>
> >> ha scritto:
> >>
> >> > Hi,
> >> >
> >> > I am trying to understand how zookeeper latency calculated, mntr
> command
> >> > always give avg_latency "0", can some one help how to calculate avg
> >> request
> >> > latency in zookeeper?
> >> >
> >>
> >>
> >> Are you also taking metrics on the client?
> >> Maybe on the server side the value is biased by very fast ops
> >>
> >>
> >> Enrico
> >>
> >> >
> >> >
> >> > Thanks,
> >> > Ram
> >> >
> >>
> >
>
Re: Zookeeper latency calculation
Posted by Michael Han <ha...@apache.org>.
Typing send too soon...
>> always give avg_latency "0"
The latency metrics depends on workloads. Try hit your cluster hard with
some artificially generated heavy read / write workloads, you will see the
number deviates from 0.
>> If I recall correctly avg_latency is an int, not float
This is now a float on master branch, and the change was made in
ZOOKEEPER-2641. I remember this because this actually breaks one of our
internal metrics system, where our system expects an int (the old type).
On Wed, Jul 17, 2019 at 8:43 PM Michael Han <ha...@apache.org> wrote:
> >> always give avg_latency "0"
>
> The latency metrics depends on workloads.
>
> On Wed, Jul 17, 2019 at 1:34 AM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
>> Il mar 16 lug 2019, 19:05 rammohan ganapavarapu <ra...@gmail.com>
>> ha scritto:
>>
>> > Hi,
>> >
>> > I am trying to understand how zookeeper latency calculated, mntr command
>> > always give avg_latency "0", can some one help how to calculate avg
>> request
>> > latency in zookeeper?
>> >
>>
>>
>> Are you also taking metrics on the client?
>> Maybe on the server side the value is biased by very fast ops
>>
>>
>> Enrico
>>
>> >
>> >
>> > Thanks,
>> > Ram
>> >
>>
>
Re: Zookeeper latency calculation
Posted by Michael Han <ha...@apache.org>.
>> always give avg_latency "0"
The latency metrics depends on workloads.
On Wed, Jul 17, 2019 at 1:34 AM Enrico Olivelli <eo...@gmail.com> wrote:
> Il mar 16 lug 2019, 19:05 rammohan ganapavarapu <ra...@gmail.com>
> ha scritto:
>
> > Hi,
> >
> > I am trying to understand how zookeeper latency calculated, mntr command
> > always give avg_latency "0", can some one help how to calculate avg
> request
> > latency in zookeeper?
> >
>
>
> Are you also taking metrics on the client?
> Maybe on the server side the value is biased by very fast ops
>
>
> Enrico
>
> >
> >
> > Thanks,
> > Ram
> >
>
Re: Zookeeper latency calculation
Posted by Enrico Olivelli <eo...@gmail.com>.
Il mar 16 lug 2019, 19:05 rammohan ganapavarapu <ra...@gmail.com>
ha scritto:
> Hi,
>
> I am trying to understand how zookeeper latency calculated, mntr command
> always give avg_latency "0", can some one help how to calculate avg request
> latency in zookeeper?
>
Are you also taking metrics on the client?
Maybe on the server side the value is biased by very fast ops
Enrico
>
>
> Thanks,
> Ram
>