You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@jmeter.apache.org by bu...@apache.org on 2014/01/03 13:57:20 UTC
[Bug 55932] Create a Graphite Listener
https://issues.apache.org/bugzilla/show_bug.cgi?id=55932
--- Comment #6 from Sebb <se...@apache.org> ---
I have been having a look at the implementation.
I don't really see that it needs Commons Math; we aleady have StatCalculator
which handles percentiles and more.
Likewise, does it really need Commons Pool?
It seems wrong to have to have 2 separate pools of SocketOutputStream
instances.
How many of these would there be?
Also, DescriptiveStatistics is not thread-safe (nor is StatCalculator).
If we do implement something like this, I think the data processing needs
either to be carefully synchronised, or the raw data should be sent to a
separate singleton background thread.
Follow-ups to the dev list please.
--
You are receiving this mail because:
You are the assignee for the bug.
Re: [Bug 55932] Create a Graphite Listener
Posted by sebb <se...@gmail.com>.
On 3 January 2014 14:45, Philippe Mouawad <ph...@gmail.com> wrote:
> On Fri, Jan 3, 2014 at 3:33 PM, sebb <se...@gmail.com> wrote:
>
>> On 3 January 2014 13:47, Philippe Mouawad <ph...@gmail.com>
>> wrote:
>> > Hello Sebb,
>> > As requested although I don't see why it cannot be in bugzilla,if it were
>> > the history of discussion could be found more easily.
>>
>> Because discussion may take a while and Bugzilla is not good for that.
>> We can add links to the mail thread with any followup, once decisions are
>> made.
>>
>> > Regards
>> > Philippe
>> > On Fri, Jan 3, 2014 at 1:57 PM, <bu...@apache.org> wrote:
>> >
>> >> https://issues.apache.org/bugzilla/show_bug.cgi?id=55932
>> >>
>> >> --- Comment #6 from Sebb <se...@apache.org> ---
>> >> I have been having a look at the implementation.
>> >>
>> >> I don't really see that it needs Commons Math; we aleady have
>> >> StatCalculator
>> >> which handles percentiles and more.
>> >>
>> > Ok I can change this.
>> >
>> >>
>> >> Likewise, does it really need Commons Pool?
>> >> It seems wrong to have to have 2 separate pools of SocketOutputStream
>> >> instances.
>> >>
>> > Can you clarify ?
>> > There is an executor pool (max size : 1 for now) and a
>> >
>> > socketOutputStreamPool in GraphiteMetricsManager.
>>
>> Sorry, that was wrong.
>>
>> There's a socketOutputStreamPool in PickleMetricsManager only
>> I thought I had spotted another in SocketOutputStreamPoolFactory but
>> that is just the implementation.
>>
>> >
>> > How many of these would there be?
>> >>
>> >
>> > Currently it is true we could use only one socket and keep it open.
>> >
>> >
>> >>
>> >> Also, DescriptiveStatistics is not thread-safe (nor is StatCalculator).
>> >>
>> > It is not a problem, as DescriptiveStatistics is accessed synchronously.
>>
>> Only write accesses are synchronised.
>> But read needs to be synchronised as well to ensure safe publication.
>>
>> >>
>> >> If we do implement something like this, I think the data processing
>> needs
>> >> either to be carefully synchronised, or the raw data should be sent to a
>> >> separate singleton background thread.
>> >>
>> >
>> > I think it is carefully synchronized in the patch. If not please point me
>> > to where you see an issue.
>>
>> See above.
>>
>> If it used a background thread with a suitable queuing mechanism,
>> there would be no need to synch the data collection, and the
>> processing would not slow the main thread.
>>
> So you mean we publish the SampleResult in a queue and a thread handles
> computation ?
Yes.
We would need to define a suitable API (interface) here so that
different implementations can be plugged in.
> Do you intend to implement a kind of RingBuffer , something like this :
> https://github.com/LMAX-Exchange/disruptor
> or much simpler ?
I was thinking of one of the standard Java queues like we already use
for returning samples in client server mode.
But it might be worth looking at disruptor.
> And is there another thread to send ?
Probably not necessary, and might be counter-productive as it would
likely need additonal sync and another queue.
> Could you propose a patch change ?
>
>
>
>>
>> What happens currently if the backend socket stalls or runs slowly?
>>
> It will only affect the publication .
> The run method will be executed while the other one is still running , so
> we could send same statistics twice, good catch.
>
>
>>
>> Also, decoupling in this way would make it easier to provide
>> implementations, as they would not need to be thread-safe.
>>
>> >
>> >>
>> >> Follow-ups to the dev list please.
>> >>
>> >> --
>> >> You are receiving this mail because:
>> >> You are the assignee for the bug.
>> >>
>> >
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
Re: [Bug 55932] Create a Graphite Listener
Posted by Philippe Mouawad <ph...@gmail.com>.
On Fri, Jan 3, 2014 at 3:33 PM, sebb <se...@gmail.com> wrote:
> On 3 January 2014 13:47, Philippe Mouawad <ph...@gmail.com>
> wrote:
> > Hello Sebb,
> > As requested although I don't see why it cannot be in bugzilla,if it were
> > the history of discussion could be found more easily.
>
> Because discussion may take a while and Bugzilla is not good for that.
> We can add links to the mail thread with any followup, once decisions are
> made.
>
> > Regards
> > Philippe
> > On Fri, Jan 3, 2014 at 1:57 PM, <bu...@apache.org> wrote:
> >
> >> https://issues.apache.org/bugzilla/show_bug.cgi?id=55932
> >>
> >> --- Comment #6 from Sebb <se...@apache.org> ---
> >> I have been having a look at the implementation.
> >>
> >> I don't really see that it needs Commons Math; we aleady have
> >> StatCalculator
> >> which handles percentiles and more.
> >>
> > Ok I can change this.
> >
> >>
> >> Likewise, does it really need Commons Pool?
> >> It seems wrong to have to have 2 separate pools of SocketOutputStream
> >> instances.
> >>
> > Can you clarify ?
> > There is an executor pool (max size : 1 for now) and a
> >
> > socketOutputStreamPool in GraphiteMetricsManager.
>
> Sorry, that was wrong.
>
> There's a socketOutputStreamPool in PickleMetricsManager only
> I thought I had spotted another in SocketOutputStreamPoolFactory but
> that is just the implementation.
>
> >
> > How many of these would there be?
> >>
> >
> > Currently it is true we could use only one socket and keep it open.
> >
> >
> >>
> >> Also, DescriptiveStatistics is not thread-safe (nor is StatCalculator).
> >>
> > It is not a problem, as DescriptiveStatistics is accessed synchronously.
>
> Only write accesses are synchronised.
> But read needs to be synchronised as well to ensure safe publication.
>
> >>
> >> If we do implement something like this, I think the data processing
> needs
> >> either to be carefully synchronised, or the raw data should be sent to a
> >> separate singleton background thread.
> >>
> >
> > I think it is carefully synchronized in the patch. If not please point me
> > to where you see an issue.
>
> See above.
>
> If it used a background thread with a suitable queuing mechanism,
> there would be no need to synch the data collection, and the
> processing would not slow the main thread.
>
So you mean we publish the SampleResult in a queue and a thread handles
computation ?
Do you intend to implement a kind of RingBuffer , something like this :
https://github.com/LMAX-Exchange/disruptor
or much simpler ?
And is there another thread to send ?
Could you propose a patch change ?
>
> What happens currently if the backend socket stalls or runs slowly?
>
It will only affect the publication .
The run method will be executed while the other one is still running , so
we could send same statistics twice, good catch.
>
> Also, decoupling in this way would make it easier to provide
> implementations, as they would not need to be thread-safe.
>
> >
> >>
> >> Follow-ups to the dev list please.
> >>
> >> --
> >> You are receiving this mail because:
> >> You are the assignee for the bug.
> >>
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
>
--
Cordialement.
Philippe Mouawad.
Re: [Bug 55932] Create a Graphite Listener
Posted by sebb <se...@gmail.com>.
On 3 January 2014 13:47, Philippe Mouawad <ph...@gmail.com> wrote:
> Hello Sebb,
> As requested although I don't see why it cannot be in bugzilla,if it were
> the history of discussion could be found more easily.
Because discussion may take a while and Bugzilla is not good for that.
We can add links to the mail thread with any followup, once decisions are made.
> Regards
> Philippe
> On Fri, Jan 3, 2014 at 1:57 PM, <bu...@apache.org> wrote:
>
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=55932
>>
>> --- Comment #6 from Sebb <se...@apache.org> ---
>> I have been having a look at the implementation.
>>
>> I don't really see that it needs Commons Math; we aleady have
>> StatCalculator
>> which handles percentiles and more.
>>
> Ok I can change this.
>
>>
>> Likewise, does it really need Commons Pool?
>> It seems wrong to have to have 2 separate pools of SocketOutputStream
>> instances.
>>
> Can you clarify ?
> There is an executor pool (max size : 1 for now) and a
>
> socketOutputStreamPool in GraphiteMetricsManager.
Sorry, that was wrong.
There's a socketOutputStreamPool in PickleMetricsManager only
I thought I had spotted another in SocketOutputStreamPoolFactory but
that is just the implementation.
>
> How many of these would there be?
>>
>
> Currently it is true we could use only one socket and keep it open.
>
>
>>
>> Also, DescriptiveStatistics is not thread-safe (nor is StatCalculator).
>>
> It is not a problem, as DescriptiveStatistics is accessed synchronously.
Only write accesses are synchronised.
But read needs to be synchronised as well to ensure safe publication.
>>
>> If we do implement something like this, I think the data processing needs
>> either to be carefully synchronised, or the raw data should be sent to a
>> separate singleton background thread.
>>
>
> I think it is carefully synchronized in the patch. If not please point me
> to where you see an issue.
See above.
If it used a background thread with a suitable queuing mechanism,
there would be no need to synch the data collection, and the
processing would not slow the main thread.
What happens currently if the backend socket stalls or runs slowly?
Also, decoupling in this way would make it easier to provide
implementations, as they would not need to be thread-safe.
>
>>
>> Follow-ups to the dev list please.
>>
>> --
>> You are receiving this mail because:
>> You are the assignee for the bug.
>>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
Re: [Bug 55932] Create a Graphite Listener
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello Sebb,
As requested although I don't see why it cannot be in bugzilla,if it were
the history of discussion could be found more easily.
Regards
Philippe
On Fri, Jan 3, 2014 at 1:57 PM, <bu...@apache.org> wrote:
> https://issues.apache.org/bugzilla/show_bug.cgi?id=55932
>
> --- Comment #6 from Sebb <se...@apache.org> ---
> I have been having a look at the implementation.
>
> I don't really see that it needs Commons Math; we aleady have
> StatCalculator
> which handles percentiles and more.
>
Ok I can change this.
>
> Likewise, does it really need Commons Pool?
> It seems wrong to have to have 2 separate pools of SocketOutputStream
> instances.
>
Can you clarify ?
There is an executor pool (max size : 1 for now) and a
socketOutputStreamPool in GraphiteMetricsManager.
How many of these would there be?
>
Currently it is true we could use only one socket and keep it open.
>
> Also, DescriptiveStatistics is not thread-safe (nor is StatCalculator).
>
It is not a problem, as DescriptiveStatistics is accessed synchronously.
>
> If we do implement something like this, I think the data processing needs
> either to be carefully synchronised, or the raw data should be sent to a
> separate singleton background thread.
>
I think it is carefully synchronized in the patch. If not please point me
to where you see an issue.
>
> Follow-ups to the dev list please.
>
> --
> You are receiving this mail because:
> You are the assignee for the bug.
>
--
Cordialement.
Philippe Mouawad.
Re: [Bug 55932] Create a Graphite Listener
Posted by sebb <se...@gmail.com>.
On 3 January 2014 22:58, Rainer Jung <ra...@kippdata.de> wrote:
> On 03.01.2014 13:57, bugzilla@apache.org wrote:
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=55932
>>
>> --- Comment #6 from Sebb <se...@apache.org> ---
>> I have been having a look at the implementation.
>>
>> I don't really see that it needs Commons Math; we aleady have StatCalculator
>> which handles percentiles and more.
>>
>> Likewise, does it really need Commons Pool?
>> It seems wrong to have to have 2 separate pools of SocketOutputStream
>> instances.
>> How many of these would there be?
>>
>> Also, DescriptiveStatistics is not thread-safe (nor is StatCalculator).
>>
>> If we do implement something like this, I think the data processing needs
>> either to be carefully synchronised, or the raw data should be sent to a
>> separate singleton background thread.
>
> FWIW: I always get a bit nervous when percentiles are calculated.
> Percentiles are expensive to calculate if one needs exact results with
> given percentage numbers (50%, 99%, 99.9% etc.). In that case one needs
> to keep all values as an ordered list to calculate the percentiles. For
> a long running test that would be expensive in terms of memory but also
> in terms of CPU (sorting). There's no way of exactly merging percentiles
> from interim statistical data.
>
> Sometimes approximations are enough. By approximation I don't mean
> estimated data, but percentages which are not exactly the ones you are
> keen for. E.g. you would get a 48% value instead of a 50% value, or a
> 99.02% value instead of a 99% value.
>
> Suppose you would know (configure) that only very few samples will take
> longer than 1000ms, then one could create fixed bins for e.g. 10ms,
> 15ms, 20ms, 25ms, 30ms, 40ms, 50ms, 75ms, 100ms, 150ms, 200ms, 250ms,
> 300ms, 400ms, 500ms, 750ms and 1000ms. Now whenever a sample finishes
> you count the sample in the bin it belongs to and do not save the data
> (of course you can still log it). At any time you can now look at the
> not need to keep all sample values around and sort them, but one does
> also not get equidistant percentiles (10%, 11%, 12%, ...).
StatCalculator already takes a similar approach, counting values
rather than storing them.
We already use it for the GUI listeners.
There are other approaches; Commons Math DescriptiveStatistics uses an
array of doubles (with a sliding window).
And there is also the following:
http://search-lucene.com/jd/mahout/math/org/apache/mahout/math/stats/OnlineSummarizer.html
However, AFAICT it does not support arbitrary percentiles, only quartiles.
Re: [Bug 55932] Create a Graphite Listener
Posted by sebb <se...@gmail.com>.
On 3 January 2014 23:22, Philippe Mouawad <ph...@gmail.com> wrote:
> Thanks Rainer, of great help!
> Regarding graphite, it offers a function to compute percentiles, so we
> could as sebb proposed it send raw results.
No, I was not proposing that.
I did not know that Graphite could do the calculations.
I was proposing that the calculations were done in the back-end thread
before sending to graphite.
The point was to separate the collection and processing of the data.
However it would be nice if graphite supported aggregated data, that
could reduce the amount to be sent to it.
> My concern was about the memory and network impact of sending all sample
> results to backend.
Likewise.
> So should we create what you propose, do you know an existing library that
> implements it already ?
See my reply to Rainer.
StatCalculator does it already.
>
> On Friday, January 3, 2014, Rainer Jung wrote:
>
>> On 03.01.2014 13:57, bugzilla@apache.org <javascript:;> wrote:
>> > https://issues.apache.org/bugzilla/show_bug.cgi?id=55932
>> >
>> > --- Comment #6 from Sebb <sebb@apache.org <javascript:;>> ---
>> > I have been having a look at the implementation.
>> >
>> > I don't really see that it needs Commons Math; we aleady have
>> StatCalculator
>> > which handles percentiles and more.
>> >
>> > Likewise, does it really need Commons Pool?
>> > It seems wrong to have to have 2 separate pools of SocketOutputStream
>> > instances.
>> > How many of these would there be?
>> >
>> > Also, DescriptiveStatistics is not thread-safe (nor is StatCalculator).
>> >
>> > If we do implement something like this, I think the data processing needs
>> > either to be carefully synchronised, or the raw data should be sent to a
>> > separate singleton background thread.
>>
>> FWIW: I always get a bit nervous when percentiles are calculated.
>> Percentiles are expensive to calculate if one needs exact results with
>> given percentage numbers (50%, 99%, 99.9% etc.). In that case one needs
>> to keep all values as an ordered list to calculate the percentiles. For
>> a long running test that would be expensive in terms of memory but also
>> in terms of CPU (sorting). There's no way of exactly merging percentiles
>> from interim statistical data.
>>
>> Sometimes approximations are enough. By approximation I don't mean
>> estimated data, but percentages which are not exactly the ones you are
>> keen for. E.g. you would get a 48% value instead of a 50% value, or a
>> 99.02% value instead of a 99% value.
>>
>> Suppose you would know (configure) that only very few samples will take
>> longer than 1000ms, then one could create fixed bins for e.g. 10ms,
>> 15ms, 20ms, 25ms, 30ms, 40ms, 50ms, 75ms, 100ms, 150ms, 200ms, 250ms,
>> 300ms, 400ms, 500ms, 750ms and 1000ms. Now whenever a sample finishes
>> you count the sample in the bin it belongs to and do not save the data
>> (of course you can still log it). At any time you can now look at the
>> bin counters and cheaply produce quantiles. For example suppose the bin
>> counters look like that:
>>
>> Duration binCount
>> 10ms 3
>> 15ms 2
>> 20ms 5
>> 25ms 10
>> 30ms 8
>> 40ms 20
>> 50ms 28
>> 75ms 100
>> 100ms 230
>> 150ms 610
>> 200ms 780
>> 250ms 530
>> 300ms 220
>> 400ms 200
>> 500ms 80
>> 750ms 90
>> 1000ms 50
>> >1000ms 30
>>
>> Now we sum up the bin count into cumulated count and take percentages:
>>
>> Duration binCount Cum. Cum.Pct binPct
>> 10ms 3 3 0.10% 0.10%
>> 15ms 2 5 0.17% 0.07%
>> 20ms 5 10 0.33% 0.17%
>> 25ms 10 20 0.67% 0.33%
>> 30ms 8 28 0.93% 0.27%
>> 40ms 20 48 1.60% 0.67%
>> 50ms 28 76 2.54% 0.93%
>> 75ms 100 176 5.87% 3.34%
>> 100ms 230 406 13.55% 7.68%
>> 150ms 610 1016 33.91% 20.36%
>> 200ms 780 1796 59.95% 26.03%
>> 250ms 530 2326 77.64% 17.69%
>> 300ms 220 2546 84.98% 7.34%
>> 400ms 200 2746 91.66% 6.68%
>> 500ms 80 2826 94.33% 2.67%
>> 750ms 90 2916 97.33% 3.00%
>> 1000ms 50 2966 99.00% 1.67%
>> >1000ms 30 2996 100.00% 1.00%
>>
>> The Cum.Pct column could be used instead of the percentiles. One does
>> not need to keep all sample values around and sort them, but one does
>> also not get equidistant percentiles (10%, 11%, 12%, ...).
>>
>> Making the table more fine grained by using more rows is cheap. I have
>> chosen here a short table to make the example easier to understand. The
>> duration limits for the bins I had chosen were human friendly (integral
>> numbers, often divisible by 10 or 100), but one could also use a
>> mathematically strict series like e.g. 100 logarithmic steps or 10
>> logarithmic steps for each factor 10 in duration time. The bin limits
>> would then increase by 26% from bin to bin, e.g. 100ms, 126ms, 158ms,
>> 200ms, 251ms, 316ms, 398ms, 501ms, 631ms, 794ms, 1000ms.
>>
>> Users who need integral percentiles, like exactly 50% or 99% values
>> instead of 48.2% or 99.05% values (example) would need to either post
>> process the sample logs or choose a more expensive live processing (in
>> terms of memory and cpu be it inside JMeter or another backend). I'd
>> expect, that often the non-integral percentiles above are good enough to
>> follow a running test and get sufficient data about its behavior (and
>> are cheap enough to produce on the fly) and the integral percentiles are
>> OK to generate during post-processing and full report generation after a
>> run.
>>
>> Just my 2c.
>>
>> Regards,
>>
>> Rainer
>>
>
>
> --
> Cordialement.
> Philippe Mouawad.
Re: [Bug 55932] Create a Graphite Listener
Posted by Philippe Mouawad <ph...@gmail.com>.
Thanks Rainer, of great help!
Regarding graphite, it offers a function to compute percentiles, so we
could as sebb proposed it send raw results.
My concern was about the memory and network impact of sending all sample
results to backend.
So should we create what you propose, do you know an existing library that
implements it already ?
On Friday, January 3, 2014, Rainer Jung wrote:
> On 03.01.2014 13:57, bugzilla@apache.org <javascript:;> wrote:
> > https://issues.apache.org/bugzilla/show_bug.cgi?id=55932
> >
> > --- Comment #6 from Sebb <sebb@apache.org <javascript:;>> ---
> > I have been having a look at the implementation.
> >
> > I don't really see that it needs Commons Math; we aleady have
> StatCalculator
> > which handles percentiles and more.
> >
> > Likewise, does it really need Commons Pool?
> > It seems wrong to have to have 2 separate pools of SocketOutputStream
> > instances.
> > How many of these would there be?
> >
> > Also, DescriptiveStatistics is not thread-safe (nor is StatCalculator).
> >
> > If we do implement something like this, I think the data processing needs
> > either to be carefully synchronised, or the raw data should be sent to a
> > separate singleton background thread.
>
> FWIW: I always get a bit nervous when percentiles are calculated.
> Percentiles are expensive to calculate if one needs exact results with
> given percentage numbers (50%, 99%, 99.9% etc.). In that case one needs
> to keep all values as an ordered list to calculate the percentiles. For
> a long running test that would be expensive in terms of memory but also
> in terms of CPU (sorting). There's no way of exactly merging percentiles
> from interim statistical data.
>
> Sometimes approximations are enough. By approximation I don't mean
> estimated data, but percentages which are not exactly the ones you are
> keen for. E.g. you would get a 48% value instead of a 50% value, or a
> 99.02% value instead of a 99% value.
>
> Suppose you would know (configure) that only very few samples will take
> longer than 1000ms, then one could create fixed bins for e.g. 10ms,
> 15ms, 20ms, 25ms, 30ms, 40ms, 50ms, 75ms, 100ms, 150ms, 200ms, 250ms,
> 300ms, 400ms, 500ms, 750ms and 1000ms. Now whenever a sample finishes
> you count the sample in the bin it belongs to and do not save the data
> (of course you can still log it). At any time you can now look at the
> bin counters and cheaply produce quantiles. For example suppose the bin
> counters look like that:
>
> Duration binCount
> 10ms 3
> 15ms 2
> 20ms 5
> 25ms 10
> 30ms 8
> 40ms 20
> 50ms 28
> 75ms 100
> 100ms 230
> 150ms 610
> 200ms 780
> 250ms 530
> 300ms 220
> 400ms 200
> 500ms 80
> 750ms 90
> 1000ms 50
> >1000ms 30
>
> Now we sum up the bin count into cumulated count and take percentages:
>
> Duration binCount Cum. Cum.Pct binPct
> 10ms 3 3 0.10% 0.10%
> 15ms 2 5 0.17% 0.07%
> 20ms 5 10 0.33% 0.17%
> 25ms 10 20 0.67% 0.33%
> 30ms 8 28 0.93% 0.27%
> 40ms 20 48 1.60% 0.67%
> 50ms 28 76 2.54% 0.93%
> 75ms 100 176 5.87% 3.34%
> 100ms 230 406 13.55% 7.68%
> 150ms 610 1016 33.91% 20.36%
> 200ms 780 1796 59.95% 26.03%
> 250ms 530 2326 77.64% 17.69%
> 300ms 220 2546 84.98% 7.34%
> 400ms 200 2746 91.66% 6.68%
> 500ms 80 2826 94.33% 2.67%
> 750ms 90 2916 97.33% 3.00%
> 1000ms 50 2966 99.00% 1.67%
> >1000ms 30 2996 100.00% 1.00%
>
> The Cum.Pct column could be used instead of the percentiles. One does
> not need to keep all sample values around and sort them, but one does
> also not get equidistant percentiles (10%, 11%, 12%, ...).
>
> Making the table more fine grained by using more rows is cheap. I have
> chosen here a short table to make the example easier to understand. The
> duration limits for the bins I had chosen were human friendly (integral
> numbers, often divisible by 10 or 100), but one could also use a
> mathematically strict series like e.g. 100 logarithmic steps or 10
> logarithmic steps for each factor 10 in duration time. The bin limits
> would then increase by 26% from bin to bin, e.g. 100ms, 126ms, 158ms,
> 200ms, 251ms, 316ms, 398ms, 501ms, 631ms, 794ms, 1000ms.
>
> Users who need integral percentiles, like exactly 50% or 99% values
> instead of 48.2% or 99.05% values (example) would need to either post
> process the sample logs or choose a more expensive live processing (in
> terms of memory and cpu be it inside JMeter or another backend). I'd
> expect, that often the non-integral percentiles above are good enough to
> follow a running test and get sufficient data about its behavior (and
> are cheap enough to produce on the fly) and the integral percentiles are
> OK to generate during post-processing and full report generation after a
> run.
>
> Just my 2c.
>
> Regards,
>
> Rainer
>
--
Cordialement.
Philippe Mouawad.
Re: [Bug 55932] Create a Graphite Listener
Posted by Rainer Jung <ra...@kippdata.de>.
On 03.01.2014 13:57, bugzilla@apache.org wrote:
> https://issues.apache.org/bugzilla/show_bug.cgi?id=55932
>
> --- Comment #6 from Sebb <se...@apache.org> ---
> I have been having a look at the implementation.
>
> I don't really see that it needs Commons Math; we aleady have StatCalculator
> which handles percentiles and more.
>
> Likewise, does it really need Commons Pool?
> It seems wrong to have to have 2 separate pools of SocketOutputStream
> instances.
> How many of these would there be?
>
> Also, DescriptiveStatistics is not thread-safe (nor is StatCalculator).
>
> If we do implement something like this, I think the data processing needs
> either to be carefully synchronised, or the raw data should be sent to a
> separate singleton background thread.
FWIW: I always get a bit nervous when percentiles are calculated.
Percentiles are expensive to calculate if one needs exact results with
given percentage numbers (50%, 99%, 99.9% etc.). In that case one needs
to keep all values as an ordered list to calculate the percentiles. For
a long running test that would be expensive in terms of memory but also
in terms of CPU (sorting). There's no way of exactly merging percentiles
from interim statistical data.
Sometimes approximations are enough. By approximation I don't mean
estimated data, but percentages which are not exactly the ones you are
keen for. E.g. you would get a 48% value instead of a 50% value, or a
99.02% value instead of a 99% value.
Suppose you would know (configure) that only very few samples will take
longer than 1000ms, then one could create fixed bins for e.g. 10ms,
15ms, 20ms, 25ms, 30ms, 40ms, 50ms, 75ms, 100ms, 150ms, 200ms, 250ms,
300ms, 400ms, 500ms, 750ms and 1000ms. Now whenever a sample finishes
you count the sample in the bin it belongs to and do not save the data
(of course you can still log it). At any time you can now look at the
bin counters and cheaply produce quantiles. For example suppose the bin
counters look like that:
Duration binCount
10ms 3
15ms 2
20ms 5
25ms 10
30ms 8
40ms 20
50ms 28
75ms 100
100ms 230
150ms 610
200ms 780
250ms 530
300ms 220
400ms 200
500ms 80
750ms 90
1000ms 50
>1000ms 30
Now we sum up the bin count into cumulated count and take percentages:
Duration binCount Cum. Cum.Pct binPct
10ms 3 3 0.10% 0.10%
15ms 2 5 0.17% 0.07%
20ms 5 10 0.33% 0.17%
25ms 10 20 0.67% 0.33%
30ms 8 28 0.93% 0.27%
40ms 20 48 1.60% 0.67%
50ms 28 76 2.54% 0.93%
75ms 100 176 5.87% 3.34%
100ms 230 406 13.55% 7.68%
150ms 610 1016 33.91% 20.36%
200ms 780 1796 59.95% 26.03%
250ms 530 2326 77.64% 17.69%
300ms 220 2546 84.98% 7.34%
400ms 200 2746 91.66% 6.68%
500ms 80 2826 94.33% 2.67%
750ms 90 2916 97.33% 3.00%
1000ms 50 2966 99.00% 1.67%
>1000ms 30 2996 100.00% 1.00%
The Cum.Pct column could be used instead of the percentiles. One does
not need to keep all sample values around and sort them, but one does
also not get equidistant percentiles (10%, 11%, 12%, ...).
Making the table more fine grained by using more rows is cheap. I have
chosen here a short table to make the example easier to understand. The
duration limits for the bins I had chosen were human friendly (integral
numbers, often divisible by 10 or 100), but one could also use a
mathematically strict series like e.g. 100 logarithmic steps or 10
logarithmic steps for each factor 10 in duration time. The bin limits
would then increase by 26% from bin to bin, e.g. 100ms, 126ms, 158ms,
200ms, 251ms, 316ms, 398ms, 501ms, 631ms, 794ms, 1000ms.
Users who need integral percentiles, like exactly 50% or 99% values
instead of 48.2% or 99.05% values (example) would need to either post
process the sample logs or choose a more expensive live processing (in
terms of memory and cpu be it inside JMeter or another backend). I'd
expect, that often the non-integral percentiles above are good enough to
follow a running test and get sufficient data about its behavior (and
are cheap enough to produce on the fly) and the integral percentiles are
OK to generate during post-processing and full report generation after a
run.
Just my 2c.
Regards,
Rainer