Throughput timer that generates Poisson process

[Vladimir Sitnikov <si...@gmail.com>]

Hi,

Some time ago I did suggest to implement throughput timer that generates
Poisson process.

JMeter thread reference:
http://mail-archives.apache.org/mod_mbox/jmeter-dev/201502.mbox/%3CCAB=Je-GON1QUxAD6nnzMLA0wh9JMpk8nh=y8UyLiHFg8MAOv3g@mail.gmail.com%3E


JMeter plugins thread reference:
https://groups.google.com/d/msg/jmeter-plugins/K8I6LIRwYjM/kWteXor28kQJ

At that time the suggestion was declined and the timer was implemented
in-house.

There was interest in merging that kind of timer to JMeter core, so I did
check if I could contribute it.
The result is positive, so it's time to negotiate the procedure.

To make long story short: the timer enables to generate Poisson arrivals
with constant throughput in a range of 0..3600 requests/hour (I'm not sure
what is the upper bound exactly, however it does exist)
The typical use case is "generate 100 orders per hour". For example, if
configured with 100/hour, the timer ensures that there are exactly 100
transactions each hour while the transactions are still launched at random
timestamps.

So I've two generic questions:
1) Go/no-go for inclusion? Please, some +1/-1 here.

2) How the timer should be named?
We used to name it as "Exponential Timer", however I do not find that name
to be understandable by regular end-user.
"Exponential Timer" was used just because "Poisson process has exponential
distribution of inter-arrival delays". So "exponential" in the name is 100%
scientifically correct, yet it is 100% protected from being understood by a
regular end-user. On the good side, it will produce good "Google results"
as the name is somewhat unique.

Theoretically, we could even drop "Poisson timer", and rework current
"Constant throughput timer" to use new algorithm, however that might be a
little bit too invasive.


I'm open to further discussion/questions/suggestions.

Technically speaking, the timer always coordinates all the threads in the
group and it creates "launch schedule" even before the first thread
arrives. Then the timer releases threads at pre-scheduled time by returning
proper "delay" value. The timer does not care when the thread was created.
It only cares on the exact timestamp the subsequent sample should take
place.


Vladimir

Re: Throughput timer that generates Poisson process

[Vladimir Sitnikov <si...@gmail.com>]

Just in case, I've pushed the code as a WIP PR:
https://github.com/apache/jmeter/pull/231

The missing part is documentation and component name.
I'm still sure "exponential timer" is close to useless name for a
day-to-day engineer.

Vladimir

Re: Throughput timer that generates Poisson process

[Antonio Gomes Rodrigues <ra...@gmail.com>]

Ok

I have always use "synchronizing timer"/rendez vous but your solution seem
interesting

Antonio

2016-09-21 14:25 GMT+02:00 Vladimir Sitnikov <si...@gmail.com>:

> Antonio>What do you mean by ""Bursty load" simulation. There is no easy way
> to test
> Antonio>"50 iterations per hour as 10 bursts of 5 items"?
>
> There are two cases there:
> 1) The main problem is how would you create a load when two actions are
> very close "by pure chance"?
> For instance, current CTT will eventually create very even distribution
> with all pauses equal (assuming
> the response times are stable).
> By default, if you launch "360 per hour", you might expect "one sample
> every 10 seconds": 0, 10, 20, 30, ...
>
> That would miss cases when "two samples took place at very similar point in
> time" like 0, 11, 12, 30, ..
>
> "Poission process" models that appropriately. Even though it creates
> "constant throughput", the particular
> arrival times are random, so multiple events can appear close to each
> other.
>
> 2) "Always start 5 samples at once". Something like that can be achieved by
> current "synchronizing timer",
> however there are cases when you need to "test rare cases that always come
> in pairs"
>
> Vladimir
>

Re: Throughput timer that generates Poisson process

[Vladimir Sitnikov <si...@gmail.com>]

Antonio>What do you mean by ""Bursty load" simulation. There is no easy way
to test
Antonio>"50 iterations per hour as 10 bursts of 5 items"?

There are two cases there:
1) The main problem is how would you create a load when two actions are
very close "by pure chance"?
For instance, current CTT will eventually create very even distribution
with all pauses equal (assuming
the response times are stable).
By default, if you launch "360 per hour", you might expect "one sample
every 10 seconds": 0, 10, 20, 30, ...

That would miss cases when "two samples took place at very similar point in
time" like 0, 11, 12, 30, ..

"Poission process" models that appropriately. Even though it creates
"constant throughput", the particular
arrival times are random, so multiple events can appear close to each other.

2) "Always start 5 samples at once". Something like that can be achieved by
current "synchronizing timer",
however there are cases when you need to "test rare cases that always come
in pairs"

Vladimir

Re: Throughput timer that generates Poisson process

[Antonio Gomes Rodrigues <ra...@gmail.com>]

2016-09-20 17:51 GMT+02:00 Vladimir Sitnikov <si...@gmail.com>:

> >How to find this limit?
> >3600 requests/hour is not very high :-(
>
> Need to run a dumb test (e.g. the one that is just executing nothing).
>
Cool


>
> Well, the timer does keep the schedule in memory, and no work was made to
> make that scalable over
> hight number of threads, etc, etc.
>
> >If the limit of requests/hour is not too low, I think it's the best
> >solution or have a checkbox to switch between the two algorithm
>
> I think it might make sense to implement additional algorithm like "if
> throughput exceeds certain limit, just use simpler algorithm".
> Ideally that flip should not require user-visible checkbox, so we can
> improve those in future.
>
The limit depend only of the algorithm?
Or it depends of the hardware too?




>
> >Like "Constant throughput timer" but more accurate?
>
> Yes. More features are listed in the links in the initial email.
>
I have checked it

What do you mean by ""Bursty load" simulation. There is no easy way to test
"50 iterations per hour as 10 bursts of 5 items"?

Antonio


>
> Vladimir
>

Re: Throughput timer that generates Poisson process

[Vladimir Sitnikov <si...@gmail.com>]

>How to find this limit?
>3600 requests/hour is not very high :-(

Need to run a dumb test (e.g. the one that is just executing nothing).

Well, the timer does keep the schedule in memory, and no work was made to
make that scalable over
hight number of threads, etc, etc.

>If the limit of requests/hour is not too low, I think it's the best
>solution or have a checkbox to switch between the two algorithm

I think it might make sense to implement additional algorithm like "if
throughput exceeds certain limit, just use simpler algorithm".
Ideally that flip should not require user-visible checkbox, so we can
improve those in future.

>Like "Constant throughput timer" but more accurate?

Yes. More features are listed in the links in the initial email.

Vladimir

Re: Throughput timer that generates Poisson process

[Antonio Gomes Rodrigues <ra...@gmail.com>]

Hi Vladimir,

2016-09-20 16:36 GMT+02:00 Vladimir Sitnikov <si...@gmail.com>:

> Hi,
>
> Some time ago I did suggest to implement throughput timer that generates
> Poisson process.
>
> JMeter thread reference:
> http://mail-archives.apache.org/mod_mbox/jmeter-dev/201502.mbox/%3CCAB=Je-
> GON1QUxAD6nnzMLA0wh9JMpk8nh=y8UyLiHFg8MAOv3g@mail.gmail.com%3E
>
>
> JMeter plugins thread reference:
> https://groups.google.com/d/msg/jmeter-plugins/K8I6LIRwYjM/kWteXor28kQJ
>
> At that time the suggestion was declined and the timer was implemented
> in-house.
>
> There was interest in merging that kind of timer to JMeter core, so I did
> check if I could contribute it.
> The result is positive, so it's time to negotiate the procedure.
>
> To make long story short: the timer enables to generate Poisson arrivals
> with constant throughput in a range of 0..3600 requests/hour (I'm not sure
> what is the upper bound exactly, however it does exist)
>
How to find this limit?
3600 requests/hour is not very high :-(



> The typical use case is "generate 100 orders per hour". For example, if
> configured with 100/hour, the timer ensures that there are exactly 100
> transactions each hour while the transactions are still launched at random
> timestamps.
>
Like "Constant throughput timer" but more accurate?


>
> So I've two generic questions:
> 1) Go/no-go for inclusion? Please, some +1/-1 here.
>
It will depend on the upper bound

I think it will be great to have a more accurate "Constant throughput
timer" in JMeter but if there is too much limitation, I think it will be
better to integrate it in another way

e.g.
In "Constant throughput timer", have a check box "accurate but limited to
XXX requests/hour" which switch from classic algorithm to your algorithm


>
> 2) How the timer should be named?
> We used to name it as "Exponential Timer", however I do not find that name
> to be understandable by regular end-user.
> "Exponential Timer" was used just because "Poisson process has exponential
> distribution of inter-arrival delays". So "exponential" in the name is 100%
> scientifically correct, yet it is 100% protected from being understood by a
> regular end-user. On the good side, it will produce good "Google results"
> as the name is somewhat unique.
>
> Theoretically, we could even drop "Poisson timer", and rework current
> "Constant throughput timer" to use new algorithm, however that might be a
> little bit too invasive.
>

If the limit of requests/hour is not too low, I think it's the best
solution or have a checkbox to switch between the two algorithm


>
> I'm open to further discussion/questions/suggestions.
>
> Technically speaking, the timer always coordinates all the threads in the
> group and it creates "launch schedule" even before the first thread
> arrives. Then the timer releases threads at pre-scheduled time by returning
> proper "delay" value. The timer does not care when the thread was created.
> It only cares on the exact timestamp the subsequent sample should take
> place.
>
>
> Vladimir
>

Re: Throughput timer that generates Poisson process

[Philippe Mouawad <ph...@gmail.com>]

On Tue, Sep 20, 2016 at 5:41 PM, Epp, Jeremiah W (Contractor) <je...@cas.org>
wrote:

> > -----Original Message-----
> > From: Vladimir Sitnikov [mailto:sitnikov.vladimir@gmail.com]
> > Sent: Tuesday, September 20, 2016 10:37 AM
> > To: Philippe Mouawad
> > Subject: Throughput timer that generates Poisson process
> >
> > with constant throughput in a range of 0..3600 requests/hour (I'm not
> sure
> > what is the upper bound exactly, however it does exist)
>
> Why is there a limit?  Can it be removed?
>
> Also, if getting it merged into JMeter core is a problem, would you be
> amenable to releasing it as a plugin instead?
>
For now, Merging is not a problem.
We're just in the process of reviewing (once code is available) then
discussing and finally accepting or rejecting as per Apache process.


> As for name, maybe something like "Poisson Arrivals Timer"?
>
> Regards,
> Wyatt
>
> Confidentiality Notice: This electronic message transmission, including
> any attachment(s), may contain confidential, proprietary, or privileged
> information from Chemical Abstracts Service ("CAS"), a division of the
> American Chemical Society ("ACS"). If you have received this transmission
> in error, be advised that any disclosure, copying, distribution, or use of
> the contents of this information is strictly prohibited. Please destroy all
> copies of the message and contact the sender immediately by either replying
> to this message or calling 614-447-3600.
>
>


-- 
Cordialement.
Philippe Mouawad.

RE: Throughput timer that generates Poisson process

["Epp, Jeremiah W (Contractor)" <je...@cas.org>]

> -----Original Message-----
> From: Vladimir Sitnikov [mailto:sitnikov.vladimir@gmail.com]
> Sent: Tuesday, September 20, 2016 10:37 AM
> To: Philippe Mouawad
> Subject: Throughput timer that generates Poisson process
> 
> with constant throughput in a range of 0..3600 requests/hour (I'm not sure
> what is the upper bound exactly, however it does exist)

Why is there a limit?  Can it be removed?

Also, if getting it merged into JMeter core is a problem, would you be
amenable to releasing it as a plugin instead?

As for name, maybe something like "Poisson Arrivals Timer"?

Regards,
Wyatt

Confidentiality Notice: This electronic message transmission, including any attachment(s), may contain confidential, proprietary, or privileged information from Chemical Abstracts Service ("CAS"), a division of the American Chemical Society ("ACS"). If you have received this transmission in error, be advised that any disclosure, copying, distribution, or use of the contents of this information is strictly prohibited. Please destroy all copies of the message and contact the sender immediately by either replying to this message or calling 614-447-3600.

Re: Throughput timer that generates Poisson process

[Vladimir Sitnikov <si...@gmail.com>]

Philippe>Is it possible to see the code before voting ?
Philippe>+1 for the theorical proposal but I would need to see the
implementation.

Of course I'll file a PR, however:
1) I need some time to ensure it does not disclose private information
2) I would like to discuss which features are needed before the patch gets
merged in.
At the end of the day, it might result in having "two constant throughput
timers" and that is no good from the UX perspective.

Vladimir

Re: Throughput timer that generates Poisson process

[Philippe Mouawad <ph...@gmail.com>]

Hello Vladimir,


On Tue, Sep 20, 2016 at 4:36 PM, Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:

> Hi,
>
> Some time ago I did suggest to implement throughput timer that generates
> Poisson process.
>
> JMeter thread reference:
> http://mail-archives.apache.org/mod_mbox/jmeter-dev/201502.mbox/%3CCAB=Je-
> GON1QUxAD6nnzMLA0wh9JMpk8nh=y8UyLiHFg8MAOv3g@mail.gmail.com%3E
>
>
> JMeter plugins thread reference:
> https://groups.google.com/d/msg/jmeter-plugins/K8I6LIRwYjM/kWteXor28kQJ
>
> At that time the suggestion was declined and the timer was implemented
> in-house.
>
> There was interest in merging that kind of timer to JMeter core, so I did
> check if I could contribute it.
> The result is positive, so it's time to negotiate the procedure.
>
Great news.


>
> To make long story short: the timer enables to generate Poisson arrivals
> with constant throughput in a range of 0..3600 requests/hour (I'm not sure
> what is the upper bound exactly, however it does exist)
> The typical use case is "generate 100 orders per hour". For example, if
> configured with 100/hour, the timer ensures that there are exactly 100
> transactions each hour while the transactions are still launched at random
> timestamps.
>
> So I've two generic questions:
> 1) Go/no-go for inclusion? Please, some +1/-1 here.
>
Is it possible to see the code before voting ?
+1 for the theorical proposal but I would need to see the implementation.


>
> 2) How the timer should be named?
> We used to name it as "Exponential Timer", however I do not find that name
> to be understandable by regular end-user.
> "Exponential Timer" was used just because "Poisson process has exponential
> distribution of inter-arrival delays". So "exponential" in the name is 100%
> scientifically correct, yet it is 100% protected from being understood by a
> regular end-user. On the good side, it will produce good "Google results"
> as the name is somewhat unique.
>
> Theoretically, we could even drop "Poisson timer", and rework current
> "Constant throughput timer" to use new algorithm, however that might be a
> little bit too invasive.
>
>
> I'm open to further discussion/questions/suggestions.
>
> Technically speaking, the timer always coordinates all the threads in the
> group and it creates "launch schedule" even before the first thread
> arrives. Then the timer releases threads at pre-scheduled time by returning
> proper "delay" value. The timer does not care when the thread was created.
> It only cares on the exact timestamp the subsequent sample should take
> place.
>
>
> Vladimir
>



-- 
Cordialement.
Philippe Mouawad.