Throughput timers vs Poisson vs etc

[Vladimir Sitnikov <si...@gmail.com>]

Hi,

When it comes to load simulation, JMeter is hard to bend around the
following requirements:
1) "Specific number of iterations per hour".
   The very basic requirement is to ensure you end up exactly 50
iterations per hour.
   Business customers would not understand if you report load test
results with 47 executions "just because the random was random".

   All timers except "Constant" can easily send excessive requests or
send less requests than required. There is no way to control it.

2) "Bursty load" simulation. There is no easy way to test "50
iterations per hour as 10 bursts of 5 items".

3) "Repeatable test profile". All the random timers produce different
pattern on each test run. This is not good for low-level analysis
(e.g. compare of CPU% charts, etc).

4) "Avoid all thread groups" to fire at 00:00:00. By default, all
thread groups would fire at 0, so there would be noticeable spike at
the start of the test.

It might look like PoissionRandomTimer and SynchronizingTimer solve
the problem, however no one uses that in real life.
"Poission Random Timer" -- 711 results in Google
"Synchronizing Timer" -- 3590 results in Google

Anyone else struggling with those issues?

-- 
Regards,
Vladimir Sitnikov

Re: Throughput timers vs Poisson vs etc

[Vladimir Sitnikov <si...@gmail.com>]

> I suggest you open bugs with if possible jmx plan showing issue.

Unfortunately, I do not use CTT, neither do my neighbor perf engineers due
to the issues listed in the thread.
I am not sure if you'd welcome deprecation of CTT (I would love to create
this issue:) in favour of exponential arrivals:
http://perfdynamics.blogspot.ru/2012/05/load-testing-with-uniform-vs.html

> Did you try this one from 3rd party:
> http://jmeter-plugins.org/wiki/ThroughputShapingTimer/

It does not solve the problems above. Well, functional issues might be
different, but it does nothing for testing bursty load, load spike at
0:0:0, etc.
I was told my request was "too uncommon to be a part of plugins set":
https://groups.google.com/d/msg/jmeter-plugins/K8I6LIRwYjM/WgDhQXghzDMJ

Vladimir

Re: Throughput timers vs Poisson vs etc

[Philippe Mouawad <ph...@gmail.com>]

On Fri, Feb 6, 2015 at 9:32 PM, Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:

> > Did you try Constant Throughput Timer
>
> I did, however it does have drawbacks.
> 1.1) It does not always work. Sorry, it was long ago I've last tested
> it, so I cannot list lots of examples from the top of my head.
> For instance, I've just created 100 threads, set thread duration to be
> 20 seconds and put constant throughput timer of 60 per minute.
> I expect exactly 20 items of "debug sampler" to appear, but 100 happens.
> Test plan:
> ThreadGroup(100 threads, 20 seconds test duration)
>   Loop(Forever)
>     DebugSampler(for_throughput)
>       ConstantThroughputTimer(60 per minute)
>     DebugSampler(work)
>

I suggest you open bugs with if possible jmx plan showing issue.


>
> 1.2) It tries to adapt to the target based on the duration of the loop
> execution.
> It works really awful for long iterations. For instance, if a single
> iteration takes 10 minutes and I want 10 per hour, then CTT has just
> no time to "adapt".
> If an iteration takes longer than CTT expected, then you're screwed.
> You get less executions than required.
>
> 1.3) CTT is not reproducible. See point 3 of original email on
> "Repeatable test profile".
>
> 1.4) CTT tries to do even delay between iterations. This plays awful
> when you have multiple scenarios running at once if the scenarios have
> the same target throughput. You end up with sin(t) kind of CPU%.
>
> 5) "Bursty load without explicit batches". If I set CTT at 1 per
> minute, it will never launch iterations closer than 60 seconds
> (assuming all the bugs like 1.1 are fixed). However, in reality
> arrivals often have spikes. For instance, in poisson process (those
> kind of arrivals are common to happen in telecom industry)
> inter-arrival durations vary (bingo!).
> In other words, if an action takes 30 seconds to execute and CTT
> launches it every 60 seconds, then this kind of testing will NEVER
> catch a case when two actions run concurrently. With inter-arrival
> delays like 10 sec, 110 sec, 80 sec, 40 sec throughput would still
> result in "1 per minute" on average, but it will catch more
> performance bugs (e.g. the one when two concurrent actions result in
> out of memory).
>
> > Yes you can, just schedule them :
> > Scheduler > Startup Delay + Duration
>
> It does not scale: suppose you have 20 thread groups (e.g. 20 UI
> scenarios).
> How would you manage all those durations?
> How would you change the duration if throughput needs to be changed?
> (hint: in scalability tests throughput is often need to be altered for
> each run)
> How would you pass the script to your buddy?
>

Okay.
Did you try this one from 3rd party:
http://jmeter-plugins.org/wiki/ThroughputShapingTimer/

>
> Even if you could configure it once, those 0:0:0 defaults will
> definitely bite you very soon (as a new group is added).
>
> Vladimir
>

Thanks for this feedback,  it will help us improve.
Regards

Re: Throughput timers vs Poisson vs etc

[Vladimir Sitnikov <si...@gmail.com>]

> Did you try Constant Throughput Timer

I did, however it does have drawbacks.
1.1) It does not always work. Sorry, it was long ago I've last tested
it, so I cannot list lots of examples from the top of my head.
For instance, I've just created 100 threads, set thread duration to be
20 seconds and put constant throughput timer of 60 per minute.
I expect exactly 20 items of "debug sampler" to appear, but 100 happens.
Test plan:
ThreadGroup(100 threads, 20 seconds test duration)
  Loop(Forever)
    DebugSampler(for_throughput)
      ConstantThroughputTimer(60 per minute)
    DebugSampler(work)

1.2) It tries to adapt to the target based on the duration of the loop
execution.
It works really awful for long iterations. For instance, if a single
iteration takes 10 minutes and I want 10 per hour, then CTT has just
no time to "adapt".
If an iteration takes longer than CTT expected, then you're screwed.
You get less executions than required.

1.3) CTT is not reproducible. See point 3 of original email on
"Repeatable test profile".

1.4) CTT tries to do even delay between iterations. This plays awful
when you have multiple scenarios running at once if the scenarios have
the same target throughput. You end up with sin(t) kind of CPU%.

5) "Bursty load without explicit batches". If I set CTT at 1 per
minute, it will never launch iterations closer than 60 seconds
(assuming all the bugs like 1.1 are fixed). However, in reality
arrivals often have spikes. For instance, in poisson process (those
kind of arrivals are common to happen in telecom industry)
inter-arrival durations vary (bingo!).
In other words, if an action takes 30 seconds to execute and CTT
launches it every 60 seconds, then this kind of testing will NEVER
catch a case when two actions run concurrently. With inter-arrival
delays like 10 sec, 110 sec, 80 sec, 40 sec throughput would still
result in "1 per minute" on average, but it will catch more
performance bugs (e.g. the one when two concurrent actions result in
out of memory).

> Yes you can, just schedule them :
> Scheduler > Startup Delay + Duration

It does not scale: suppose you have 20 thread groups (e.g. 20 UI scenarios).
How would you manage all those durations?
How would you change the duration if throughput needs to be changed?
(hint: in scalability tests throughput is often need to be altered for
each run)
How would you pass the script to your buddy?

Even if you could configure it once, those 0:0:0 defaults will
definitely bite you very soon (as a new group is added).

Vladimir

Re: Throughput timers vs Poisson vs etc

[Philippe Mouawad <ph...@gmail.com>]

Hi Vladimir,
Partial Answers below.
Regards

On Fri, Feb 6, 2015 at 2:20 PM, Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:

> Hi,
>
> When it comes to load simulation, JMeter is hard to bend around the
> following requirements:
> 1) "Specific number of iterations per hour".
>    The very basic requirement is to ensure you end up exactly 50
> iterations per hour.
>    Business customers would not understand if you report load test
> results with 47 executions "just because the random was random".
>
>    All timers except "Constant" can easily send excessive requests or
> send less requests than required. There is no way to control it.
>
> Did you try Constant Throughput Timer ?
http://jmeter.apache.org/usermanual/component_reference.html#Constant_Throughput_Timer


> 2) "Bursty load" simulation. There is no easy way to test "50
> iterations per hour as 10 bursts of 5 items".
>

Will think about it

>
> 3) "Repeatable test profile". All the random timers produce different
> pattern on each test run. This is not good for low-level analysis
> (e.g. compare of CPU% charts, etc).
>

Will think about it

>
> 4) "Avoid all thread groups" to fire at 00:00:00. By default, all
> thread groups would fire at 0, so there would be noticeable spike at
> the start of the test.
>

Yes you can, just schedule them :
Scheduler > Startup Delay + Duration

>
> It might look like PoissionRandomTimer and SynchronizingTimer solve
> the problem, however no one uses that in real life.
> "Poission Random Timer" -- 711 results in Google
> "Synchronizing Timer" -- 3590 results in Google
>
> Anyone else struggling with those issues?
>
> --
> Regards,
> Vladimir Sitnikov
>



-- 
Cordialement.
Philippe Mouawad.