You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jmeter.apache.org by Milamber <mi...@apache.org> on 2015/09/01 18:23:58 UTC
Re: Design of Timeout test element / sampler interrupter
Hello,
I make some tests.
* On basic scenario, with 2 http requests on the same url, Callable
option seems most quick compare to Runnable (~70% more quick)
* Basic tests with the same url + Assertion Duration : the behavior is ok
* Basic tests with the same url + a Constant Timer set to 2 sec (and
Interrupt Timer set to 2 sec), the results seems not ok: the url is
fully retrieve with a response time > 5 seconds (the request isn't
interrupt, the response tab show the HTML code of the page). If I
disable the Constant Timer, the request is interrupt in 2 sec. (the
response tab show a (normal) exception causing by the interruption)
Probably an issue...
* The fields Task type and Call delay isn't localized and the values
aren't reset with your add a second Interrupt timer (I suppose you knows
that, current branch is a PoC)
Milamber
[the URL for test (deflate is disable in HTTP Header):
https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.2/html-single/Administration_and_Configuration_Guide/index.html
]
On 30/08/2015 17:37, sebb wrote:
> OK, the SampleTimeout branch now has a version that use Runnable by
> default, but you can use Callable (type==CALL) with an optional delay
> to simulate an interrupt that takes a long time.
>
> It works fine for me in simple tests.
>
> If you can break it, so much the better.
>
>
> On 30 August 2015 at 16:36, sebb <se...@gmail.com> wrote:
>> On 30 August 2015 at 11:54, sebb <se...@gmail.com> wrote:
>>> On 30 August 2015 at 07:12, Philippe Mouawad <ph...@gmail.com> wrote:
>>>> On Sunday, August 30, 2015, sebb <se...@gmail.com> wrote:
>>>>
>>>>> I've had a look at the classes that implement SampleListener, and
>>>>> apart from ResultAction and TransactionSampler, only the Listeners use
>>>>> it. Since usage of these should be minimised in a production test,
>>>>> it's likely that there won't be as many implementations as I had
>>>>> feard.
>>>> Implementation of SampleListener ?
>>>> Usage of which should be minimised ? SampleListener ?
>>> Usage of additional Listeners should be minimised in a production test.
>>>
>>>>> Also if the implementation is empty, the overhead will be quite small.
>>>>>
>>>>> [There is a work-round if it does prove expensive: the SampleListener
>>>>> interface could be split into two parent interfaces.]
>>>>>
>>>>> So assuming that JMeterThread implements sampleStarted/sampleStopped,
>>>>> the Timeout element can use the Start to set up the timer and the Stop
>>>>> to cancel it. This will reduce the number outstanding as much as
>>>>> possible.
>>>>>
>>>>> The timeouts have to be implemented using separate threads for two reasons:
>>>>> - it's obviously not possible to interrupt a sampler from the same
>>>>> thread as the sampler
>>>>> - depending on the sampler, and its state, the interrupt may take a
>>>>> while to complete, so each interrupt must be done in its own thread
>>>> Are you sure, calling interrupt is usually just about setting a flag no?
>>> Ultimately yes, I guess a flag will be set.
>>> However I'm not sure that it is always instantaneous, as there may be
>>> locks to aquire.
>>>
>>>> Having 1 thread for each interruption, could lead to hundred of
>>>> threads running for high throughput threads (500 res/s for example), it
>>>> won't scale.
>>> That assumes that all the samples in all the threads have timeouts enabled.
>> It also assumes that the threads are running for the life of the
>> timeout, which I've just realised is not the case.
>>>> Why can't we have 1 Thread (TimeoutChecker) called every N milliseconds
>>>> that checks all registered JMeterThreads to check and call interrupt if
>>>> necessary ?
>>>>
>>> That's quite a bit of work to code, but if there is already such a
>>> queing mechanism it would be worth trying.
>> Actually the ScheduledExecutorService does just that.
>> It uses one thread to handle the timeouts in sequence, and then starts
>> a new Runnable when the timeout expires.
>>
>> Now that I have created the service as a singleton, there will only be
>> one extra thread (the executor) most of the time.
>> Since the timer tasks are cancelled if the sample completes in time,
>> there will only ever be at most one extra thread for each overdue
>> sample.
>>
>> I think it is a reasonable assumption to assume that the number of
>> such samples will generally be small.
>> At the very most it could only be one per thread.
>> And remember that the new thread is only started when the timeout expires.
>> It is extremely unlikely that they will all expire at once, and anyway
>> (according to you) they don't take long to run so they won't build up.
>>
>> I've just noticed that there is a version of the Executor Service that
>> uses Callable rather than Runnable (not sure why I missed it before).
>> I asume this means the tasks are run in the same thread (I'll check this).
>> If the interrupts really do happen quickly, this might be a better choice.
>> Even if not, then it may not matter so long as any delays don't
>> continue to build up.
>>
>>>>> It should be possible to use a single shared instance of the
>>>>> ScheduledExecutorService; that could be lazily created using IODH. [I
>>>>> can try that with the current implementation]
>>> I've already tried it.
>>>
>>>> See my note above
>>>>> As to whether the Timeout class should be a Timer or some other type
>>>>> of test element - that does not matter so long as it can be applied to
>>>>> the samplers individually or when in scope.
>>>>>
>>>>> I chose Timer because it was already called in the right place, but I
>>>>> assume JMeterThread can call any Test class provided that it
>>>>> implemented the SampleInterface.
>>>>>
>>>>> It must be one of the existing Test Element classes that are handled
>>>>> by the Menu system otherwise it will need special handling.
>>>>>
>>>>> The scope requirement rules out Config elements and Logic Controllers.
>>>> It does not seem like a pre-processor to me, nor a post-procesor, nor a
>>>>> Listener
>>>>>
>>>>> So AFAICT the only remaining options are the Timer and Assertions.
>>>>>
>>>>> I think both are justifiable.
>>>> Why isn't it part of Sampler abstract class and as such a field in Sampler?
>>> How does the user indicate that a Timeout should be applied to a
>>> particular Sampler?
>>> It's a lot of work to add Timeout fields to every GUI.
>>> Whereas being able to add a child test element to each applicable
>>> sampler is already supported.
>>> Further, such a test element can be applied to multiple samplers in scope.
>>> Much easier to enable and disable a single element that having to
>>> update each Sampler.
>>>
>>> That's why it needs to be a separate test element.
>>>
>>>>
>>>> For me none of Timer nor Assertions are conceptually valid.
>>>> The behaviour is not a pause(so not Timer for me), it's not an Assertion
>>>> neither as for me an Assertion only checks something.
>>>> Although not fully satisfying it look to me more of a PreProcessor as it
>>>> sets a timeout on the Sampler , it can also be considered as a Post
>>>> Processor.
>>> OK, I could live with it being a Pre-Procesor.
>>> That has the correct scoping rules.
>>>
>>>>
>>>>> The name of the class can of course be changed from InterruptTimer - I
>>>>> think that is probably not the best choice. Maybe something like
>>>>> SamplerTimeout?
>>>>>
>>>> Yes or SamplerTimeouter or SamplerInterrupter
>>>>
>>> Or even SampleTimeout - that's what it does, it applies a timeout to a sample.
>>>
>>>> --
>>>> Cordialement.
>>>> Philippe Mouawad.
Re: Design of Timeout test element / sampler interrupter
Posted by sebb <se...@gmail.com>.
On 1 September 2015 at 23:48, Milamber <mi...@apache.org> wrote:
>
>>> * On basic scenario, with 2 http requests on the same url, Callable
>>> option
>>> seems most quick compare to Runnable (~70% more quick)
>>
>> I'm surprised it makes that much difference.
>>
>> What were you measuring?
>
>
> The URL below (in redhat documentation website)
>
>> How many threads?
>
>
> One VU with 10 loops
>
> The latest version of the branch seems have some differences:
>
OK, I see. Thanks again.
> 2015/09/01 23:41:14 WARN - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took 3.15911E-4
> secs
> 2015/09/01 23:41:16 WARN - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took 7.38335E-4
> secs
> 2015/09/01 23:41:18 WARN - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took 7.26091E-4
> secs
> 2015/09/01 23:41:20 WARN - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took 7.4593E-4
> secs
> 2015/09/01 23:41:22 WARN - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took 0.00181855
> secs
> 2015/09/01 23:41:24 WARN - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took 5.33232E-4
> secs
> 2015/09/01 23:41:26 WARN - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took 4.05846E-4
> secs
> 2015/09/01 23:41:28 WARN - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took 5.50154E-4
> secs
> 2015/09/01 23:41:30 WARN - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took 6.93174E-4
> secs
> 2015/09/01 23:41:32 WARN - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took 6.11455E-4
> secs
> 2015/09/01 23:41:34 WARN - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took 6.1518E-4
> secs
> 2015/09/01 23:41:36 WARN - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took 6.24097E-4
> secs
> 2015/09/01 23:41:38 WARN - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took 7.53767E-4
> secs
> 2015/09/01 23:41:40 WARN - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took 6.84471E-4
> secs
> 2015/09/01 23:41:42 WARN - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took 7.34266E-4
> secs
> 2015/09/01 23:41:44 WARN - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took 5.88123E-4
> secs
> 2015/09/01 23:41:46 WARN - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took 3.34427E-4
> secs
> 2015/09/01 23:41:48 WARN - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took 6.3848E-4
> secs
> 2015/09/01 23:41:50 WARN - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took 7.77866E-4
> secs
> 2015/09/01 23:41:52 WARN - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took 7.13642E-4
> secs
>
> The record times is very close (in nano seconds), perhaps, not important
> (significant).
The Run and Call versions use the same code to interrupt the sampler;
the only difference is how the interrupt task is scheduled.
So this should not affect the amount of time it takes to perform the interrupt.
The recorded times don't seem to show a significant difference - apart
from one outlier value they are all around 300-800 microseconds (I
think I've got the scaling right)
However the scheduling method may affect the overall performance of the test.
Starting a Runnable is likely to be more expensive than invoking a Callable.
Whether this difference is significant or not remains to be seen.
But given that the interrupt does not take a significant amount of
time, it may not be worth worrying about.
The code can use Callable by default.
I've done some work on implementing the test element as a PreProcessor
but this is not quite finished.
>
>
>
>>
>>> * Basic tests with the same url + Assertion Duration : the behavior is ok
>>>
>>> * Basic tests with the same url + a Constant Timer set to 2 sec (and
>>> Interrupt Timer set to 2 sec), the results seems not ok: the url is fully
>>> retrieve with a response time > 5 seconds (the request isn't interrupt,
>>> the
>>> response tab show the HTML code of the page). If I disable the Constant
>>> Timer, the request is interrupt in 2 sec. (the response tab show a
>>> (normal)
>>> exception causing by the interruption)
>>> Probably an issue...
>>
>> No, this is expected because the InterruptTimer is called before the
>> delay is performed.
>>
>> I have not yet implemented the use of sampleStarted/sampleStopped
>> which will allow the test element to be called after any other timers.
>>
>> At present the interrupt test element has no way to allow for other
>> timer delays because it does not know what they will be.
>> These occur after it is called (and before the sample).
>> That is why sampleStarted/sampleStopped are needed.
>> Or some other change is needed to JMeterThread that will allow the
>> interrupt timer to schedule the interrupt at the correct time.
>>
>>> * The fields Task type and Call delay isn't localized and the values
>>> aren't
>>> reset with your add a second Interrupt timer (I suppose you knows that,
>>> current branch is a PoC)
>>
>> Yes, they are for testing purposes only and aren't needed in the final
>> release.
>>
>>> Milamber
>>>
>>> [the URL for test (deflate is disable in HTTP Header):
>>>
>>> https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.2/html-single/Administration_and_Configuration_Guide/index.html
>>> ]
>>>
>>>
>>>
>>>
>>> On 30/08/2015 17:37, sebb wrote:
>>>>
>>>> OK, the SampleTimeout branch now has a version that use Runnable by
>>>> default, but you can use Callable (type==CALL) with an optional delay
>>>> to simulate an interrupt that takes a long time.
>>>>
>>>> It works fine for me in simple tests.
>>>>
>>>> If you can break it, so much the better.
>>>>
>>>>
>>>> On 30 August 2015 at 16:36, sebb <se...@gmail.com> wrote:
>>>>>
>>>>> On 30 August 2015 at 11:54, sebb <se...@gmail.com> wrote:
>>>>>>
>>>>>> On 30 August 2015 at 07:12, Philippe Mouawad
>>>>>> <ph...@gmail.com> wrote:
>>>>>>>
>>>>>>> On Sunday, August 30, 2015, sebb <se...@gmail.com> wrote:
>>>>>>>
>>>>>>>> I've had a look at the classes that implement SampleListener, and
>>>>>>>> apart from ResultAction and TransactionSampler, only the Listeners
>>>>>>>> use
>>>>>>>> it. Since usage of these should be minimised in a production test,
>>>>>>>> it's likely that there won't be as many implementations as I had
>>>>>>>> feard.
>>>>>>>
>>>>>>> Implementation of SampleListener ?
>>>>>>> Usage of which should be minimised ? SampleListener ?
>>>>>>
>>>>>> Usage of additional Listeners should be minimised in a production
>>>>>> test.
>>>>>>
>>>>>>>> Also if the implementation is empty, the overhead will be quite
>>>>>>>> small.
>>>>>>>>
>>>>>>>> [There is a work-round if it does prove expensive: the
>>>>>>>> SampleListener
>>>>>>>> interface could be split into two parent interfaces.]
>>>>>>>>
>>>>>>>> So assuming that JMeterThread implements
>>>>>>>> sampleStarted/sampleStopped,
>>>>>>>> the Timeout element can use the Start to set up the timer and the
>>>>>>>> Stop
>>>>>>>> to cancel it. This will reduce the number outstanding as much as
>>>>>>>> possible.
>>>>>>>>
>>>>>>>> The timeouts have to be implemented using separate threads for two
>>>>>>>> reasons:
>>>>>>>> - it's obviously not possible to interrupt a sampler from the same
>>>>>>>> thread as the sampler
>>>>>>>> - depending on the sampler, and its state, the interrupt may take a
>>>>>>>> while to complete, so each interrupt must be done in its own thread
>>>>>>>
>>>>>>> Are you sure, calling interrupt is usually just about setting a flag
>>>>>>> no?
>>>>>>
>>>>>> Ultimately yes, I guess a flag will be set.
>>>>>> However I'm not sure that it is always instantaneous, as there may be
>>>>>> locks to aquire.
>>>>>>
>>>>>>> Having 1 thread for each interruption, could lead to hundred of
>>>>>>> threads running for high throughput threads (500 res/s for example),
>>>>>>> it
>>>>>>> won't scale.
>>>>>>
>>>>>> That assumes that all the samples in all the threads have timeouts
>>>>>> enabled.
>>>>>
>>>>> It also assumes that the threads are running for the life of the
>>>>> timeout, which I've just realised is not the case.
>>>>>>>
>>>>>>> Why can't we have 1 Thread (TimeoutChecker) called every N
>>>>>>> milliseconds
>>>>>>> that checks all registered JMeterThreads to check and call interrupt
>>>>>>> if
>>>>>>> necessary ?
>>>>>>>
>>>>>> That's quite a bit of work to code, but if there is already such a
>>>>>> queing mechanism it would be worth trying.
>>>>>
>>>>> Actually the ScheduledExecutorService does just that.
>>>>> It uses one thread to handle the timeouts in sequence, and then starts
>>>>> a new Runnable when the timeout expires.
>>>>>
>>>>> Now that I have created the service as a singleton, there will only be
>>>>> one extra thread (the executor) most of the time.
>>>>> Since the timer tasks are cancelled if the sample completes in time,
>>>>> there will only ever be at most one extra thread for each overdue
>>>>> sample.
>>>>>
>>>>> I think it is a reasonable assumption to assume that the number of
>>>>> such samples will generally be small.
>>>>> At the very most it could only be one per thread.
>>>>> And remember that the new thread is only started when the timeout
>>>>> expires.
>>>>> It is extremely unlikely that they will all expire at once, and anyway
>>>>> (according to you) they don't take long to run so they won't build up.
>>>>>
>>>>> I've just noticed that there is a version of the Executor Service that
>>>>> uses Callable rather than Runnable (not sure why I missed it before).
>>>>> I asume this means the tasks are run in the same thread (I'll check
>>>>> this).
>>>>> If the interrupts really do happen quickly, this might be a better
>>>>> choice.
>>>>> Even if not, then it may not matter so long as any delays don't
>>>>> continue to build up.
>>>>>
>>>>>>>> It should be possible to use a single shared instance of the
>>>>>>>> ScheduledExecutorService; that could be lazily created using IODH.
>>>>>>>> [I
>>>>>>>> can try that with the current implementation]
>>>>>>
>>>>>> I've already tried it.
>>>>>>
>>>>>>> See my note above
>>>>>>>>
>>>>>>>> As to whether the Timeout class should be a Timer or some other type
>>>>>>>> of test element - that does not matter so long as it can be applied
>>>>>>>> to
>>>>>>>> the samplers individually or when in scope.
>>>>>>>>
>>>>>>>> I chose Timer because it was already called in the right place, but
>>>>>>>> I
>>>>>>>> assume JMeterThread can call any Test class provided that it
>>>>>>>> implemented the SampleInterface.
>>>>>>>>
>>>>>>>> It must be one of the existing Test Element classes that are handled
>>>>>>>> by the Menu system otherwise it will need special handling.
>>>>>>>>
>>>>>>>> The scope requirement rules out Config elements and Logic
>>>>>>>> Controllers.
>>>>>>>
>>>>>>> It does not seem like a pre-processor to me, nor a post-procesor, nor
>>>>>>> a
>>>>>>>>
>>>>>>>> Listener
>>>>>>>>
>>>>>>>> So AFAICT the only remaining options are the Timer and Assertions.
>>>>>>>>
>>>>>>>> I think both are justifiable.
>>>>>>>
>>>>>>> Why isn't it part of Sampler abstract class and as such a field in
>>>>>>> Sampler?
>>>>>>
>>>>>> How does the user indicate that a Timeout should be applied to a
>>>>>> particular Sampler?
>>>>>> It's a lot of work to add Timeout fields to every GUI.
>>>>>> Whereas being able to add a child test element to each applicable
>>>>>> sampler is already supported.
>>>>>> Further, such a test element can be applied to multiple samplers in
>>>>>> scope.
>>>>>> Much easier to enable and disable a single element that having to
>>>>>> update each Sampler.
>>>>>>
>>>>>> That's why it needs to be a separate test element.
>>>>>>
>>>>>>> For me none of Timer nor Assertions are conceptually valid.
>>>>>>> The behaviour is not a pause(so not Timer for me), it's not an
>>>>>>> Assertion
>>>>>>> neither as for me an Assertion only checks something.
>>>>>>> Although not fully satisfying it look to me more of a PreProcessor as
>>>>>>> it
>>>>>>> sets a timeout on the Sampler , it can also be considered as a Post
>>>>>>> Processor.
>>>>>>
>>>>>> OK, I could live with it being a Pre-Procesor.
>>>>>> That has the correct scoping rules.
>>>>>>
>>>>>>>> The name of the class can of course be changed from InterruptTimer -
>>>>>>>> I
>>>>>>>> think that is probably not the best choice. Maybe something like
>>>>>>>> SamplerTimeout?
>>>>>>>>
>>>>>>> Yes or SamplerTimeouter or SamplerInterrupter
>>>>>>>
>>>>>> Or even SampleTimeout - that's what it does, it applies a timeout to a
>>>>>> sample.
>>>>>>
>>>>>>> --
>>>>>>> Cordialement.
>>>>>>> Philippe Mouawad.
>>>
>>>
>
Re: Design of Timeout test element / sampler interrupter
Posted by Milamber <mi...@apache.org>.
>> * On basic scenario, with 2 http requests on the same url, Callable option
>> seems most quick compare to Runnable (~70% more quick)
> I'm surprised it makes that much difference.
>
> What were you measuring?
The URL below (in redhat documentation website)
> How many threads?
One VU with 10 loops
The latest version of the branch seems have some differences:
2015/09/01 23:41:14 WARN - jmeter.timers.InterruptTimer: Run Done
interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took
3.15911E-4 secs
2015/09/01 23:41:16 WARN - jmeter.timers.InterruptTimer: Call Done
interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took
7.38335E-4 secs
2015/09/01 23:41:18 WARN - jmeter.timers.InterruptTimer: Run Done
interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took
7.26091E-4 secs
2015/09/01 23:41:20 WARN - jmeter.timers.InterruptTimer: Call Done
interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took
7.4593E-4 secs
2015/09/01 23:41:22 WARN - jmeter.timers.InterruptTimer: Run Done
interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took
0.00181855 secs
2015/09/01 23:41:24 WARN - jmeter.timers.InterruptTimer: Call Done
interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took
5.33232E-4 secs
2015/09/01 23:41:26 WARN - jmeter.timers.InterruptTimer: Run Done
interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took
4.05846E-4 secs
2015/09/01 23:41:28 WARN - jmeter.timers.InterruptTimer: Call Done
interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took
5.50154E-4 secs
2015/09/01 23:41:30 WARN - jmeter.timers.InterruptTimer: Run Done
interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took
6.93174E-4 secs
2015/09/01 23:41:32 WARN - jmeter.timers.InterruptTimer: Call Done
interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took
6.11455E-4 secs
2015/09/01 23:41:34 WARN - jmeter.timers.InterruptTimer: Run Done
interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took
6.1518E-4 secs
2015/09/01 23:41:36 WARN - jmeter.timers.InterruptTimer: Call Done
interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took
6.24097E-4 secs
2015/09/01 23:41:38 WARN - jmeter.timers.InterruptTimer: Run Done
interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took
7.53767E-4 secs
2015/09/01 23:41:40 WARN - jmeter.timers.InterruptTimer: Call Done
interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took
6.84471E-4 secs
2015/09/01 23:41:42 WARN - jmeter.timers.InterruptTimer: Run Done
interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took
7.34266E-4 secs
2015/09/01 23:41:44 WARN - jmeter.timers.InterruptTimer: Call Done
interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took
5.88123E-4 secs
2015/09/01 23:41:46 WARN - jmeter.timers.InterruptTimer: Run Done
interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took
3.34427E-4 secs
2015/09/01 23:41:48 WARN - jmeter.timers.InterruptTimer: Call Done
interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took
6.3848E-4 secs
2015/09/01 23:41:50 WARN - jmeter.timers.InterruptTimer: Run Done
interrupting HTTPSamplerProxy @469583928 'HTTP Request 1' took
7.77866E-4 secs
2015/09/01 23:41:52 WARN - jmeter.timers.InterruptTimer: Call Done
interrupting HTTPSamplerProxy @508309015 'HTTP Request 2' took
7.13642E-4 secs
The record times is very close (in nano seconds), perhaps, not important
(significant).
>
>> * Basic tests with the same url + Assertion Duration : the behavior is ok
>>
>> * Basic tests with the same url + a Constant Timer set to 2 sec (and
>> Interrupt Timer set to 2 sec), the results seems not ok: the url is fully
>> retrieve with a response time > 5 seconds (the request isn't interrupt, the
>> response tab show the HTML code of the page). If I disable the Constant
>> Timer, the request is interrupt in 2 sec. (the response tab show a (normal)
>> exception causing by the interruption)
>> Probably an issue...
> No, this is expected because the InterruptTimer is called before the
> delay is performed.
>
> I have not yet implemented the use of sampleStarted/sampleStopped
> which will allow the test element to be called after any other timers.
>
> At present the interrupt test element has no way to allow for other
> timer delays because it does not know what they will be.
> These occur after it is called (and before the sample).
> That is why sampleStarted/sampleStopped are needed.
> Or some other change is needed to JMeterThread that will allow the
> interrupt timer to schedule the interrupt at the correct time.
>
>> * The fields Task type and Call delay isn't localized and the values aren't
>> reset with your add a second Interrupt timer (I suppose you knows that,
>> current branch is a PoC)
> Yes, they are for testing purposes only and aren't needed in the final release.
>
>> Milamber
>>
>> [the URL for test (deflate is disable in HTTP Header):
>> https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.2/html-single/Administration_and_Configuration_Guide/index.html
>> ]
>>
>>
>>
>>
>> On 30/08/2015 17:37, sebb wrote:
>>> OK, the SampleTimeout branch now has a version that use Runnable by
>>> default, but you can use Callable (type==CALL) with an optional delay
>>> to simulate an interrupt that takes a long time.
>>>
>>> It works fine for me in simple tests.
>>>
>>> If you can break it, so much the better.
>>>
>>>
>>> On 30 August 2015 at 16:36, sebb <se...@gmail.com> wrote:
>>>> On 30 August 2015 at 11:54, sebb <se...@gmail.com> wrote:
>>>>> On 30 August 2015 at 07:12, Philippe Mouawad
>>>>> <ph...@gmail.com> wrote:
>>>>>> On Sunday, August 30, 2015, sebb <se...@gmail.com> wrote:
>>>>>>
>>>>>>> I've had a look at the classes that implement SampleListener, and
>>>>>>> apart from ResultAction and TransactionSampler, only the Listeners use
>>>>>>> it. Since usage of these should be minimised in a production test,
>>>>>>> it's likely that there won't be as many implementations as I had
>>>>>>> feard.
>>>>>> Implementation of SampleListener ?
>>>>>> Usage of which should be minimised ? SampleListener ?
>>>>> Usage of additional Listeners should be minimised in a production test.
>>>>>
>>>>>>> Also if the implementation is empty, the overhead will be quite small.
>>>>>>>
>>>>>>> [There is a work-round if it does prove expensive: the SampleListener
>>>>>>> interface could be split into two parent interfaces.]
>>>>>>>
>>>>>>> So assuming that JMeterThread implements sampleStarted/sampleStopped,
>>>>>>> the Timeout element can use the Start to set up the timer and the Stop
>>>>>>> to cancel it. This will reduce the number outstanding as much as
>>>>>>> possible.
>>>>>>>
>>>>>>> The timeouts have to be implemented using separate threads for two
>>>>>>> reasons:
>>>>>>> - it's obviously not possible to interrupt a sampler from the same
>>>>>>> thread as the sampler
>>>>>>> - depending on the sampler, and its state, the interrupt may take a
>>>>>>> while to complete, so each interrupt must be done in its own thread
>>>>>> Are you sure, calling interrupt is usually just about setting a flag
>>>>>> no?
>>>>> Ultimately yes, I guess a flag will be set.
>>>>> However I'm not sure that it is always instantaneous, as there may be
>>>>> locks to aquire.
>>>>>
>>>>>> Having 1 thread for each interruption, could lead to hundred of
>>>>>> threads running for high throughput threads (500 res/s for example), it
>>>>>> won't scale.
>>>>> That assumes that all the samples in all the threads have timeouts
>>>>> enabled.
>>>> It also assumes that the threads are running for the life of the
>>>> timeout, which I've just realised is not the case.
>>>>>> Why can't we have 1 Thread (TimeoutChecker) called every N milliseconds
>>>>>> that checks all registered JMeterThreads to check and call interrupt if
>>>>>> necessary ?
>>>>>>
>>>>> That's quite a bit of work to code, but if there is already such a
>>>>> queing mechanism it would be worth trying.
>>>> Actually the ScheduledExecutorService does just that.
>>>> It uses one thread to handle the timeouts in sequence, and then starts
>>>> a new Runnable when the timeout expires.
>>>>
>>>> Now that I have created the service as a singleton, there will only be
>>>> one extra thread (the executor) most of the time.
>>>> Since the timer tasks are cancelled if the sample completes in time,
>>>> there will only ever be at most one extra thread for each overdue
>>>> sample.
>>>>
>>>> I think it is a reasonable assumption to assume that the number of
>>>> such samples will generally be small.
>>>> At the very most it could only be one per thread.
>>>> And remember that the new thread is only started when the timeout
>>>> expires.
>>>> It is extremely unlikely that they will all expire at once, and anyway
>>>> (according to you) they don't take long to run so they won't build up.
>>>>
>>>> I've just noticed that there is a version of the Executor Service that
>>>> uses Callable rather than Runnable (not sure why I missed it before).
>>>> I asume this means the tasks are run in the same thread (I'll check
>>>> this).
>>>> If the interrupts really do happen quickly, this might be a better
>>>> choice.
>>>> Even if not, then it may not matter so long as any delays don't
>>>> continue to build up.
>>>>
>>>>>>> It should be possible to use a single shared instance of the
>>>>>>> ScheduledExecutorService; that could be lazily created using IODH. [I
>>>>>>> can try that with the current implementation]
>>>>> I've already tried it.
>>>>>
>>>>>> See my note above
>>>>>>> As to whether the Timeout class should be a Timer or some other type
>>>>>>> of test element - that does not matter so long as it can be applied to
>>>>>>> the samplers individually or when in scope.
>>>>>>>
>>>>>>> I chose Timer because it was already called in the right place, but I
>>>>>>> assume JMeterThread can call any Test class provided that it
>>>>>>> implemented the SampleInterface.
>>>>>>>
>>>>>>> It must be one of the existing Test Element classes that are handled
>>>>>>> by the Menu system otherwise it will need special handling.
>>>>>>>
>>>>>>> The scope requirement rules out Config elements and Logic Controllers.
>>>>>> It does not seem like a pre-processor to me, nor a post-procesor, nor a
>>>>>>> Listener
>>>>>>>
>>>>>>> So AFAICT the only remaining options are the Timer and Assertions.
>>>>>>>
>>>>>>> I think both are justifiable.
>>>>>> Why isn't it part of Sampler abstract class and as such a field in
>>>>>> Sampler?
>>>>> How does the user indicate that a Timeout should be applied to a
>>>>> particular Sampler?
>>>>> It's a lot of work to add Timeout fields to every GUI.
>>>>> Whereas being able to add a child test element to each applicable
>>>>> sampler is already supported.
>>>>> Further, such a test element can be applied to multiple samplers in
>>>>> scope.
>>>>> Much easier to enable and disable a single element that having to
>>>>> update each Sampler.
>>>>>
>>>>> That's why it needs to be a separate test element.
>>>>>
>>>>>> For me none of Timer nor Assertions are conceptually valid.
>>>>>> The behaviour is not a pause(so not Timer for me), it's not an
>>>>>> Assertion
>>>>>> neither as for me an Assertion only checks something.
>>>>>> Although not fully satisfying it look to me more of a PreProcessor as
>>>>>> it
>>>>>> sets a timeout on the Sampler , it can also be considered as a Post
>>>>>> Processor.
>>>>> OK, I could live with it being a Pre-Procesor.
>>>>> That has the correct scoping rules.
>>>>>
>>>>>>> The name of the class can of course be changed from InterruptTimer - I
>>>>>>> think that is probably not the best choice. Maybe something like
>>>>>>> SamplerTimeout?
>>>>>>>
>>>>>> Yes or SamplerTimeouter or SamplerInterrupter
>>>>>>
>>>>> Or even SampleTimeout - that's what it does, it applies a timeout to a
>>>>> sample.
>>>>>
>>>>>> --
>>>>>> Cordialement.
>>>>>> Philippe Mouawad.
>>
Re: Design of Timeout test element / sampler interrupter
Posted by Milamber <mi...@apache.org>.
I've tested with the updated branch, and with the new version of
JMeterThread, the behaviors of InterruptTimer & ConstantTimer have
better predictable (in my mind). All http requests stop by the Interrupt
Timer after 2sec.
On 01/09/2015 18:45, sebb wrote:
> OK, the branch now has a version of JMeterThread that supports
> sampleStarted/Stopped.
>
> Disable it by setting the property
>
> temp.sampleStarted=true
>
>
> If enabled, it allows interrupts to work in conjunction with timers.
>
>
> On 1 September 2015 at 17:40, sebb <se...@gmail.com> wrote:
>> On 1 September 2015 at 17:23, Milamber <mi...@apache.org> wrote:
>>> Hello,
>>>
>>> I make some tests.
>> Thanks!
>>
>>> * On basic scenario, with 2 http requests on the same url, Callable option
>>> seems most quick compare to Runnable (~70% more quick)
>> I'm surprised it makes that much difference.
>>
>> What were you measuring?
>> How many threads?
>>
>>> * Basic tests with the same url + Assertion Duration : the behavior is ok
>>>
>>> * Basic tests with the same url + a Constant Timer set to 2 sec (and
>>> Interrupt Timer set to 2 sec), the results seems not ok: the url is fully
>>> retrieve with a response time > 5 seconds (the request isn't interrupt, the
>>> response tab show the HTML code of the page). If I disable the Constant
>>> Timer, the request is interrupt in 2 sec. (the response tab show a (normal)
>>> exception causing by the interruption)
>>> Probably an issue...
>> No, this is expected because the InterruptTimer is called before the
>> delay is performed.
>>
>> I have not yet implemented the use of sampleStarted/sampleStopped
>> which will allow the test element to be called after any other timers.
>>
>> At present the interrupt test element has no way to allow for other
>> timer delays because it does not know what they will be.
>> These occur after it is called (and before the sample).
>> That is why sampleStarted/sampleStopped are needed.
>> Or some other change is needed to JMeterThread that will allow the
>> interrupt timer to schedule the interrupt at the correct time.
>>
>>> * The fields Task type and Call delay isn't localized and the values aren't
>>> reset with your add a second Interrupt timer (I suppose you knows that,
>>> current branch is a PoC)
>> Yes, they are for testing purposes only and aren't needed in the final release.
>>
>>> Milamber
>>>
>>> [the URL for test (deflate is disable in HTTP Header):
>>> https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.2/html-single/Administration_and_Configuration_Guide/index.html
>>> ]
>>>
>>>
>>>
>>>
>>> On 30/08/2015 17:37, sebb wrote:
>>>> OK, the SampleTimeout branch now has a version that use Runnable by
>>>> default, but you can use Callable (type==CALL) with an optional delay
>>>> to simulate an interrupt that takes a long time.
>>>>
>>>> It works fine for me in simple tests.
>>>>
>>>> If you can break it, so much the better.
>>>>
>>>>
>>>> On 30 August 2015 at 16:36, sebb <se...@gmail.com> wrote:
>>>>> On 30 August 2015 at 11:54, sebb <se...@gmail.com> wrote:
>>>>>> On 30 August 2015 at 07:12, Philippe Mouawad
>>>>>> <ph...@gmail.com> wrote:
>>>>>>> On Sunday, August 30, 2015, sebb <se...@gmail.com> wrote:
>>>>>>>
>>>>>>>> I've had a look at the classes that implement SampleListener, and
>>>>>>>> apart from ResultAction and TransactionSampler, only the Listeners use
>>>>>>>> it. Since usage of these should be minimised in a production test,
>>>>>>>> it's likely that there won't be as many implementations as I had
>>>>>>>> feard.
>>>>>>> Implementation of SampleListener ?
>>>>>>> Usage of which should be minimised ? SampleListener ?
>>>>>> Usage of additional Listeners should be minimised in a production test.
>>>>>>
>>>>>>>> Also if the implementation is empty, the overhead will be quite small.
>>>>>>>>
>>>>>>>> [There is a work-round if it does prove expensive: the SampleListener
>>>>>>>> interface could be split into two parent interfaces.]
>>>>>>>>
>>>>>>>> So assuming that JMeterThread implements sampleStarted/sampleStopped,
>>>>>>>> the Timeout element can use the Start to set up the timer and the Stop
>>>>>>>> to cancel it. This will reduce the number outstanding as much as
>>>>>>>> possible.
>>>>>>>>
>>>>>>>> The timeouts have to be implemented using separate threads for two
>>>>>>>> reasons:
>>>>>>>> - it's obviously not possible to interrupt a sampler from the same
>>>>>>>> thread as the sampler
>>>>>>>> - depending on the sampler, and its state, the interrupt may take a
>>>>>>>> while to complete, so each interrupt must be done in its own thread
>>>>>>> Are you sure, calling interrupt is usually just about setting a flag
>>>>>>> no?
>>>>>> Ultimately yes, I guess a flag will be set.
>>>>>> However I'm not sure that it is always instantaneous, as there may be
>>>>>> locks to aquire.
>>>>>>
>>>>>>> Having 1 thread for each interruption, could lead to hundred of
>>>>>>> threads running for high throughput threads (500 res/s for example), it
>>>>>>> won't scale.
>>>>>> That assumes that all the samples in all the threads have timeouts
>>>>>> enabled.
>>>>> It also assumes that the threads are running for the life of the
>>>>> timeout, which I've just realised is not the case.
>>>>>>> Why can't we have 1 Thread (TimeoutChecker) called every N milliseconds
>>>>>>> that checks all registered JMeterThreads to check and call interrupt if
>>>>>>> necessary ?
>>>>>>>
>>>>>> That's quite a bit of work to code, but if there is already such a
>>>>>> queing mechanism it would be worth trying.
>>>>> Actually the ScheduledExecutorService does just that.
>>>>> It uses one thread to handle the timeouts in sequence, and then starts
>>>>> a new Runnable when the timeout expires.
>>>>>
>>>>> Now that I have created the service as a singleton, there will only be
>>>>> one extra thread (the executor) most of the time.
>>>>> Since the timer tasks are cancelled if the sample completes in time,
>>>>> there will only ever be at most one extra thread for each overdue
>>>>> sample.
>>>>>
>>>>> I think it is a reasonable assumption to assume that the number of
>>>>> such samples will generally be small.
>>>>> At the very most it could only be one per thread.
>>>>> And remember that the new thread is only started when the timeout
>>>>> expires.
>>>>> It is extremely unlikely that they will all expire at once, and anyway
>>>>> (according to you) they don't take long to run so they won't build up.
>>>>>
>>>>> I've just noticed that there is a version of the Executor Service that
>>>>> uses Callable rather than Runnable (not sure why I missed it before).
>>>>> I asume this means the tasks are run in the same thread (I'll check
>>>>> this).
>>>>> If the interrupts really do happen quickly, this might be a better
>>>>> choice.
>>>>> Even if not, then it may not matter so long as any delays don't
>>>>> continue to build up.
>>>>>
>>>>>>>> It should be possible to use a single shared instance of the
>>>>>>>> ScheduledExecutorService; that could be lazily created using IODH. [I
>>>>>>>> can try that with the current implementation]
>>>>>> I've already tried it.
>>>>>>
>>>>>>> See my note above
>>>>>>>> As to whether the Timeout class should be a Timer or some other type
>>>>>>>> of test element - that does not matter so long as it can be applied to
>>>>>>>> the samplers individually or when in scope.
>>>>>>>>
>>>>>>>> I chose Timer because it was already called in the right place, but I
>>>>>>>> assume JMeterThread can call any Test class provided that it
>>>>>>>> implemented the SampleInterface.
>>>>>>>>
>>>>>>>> It must be one of the existing Test Element classes that are handled
>>>>>>>> by the Menu system otherwise it will need special handling.
>>>>>>>>
>>>>>>>> The scope requirement rules out Config elements and Logic Controllers.
>>>>>>> It does not seem like a pre-processor to me, nor a post-procesor, nor a
>>>>>>>> Listener
>>>>>>>>
>>>>>>>> So AFAICT the only remaining options are the Timer and Assertions.
>>>>>>>>
>>>>>>>> I think both are justifiable.
>>>>>>> Why isn't it part of Sampler abstract class and as such a field in
>>>>>>> Sampler?
>>>>>> How does the user indicate that a Timeout should be applied to a
>>>>>> particular Sampler?
>>>>>> It's a lot of work to add Timeout fields to every GUI.
>>>>>> Whereas being able to add a child test element to each applicable
>>>>>> sampler is already supported.
>>>>>> Further, such a test element can be applied to multiple samplers in
>>>>>> scope.
>>>>>> Much easier to enable and disable a single element that having to
>>>>>> update each Sampler.
>>>>>>
>>>>>> That's why it needs to be a separate test element.
>>>>>>
>>>>>>> For me none of Timer nor Assertions are conceptually valid.
>>>>>>> The behaviour is not a pause(so not Timer for me), it's not an
>>>>>>> Assertion
>>>>>>> neither as for me an Assertion only checks something.
>>>>>>> Although not fully satisfying it look to me more of a PreProcessor as
>>>>>>> it
>>>>>>> sets a timeout on the Sampler , it can also be considered as a Post
>>>>>>> Processor.
>>>>>> OK, I could live with it being a Pre-Procesor.
>>>>>> That has the correct scoping rules.
>>>>>>
>>>>>>>> The name of the class can of course be changed from InterruptTimer - I
>>>>>>>> think that is probably not the best choice. Maybe something like
>>>>>>>> SamplerTimeout?
>>>>>>>>
>>>>>>> Yes or SamplerTimeouter or SamplerInterrupter
>>>>>>>
>>>>>> Or even SampleTimeout - that's what it does, it applies a timeout to a
>>>>>> sample.
>>>>>>
>>>>>>> --
>>>>>>> Cordialement.
>>>>>>> Philippe Mouawad.
>>>
Re: Design of Timeout test element / sampler interrupter
Posted by sebb <se...@gmail.com>.
OK, the branch now has a version of JMeterThread that supports
sampleStarted/Stopped.
Disable it by setting the property
temp.sampleStarted=true
If enabled, it allows interrupts to work in conjunction with timers.
On 1 September 2015 at 17:40, sebb <se...@gmail.com> wrote:
> On 1 September 2015 at 17:23, Milamber <mi...@apache.org> wrote:
>> Hello,
>>
>> I make some tests.
>
> Thanks!
>
>> * On basic scenario, with 2 http requests on the same url, Callable option
>> seems most quick compare to Runnable (~70% more quick)
>
> I'm surprised it makes that much difference.
>
> What were you measuring?
> How many threads?
>
>> * Basic tests with the same url + Assertion Duration : the behavior is ok
>>
>> * Basic tests with the same url + a Constant Timer set to 2 sec (and
>> Interrupt Timer set to 2 sec), the results seems not ok: the url is fully
>> retrieve with a response time > 5 seconds (the request isn't interrupt, the
>> response tab show the HTML code of the page). If I disable the Constant
>> Timer, the request is interrupt in 2 sec. (the response tab show a (normal)
>> exception causing by the interruption)
>> Probably an issue...
>
> No, this is expected because the InterruptTimer is called before the
> delay is performed.
>
> I have not yet implemented the use of sampleStarted/sampleStopped
> which will allow the test element to be called after any other timers.
>
> At present the interrupt test element has no way to allow for other
> timer delays because it does not know what they will be.
> These occur after it is called (and before the sample).
> That is why sampleStarted/sampleStopped are needed.
> Or some other change is needed to JMeterThread that will allow the
> interrupt timer to schedule the interrupt at the correct time.
>
>> * The fields Task type and Call delay isn't localized and the values aren't
>> reset with your add a second Interrupt timer (I suppose you knows that,
>> current branch is a PoC)
>
> Yes, they are for testing purposes only and aren't needed in the final release.
>
>> Milamber
>>
>> [the URL for test (deflate is disable in HTTP Header):
>> https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.2/html-single/Administration_and_Configuration_Guide/index.html
>> ]
>>
>>
>>
>>
>> On 30/08/2015 17:37, sebb wrote:
>>>
>>> OK, the SampleTimeout branch now has a version that use Runnable by
>>> default, but you can use Callable (type==CALL) with an optional delay
>>> to simulate an interrupt that takes a long time.
>>>
>>> It works fine for me in simple tests.
>>>
>>> If you can break it, so much the better.
>>>
>>>
>>> On 30 August 2015 at 16:36, sebb <se...@gmail.com> wrote:
>>>>
>>>> On 30 August 2015 at 11:54, sebb <se...@gmail.com> wrote:
>>>>>
>>>>> On 30 August 2015 at 07:12, Philippe Mouawad
>>>>> <ph...@gmail.com> wrote:
>>>>>>
>>>>>> On Sunday, August 30, 2015, sebb <se...@gmail.com> wrote:
>>>>>>
>>>>>>> I've had a look at the classes that implement SampleListener, and
>>>>>>> apart from ResultAction and TransactionSampler, only the Listeners use
>>>>>>> it. Since usage of these should be minimised in a production test,
>>>>>>> it's likely that there won't be as many implementations as I had
>>>>>>> feard.
>>>>>>
>>>>>> Implementation of SampleListener ?
>>>>>> Usage of which should be minimised ? SampleListener ?
>>>>>
>>>>> Usage of additional Listeners should be minimised in a production test.
>>>>>
>>>>>>> Also if the implementation is empty, the overhead will be quite small.
>>>>>>>
>>>>>>> [There is a work-round if it does prove expensive: the SampleListener
>>>>>>> interface could be split into two parent interfaces.]
>>>>>>>
>>>>>>> So assuming that JMeterThread implements sampleStarted/sampleStopped,
>>>>>>> the Timeout element can use the Start to set up the timer and the Stop
>>>>>>> to cancel it. This will reduce the number outstanding as much as
>>>>>>> possible.
>>>>>>>
>>>>>>> The timeouts have to be implemented using separate threads for two
>>>>>>> reasons:
>>>>>>> - it's obviously not possible to interrupt a sampler from the same
>>>>>>> thread as the sampler
>>>>>>> - depending on the sampler, and its state, the interrupt may take a
>>>>>>> while to complete, so each interrupt must be done in its own thread
>>>>>>
>>>>>> Are you sure, calling interrupt is usually just about setting a flag
>>>>>> no?
>>>>>
>>>>> Ultimately yes, I guess a flag will be set.
>>>>> However I'm not sure that it is always instantaneous, as there may be
>>>>> locks to aquire.
>>>>>
>>>>>> Having 1 thread for each interruption, could lead to hundred of
>>>>>> threads running for high throughput threads (500 res/s for example), it
>>>>>> won't scale.
>>>>>
>>>>> That assumes that all the samples in all the threads have timeouts
>>>>> enabled.
>>>>
>>>> It also assumes that the threads are running for the life of the
>>>> timeout, which I've just realised is not the case.
>>>>>>
>>>>>> Why can't we have 1 Thread (TimeoutChecker) called every N milliseconds
>>>>>> that checks all registered JMeterThreads to check and call interrupt if
>>>>>> necessary ?
>>>>>>
>>>>> That's quite a bit of work to code, but if there is already such a
>>>>> queing mechanism it would be worth trying.
>>>>
>>>> Actually the ScheduledExecutorService does just that.
>>>> It uses one thread to handle the timeouts in sequence, and then starts
>>>> a new Runnable when the timeout expires.
>>>>
>>>> Now that I have created the service as a singleton, there will only be
>>>> one extra thread (the executor) most of the time.
>>>> Since the timer tasks are cancelled if the sample completes in time,
>>>> there will only ever be at most one extra thread for each overdue
>>>> sample.
>>>>
>>>> I think it is a reasonable assumption to assume that the number of
>>>> such samples will generally be small.
>>>> At the very most it could only be one per thread.
>>>> And remember that the new thread is only started when the timeout
>>>> expires.
>>>> It is extremely unlikely that they will all expire at once, and anyway
>>>> (according to you) they don't take long to run so they won't build up.
>>>>
>>>> I've just noticed that there is a version of the Executor Service that
>>>> uses Callable rather than Runnable (not sure why I missed it before).
>>>> I asume this means the tasks are run in the same thread (I'll check
>>>> this).
>>>> If the interrupts really do happen quickly, this might be a better
>>>> choice.
>>>> Even if not, then it may not matter so long as any delays don't
>>>> continue to build up.
>>>>
>>>>>>> It should be possible to use a single shared instance of the
>>>>>>> ScheduledExecutorService; that could be lazily created using IODH. [I
>>>>>>> can try that with the current implementation]
>>>>>
>>>>> I've already tried it.
>>>>>
>>>>>> See my note above
>>>>>>>
>>>>>>> As to whether the Timeout class should be a Timer or some other type
>>>>>>> of test element - that does not matter so long as it can be applied to
>>>>>>> the samplers individually or when in scope.
>>>>>>>
>>>>>>> I chose Timer because it was already called in the right place, but I
>>>>>>> assume JMeterThread can call any Test class provided that it
>>>>>>> implemented the SampleInterface.
>>>>>>>
>>>>>>> It must be one of the existing Test Element classes that are handled
>>>>>>> by the Menu system otherwise it will need special handling.
>>>>>>>
>>>>>>> The scope requirement rules out Config elements and Logic Controllers.
>>>>>>
>>>>>> It does not seem like a pre-processor to me, nor a post-procesor, nor a
>>>>>>>
>>>>>>> Listener
>>>>>>>
>>>>>>> So AFAICT the only remaining options are the Timer and Assertions.
>>>>>>>
>>>>>>> I think both are justifiable.
>>>>>>
>>>>>> Why isn't it part of Sampler abstract class and as such a field in
>>>>>> Sampler?
>>>>>
>>>>> How does the user indicate that a Timeout should be applied to a
>>>>> particular Sampler?
>>>>> It's a lot of work to add Timeout fields to every GUI.
>>>>> Whereas being able to add a child test element to each applicable
>>>>> sampler is already supported.
>>>>> Further, such a test element can be applied to multiple samplers in
>>>>> scope.
>>>>> Much easier to enable and disable a single element that having to
>>>>> update each Sampler.
>>>>>
>>>>> That's why it needs to be a separate test element.
>>>>>
>>>>>>
>>>>>> For me none of Timer nor Assertions are conceptually valid.
>>>>>> The behaviour is not a pause(so not Timer for me), it's not an
>>>>>> Assertion
>>>>>> neither as for me an Assertion only checks something.
>>>>>> Although not fully satisfying it look to me more of a PreProcessor as
>>>>>> it
>>>>>> sets a timeout on the Sampler , it can also be considered as a Post
>>>>>> Processor.
>>>>>
>>>>> OK, I could live with it being a Pre-Procesor.
>>>>> That has the correct scoping rules.
>>>>>
>>>>>>
>>>>>>> The name of the class can of course be changed from InterruptTimer - I
>>>>>>> think that is probably not the best choice. Maybe something like
>>>>>>> SamplerTimeout?
>>>>>>>
>>>>>> Yes or SamplerTimeouter or SamplerInterrupter
>>>>>>
>>>>> Or even SampleTimeout - that's what it does, it applies a timeout to a
>>>>> sample.
>>>>>
>>>>>> --
>>>>>> Cordialement.
>>>>>> Philippe Mouawad.
>>
>>
Re: Design of Timeout test element / sampler interrupter
Posted by sebb <se...@gmail.com>.
On 1 September 2015 at 17:23, Milamber <mi...@apache.org> wrote:
> Hello,
>
> I make some tests.
Thanks!
> * On basic scenario, with 2 http requests on the same url, Callable option
> seems most quick compare to Runnable (~70% more quick)
I'm surprised it makes that much difference.
What were you measuring?
How many threads?
> * Basic tests with the same url + Assertion Duration : the behavior is ok
>
> * Basic tests with the same url + a Constant Timer set to 2 sec (and
> Interrupt Timer set to 2 sec), the results seems not ok: the url is fully
> retrieve with a response time > 5 seconds (the request isn't interrupt, the
> response tab show the HTML code of the page). If I disable the Constant
> Timer, the request is interrupt in 2 sec. (the response tab show a (normal)
> exception causing by the interruption)
> Probably an issue...
No, this is expected because the InterruptTimer is called before the
delay is performed.
I have not yet implemented the use of sampleStarted/sampleStopped
which will allow the test element to be called after any other timers.
At present the interrupt test element has no way to allow for other
timer delays because it does not know what they will be.
These occur after it is called (and before the sample).
That is why sampleStarted/sampleStopped are needed.
Or some other change is needed to JMeterThread that will allow the
interrupt timer to schedule the interrupt at the correct time.
> * The fields Task type and Call delay isn't localized and the values aren't
> reset with your add a second Interrupt timer (I suppose you knows that,
> current branch is a PoC)
Yes, they are for testing purposes only and aren't needed in the final release.
> Milamber
>
> [the URL for test (deflate is disable in HTTP Header):
> https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.2/html-single/Administration_and_Configuration_Guide/index.html
> ]
>
>
>
>
> On 30/08/2015 17:37, sebb wrote:
>>
>> OK, the SampleTimeout branch now has a version that use Runnable by
>> default, but you can use Callable (type==CALL) with an optional delay
>> to simulate an interrupt that takes a long time.
>>
>> It works fine for me in simple tests.
>>
>> If you can break it, so much the better.
>>
>>
>> On 30 August 2015 at 16:36, sebb <se...@gmail.com> wrote:
>>>
>>> On 30 August 2015 at 11:54, sebb <se...@gmail.com> wrote:
>>>>
>>>> On 30 August 2015 at 07:12, Philippe Mouawad
>>>> <ph...@gmail.com> wrote:
>>>>>
>>>>> On Sunday, August 30, 2015, sebb <se...@gmail.com> wrote:
>>>>>
>>>>>> I've had a look at the classes that implement SampleListener, and
>>>>>> apart from ResultAction and TransactionSampler, only the Listeners use
>>>>>> it. Since usage of these should be minimised in a production test,
>>>>>> it's likely that there won't be as many implementations as I had
>>>>>> feard.
>>>>>
>>>>> Implementation of SampleListener ?
>>>>> Usage of which should be minimised ? SampleListener ?
>>>>
>>>> Usage of additional Listeners should be minimised in a production test.
>>>>
>>>>>> Also if the implementation is empty, the overhead will be quite small.
>>>>>>
>>>>>> [There is a work-round if it does prove expensive: the SampleListener
>>>>>> interface could be split into two parent interfaces.]
>>>>>>
>>>>>> So assuming that JMeterThread implements sampleStarted/sampleStopped,
>>>>>> the Timeout element can use the Start to set up the timer and the Stop
>>>>>> to cancel it. This will reduce the number outstanding as much as
>>>>>> possible.
>>>>>>
>>>>>> The timeouts have to be implemented using separate threads for two
>>>>>> reasons:
>>>>>> - it's obviously not possible to interrupt a sampler from the same
>>>>>> thread as the sampler
>>>>>> - depending on the sampler, and its state, the interrupt may take a
>>>>>> while to complete, so each interrupt must be done in its own thread
>>>>>
>>>>> Are you sure, calling interrupt is usually just about setting a flag
>>>>> no?
>>>>
>>>> Ultimately yes, I guess a flag will be set.
>>>> However I'm not sure that it is always instantaneous, as there may be
>>>> locks to aquire.
>>>>
>>>>> Having 1 thread for each interruption, could lead to hundred of
>>>>> threads running for high throughput threads (500 res/s for example), it
>>>>> won't scale.
>>>>
>>>> That assumes that all the samples in all the threads have timeouts
>>>> enabled.
>>>
>>> It also assumes that the threads are running for the life of the
>>> timeout, which I've just realised is not the case.
>>>>>
>>>>> Why can't we have 1 Thread (TimeoutChecker) called every N milliseconds
>>>>> that checks all registered JMeterThreads to check and call interrupt if
>>>>> necessary ?
>>>>>
>>>> That's quite a bit of work to code, but if there is already such a
>>>> queing mechanism it would be worth trying.
>>>
>>> Actually the ScheduledExecutorService does just that.
>>> It uses one thread to handle the timeouts in sequence, and then starts
>>> a new Runnable when the timeout expires.
>>>
>>> Now that I have created the service as a singleton, there will only be
>>> one extra thread (the executor) most of the time.
>>> Since the timer tasks are cancelled if the sample completes in time,
>>> there will only ever be at most one extra thread for each overdue
>>> sample.
>>>
>>> I think it is a reasonable assumption to assume that the number of
>>> such samples will generally be small.
>>> At the very most it could only be one per thread.
>>> And remember that the new thread is only started when the timeout
>>> expires.
>>> It is extremely unlikely that they will all expire at once, and anyway
>>> (according to you) they don't take long to run so they won't build up.
>>>
>>> I've just noticed that there is a version of the Executor Service that
>>> uses Callable rather than Runnable (not sure why I missed it before).
>>> I asume this means the tasks are run in the same thread (I'll check
>>> this).
>>> If the interrupts really do happen quickly, this might be a better
>>> choice.
>>> Even if not, then it may not matter so long as any delays don't
>>> continue to build up.
>>>
>>>>>> It should be possible to use a single shared instance of the
>>>>>> ScheduledExecutorService; that could be lazily created using IODH. [I
>>>>>> can try that with the current implementation]
>>>>
>>>> I've already tried it.
>>>>
>>>>> See my note above
>>>>>>
>>>>>> As to whether the Timeout class should be a Timer or some other type
>>>>>> of test element - that does not matter so long as it can be applied to
>>>>>> the samplers individually or when in scope.
>>>>>>
>>>>>> I chose Timer because it was already called in the right place, but I
>>>>>> assume JMeterThread can call any Test class provided that it
>>>>>> implemented the SampleInterface.
>>>>>>
>>>>>> It must be one of the existing Test Element classes that are handled
>>>>>> by the Menu system otherwise it will need special handling.
>>>>>>
>>>>>> The scope requirement rules out Config elements and Logic Controllers.
>>>>>
>>>>> It does not seem like a pre-processor to me, nor a post-procesor, nor a
>>>>>>
>>>>>> Listener
>>>>>>
>>>>>> So AFAICT the only remaining options are the Timer and Assertions.
>>>>>>
>>>>>> I think both are justifiable.
>>>>>
>>>>> Why isn't it part of Sampler abstract class and as such a field in
>>>>> Sampler?
>>>>
>>>> How does the user indicate that a Timeout should be applied to a
>>>> particular Sampler?
>>>> It's a lot of work to add Timeout fields to every GUI.
>>>> Whereas being able to add a child test element to each applicable
>>>> sampler is already supported.
>>>> Further, such a test element can be applied to multiple samplers in
>>>> scope.
>>>> Much easier to enable and disable a single element that having to
>>>> update each Sampler.
>>>>
>>>> That's why it needs to be a separate test element.
>>>>
>>>>>
>>>>> For me none of Timer nor Assertions are conceptually valid.
>>>>> The behaviour is not a pause(so not Timer for me), it's not an
>>>>> Assertion
>>>>> neither as for me an Assertion only checks something.
>>>>> Although not fully satisfying it look to me more of a PreProcessor as
>>>>> it
>>>>> sets a timeout on the Sampler , it can also be considered as a Post
>>>>> Processor.
>>>>
>>>> OK, I could live with it being a Pre-Procesor.
>>>> That has the correct scoping rules.
>>>>
>>>>>
>>>>>> The name of the class can of course be changed from InterruptTimer - I
>>>>>> think that is probably not the best choice. Maybe something like
>>>>>> SamplerTimeout?
>>>>>>
>>>>> Yes or SamplerTimeouter or SamplerInterrupter
>>>>>
>>>> Or even SampleTimeout - that's what it does, it applies a timeout to a
>>>> sample.
>>>>
>>>>> --
>>>>> Cordialement.
>>>>> Philippe Mouawad.
>
>