Default value for parameters

[Antonio Gomes Rodrigues <ra...@gmail.com>]

 Hi,


In this issue : *Bug 59152*
<https://bz.apache.org/bugzilla/show_bug.cgi?id=59152> - Thread Group:
Change "Action to be taken after a Sample Error" value from "Continue" to
"Start Next thread loop"

I have modified default value in "Thread Group" component.

Like I have said in the issue, for me the right default value is "Start
Next thread loop" to :

   - avoid raising too much exception in application and slow it
   - be more realist (for example if a user failed to login and/or failed
   to fill out a form, he will stop the "iteration" and not avoid the error
   and try to execute next action)

It's why I think the majority of JMeter users use like it (or not because
they don't know this feature)


What do you think ?


There another default value I think it needs to be changed.

For example, to "HTTP Cache Manager", I think, "Clear cache each
Iteration?" must be true and not false like actually

But it's only my opinion and before to check all default value and modify
them I would like to know the process to avoid wasting your and my time


Do you have a process?


Thanks,

Antonio

Re: Default value for parameters

[Steven Swor <sw...@gmail.com>]

My $.02 below:

On Wed, May 25, 2016 at 12:46 AM, sebb <se...@gmail.com> wrote:

> On 24 May 2016 at 15:37, Antonio Gomes Rodrigues <ra...@gmail.com> wrote:
> > Hi Sebb,
> >
> > Thanks to your answer
> >
> > My answer in your mail
> >
> >
> > 2016-05-24 16:24 GMT+02:00 sebb <se...@gmail.com>:
> >
> >> On 24 May 2016 at 12:18, Antonio Gomes Rodrigues <ra...@gmail.com>
> wrote:
> >> >  Hi,
> >> >
> >> >
> >> > In this issue : *Bug 59152*
> >> > <https://bz.apache.org/bugzilla/show_bug.cgi?id=59152> - Thread
> Group:
> >> > Change "Action to be taken after a Sample Error" value from
> "Continue" to
> >> > "Start Next thread loop"
> >> >
> >> > I have modified default value in "Thread Group" component.
> >> >
> >> > Like I have said in the issue, for me the right default value is
> "Start
> >> > Next thread loop" to :
> >> >
> >> >    - avoid raising too much exception in application and slow it
> >> >    - be more realist (for example if a user failed to login and/or
> failed
> >> >    to fill out a form, he will stop the "iteration" and not avoid the
> >> error
> >> >    and try to execute next action)
> >>
> >> Surely it would be more sensible to stop the entire thread/test in this
> >> case?
> >>
> > It depends
> > If all of you dataset is faulty, yes
> > If it's only a small part of your dataset which is faulty, no.
>
> That was my argument here: it's not possible to choose the 'right'
> on-error behaviour.
>
> When I was a JMeter newbie, I made the mistake of setting the error
behavior in one of my test plans to "start next loop", when the first
sampler was pointing to the wrong URL and had no timer on it (although some
of the later samplers did). My single thread test execution ended up
bringing down the whole environment. I get the feeling it would trip up a
lot of users if the default behaviour was changed.


> >
> >
> >
> >>
> >> > It's why I think the majority of JMeter users use like it (or not
> because
> >> > they don't know this feature)
> >> >
> >> >
> >> > What do you think ?
> >> >
> >>
> >> We already discussed this and decided against it.
> >>
> >
> > Ok
> >
> > If you think that mainstream test case is to "continue on errors", I have
> > nothing to say
> >
> > I can still use template functionality for my use to avoid extra work to
> > modify this value.
> >
>
> That's the best idea.
>
> You can even publish your templates if you think they will help others.
>
> >
> >>
> >> It only makes sense to skip the rest of the samples in a loop if each
> >> is dependent on the previous one completing successfully.
> >> And only if subsequent loops are independent of previous loops,
> >> otherwise it makes more sense to stop the thread or even the entire
> >> test.
> >>
> >> That may be true for some test plans but by no means all.
> >>
> >> The change would produce unexpected behaviour for existing users.
> >>
> >> > There another default value I think it needs to be changed.
> >>
> >> Please use separate e-mails for separate issues.
> >>
> >> > For example, to "HTTP Cache Manager", I think, "Clear cache each
> >> > Iteration?" must be true and not false like actually
> >> >
> >> > But it's only my opinion and before to check all default value and
> modify
> >> > them I would like to know the process to avoid wasting your and my
> time
> >> >
> >> >
> >> > Do you have a process?
> >>
> >> People get used to a particular behaviour, and moving things around or
> >> changing defaults causes problems for existing users when creating
> >> test plans.
> >>
> >
> >> Imagine how you would feel if the default were changed to Stop Test.
> >> (as noted above, it's a reasonable choice)
> >>
> >> So defaults should only be changed if there is a pressing need to do so.
> >>
> >> If you think there is a compelling case for changing the behaviour of
> >> JMeter, it should be discussed on this list.
> >>
> >
> > No problem, what I would like to know is if there is a process and you
> > answer me
> >
> >
> > Before modify a default value, I will ask it before in this list.
>

When I test an application, I prefer to use separate test runs for "clear
cache on each iteration", or separate thread groups with their own cache
manager. It's totally up to the user running the test how they want to
configure the caching behavior, based on what they want to see. For load
testing a web application, preserving the cache between iterations may
produce a more realistic workload on the system under test - there may be
any number of intermediate caches sitting between the user and your app,
and they may not be part of your actual platform.

Think of a user browsing a public web site while at work... even if that
particular user's browser cache is empty, the static content could be
served up by their corporate web proxy's cache if one of their colleagues
had been on the site before... so those requests would still be offloaded.
This is also what would happen if you were using a content delivery network
for your site. A lot of the traffic would be handled by the CDN, so the
actual amount of requests that made it through to your application would be
less. Disabling "clear cache on each iteration" (the current default) would
emulate that sort of behavior even if the test plan was pointing directly
at your site.

>
> >>
> >> But UI compatibility is very important, especially for defaults where
> >> the user is not forced to choose.
> >>
> >> But even where choices have to be made, consistency is important.
> >> Imagine how awkward it would be if the Yes No Cancel buttons were
> >> presented in a different order.
> >>
> >> >
> >> > Thanks,
> >> >
> >> > Antonio
> >>
>

Re: Default value for parameters

[sebb <se...@gmail.com>]

On 24 May 2016 at 15:37, Antonio Gomes Rodrigues <ra...@gmail.com> wrote:
> Hi Sebb,
>
> Thanks to your answer
>
> My answer in your mail
>
>
> 2016-05-24 16:24 GMT+02:00 sebb <se...@gmail.com>:
>
>> On 24 May 2016 at 12:18, Antonio Gomes Rodrigues <ra...@gmail.com> wrote:
>> >  Hi,
>> >
>> >
>> > In this issue : *Bug 59152*
>> > <https://bz.apache.org/bugzilla/show_bug.cgi?id=59152> - Thread Group:
>> > Change "Action to be taken after a Sample Error" value from "Continue" to
>> > "Start Next thread loop"
>> >
>> > I have modified default value in "Thread Group" component.
>> >
>> > Like I have said in the issue, for me the right default value is "Start
>> > Next thread loop" to :
>> >
>> >    - avoid raising too much exception in application and slow it
>> >    - be more realist (for example if a user failed to login and/or failed
>> >    to fill out a form, he will stop the "iteration" and not avoid the
>> error
>> >    and try to execute next action)
>>
>> Surely it would be more sensible to stop the entire thread/test in this
>> case?
>>
> It depends
> If all of you dataset is faulty, yes
> If it's only a small part of your dataset which is faulty, no.

That was my argument here: it's not possible to choose the 'right'
on-error behaviour.

>
>
>
>>
>> > It's why I think the majority of JMeter users use like it (or not because
>> > they don't know this feature)
>> >
>> >
>> > What do you think ?
>> >
>>
>> We already discussed this and decided against it.
>>
>
> Ok
>
> If you think that mainstream test case is to "continue on errors", I have
> nothing to say
>
> I can still use template functionality for my use to avoid extra work to
> modify this value.
>

That's the best idea.

You can even publish your templates if you think they will help others.

>
>>
>> It only makes sense to skip the rest of the samples in a loop if each
>> is dependent on the previous one completing successfully.
>> And only if subsequent loops are independent of previous loops,
>> otherwise it makes more sense to stop the thread or even the entire
>> test.
>>
>> That may be true for some test plans but by no means all.
>>
>> The change would produce unexpected behaviour for existing users.
>>
>> > There another default value I think it needs to be changed.
>>
>> Please use separate e-mails for separate issues.
>>
>> > For example, to "HTTP Cache Manager", I think, "Clear cache each
>> > Iteration?" must be true and not false like actually
>> >
>> > But it's only my opinion and before to check all default value and modify
>> > them I would like to know the process to avoid wasting your and my time
>> >
>> >
>> > Do you have a process?
>>
>> People get used to a particular behaviour, and moving things around or
>> changing defaults causes problems for existing users when creating
>> test plans.
>>
>
>> Imagine how you would feel if the default were changed to Stop Test.
>> (as noted above, it's a reasonable choice)
>>
>> So defaults should only be changed if there is a pressing need to do so.
>>
>> If you think there is a compelling case for changing the behaviour of
>> JMeter, it should be discussed on this list.
>>
>
> No problem, what I would like to know is if there is a process and you
> answer me
>
>
> Before modify a default value, I will ask it before in this list.
>
>>
>> But UI compatibility is very important, especially for defaults where
>> the user is not forced to choose.
>>
>> But even where choices have to be made, consistency is important.
>> Imagine how awkward it would be if the Yes No Cancel buttons were
>> presented in a different order.
>>
>> >
>> > Thanks,
>> >
>> > Antonio
>>

Re: Default value for parameters

[Antonio Gomes Rodrigues <ra...@gmail.com>]

Hi Sebb,

Thanks to your answer

My answer in your mail


2016-05-24 16:24 GMT+02:00 sebb <se...@gmail.com>:

> On 24 May 2016 at 12:18, Antonio Gomes Rodrigues <ra...@gmail.com> wrote:
> >  Hi,
> >
> >
> > In this issue : *Bug 59152*
> > <https://bz.apache.org/bugzilla/show_bug.cgi?id=59152> - Thread Group:
> > Change "Action to be taken after a Sample Error" value from "Continue" to
> > "Start Next thread loop"
> >
> > I have modified default value in "Thread Group" component.
> >
> > Like I have said in the issue, for me the right default value is "Start
> > Next thread loop" to :
> >
> >    - avoid raising too much exception in application and slow it
> >    - be more realist (for example if a user failed to login and/or failed
> >    to fill out a form, he will stop the "iteration" and not avoid the
> error
> >    and try to execute next action)
>
> Surely it would be more sensible to stop the entire thread/test in this
> case?
>
It depends
If all of you dataset is faulty, yes
If it's only a small part of your dataset which is faulty, no.




>
> > It's why I think the majority of JMeter users use like it (or not because
> > they don't know this feature)
> >
> >
> > What do you think ?
> >
>
> We already discussed this and decided against it.
>

Ok

If you think that mainstream test case is to "continue on errors", I have
nothing to say

I can still use template functionality for my use to avoid extra work to
modify this value.



>
> It only makes sense to skip the rest of the samples in a loop if each
> is dependent on the previous one completing successfully.
> And only if subsequent loops are independent of previous loops,
> otherwise it makes more sense to stop the thread or even the entire
> test.
>
> That may be true for some test plans but by no means all.
>
> The change would produce unexpected behaviour for existing users.
>
> > There another default value I think it needs to be changed.
>
> Please use separate e-mails for separate issues.
>
> > For example, to "HTTP Cache Manager", I think, "Clear cache each
> > Iteration?" must be true and not false like actually
> >
> > But it's only my opinion and before to check all default value and modify
> > them I would like to know the process to avoid wasting your and my time
> >
> >
> > Do you have a process?
>
> People get used to a particular behaviour, and moving things around or
> changing defaults causes problems for existing users when creating
> test plans.
>

> Imagine how you would feel if the default were changed to Stop Test.
> (as noted above, it's a reasonable choice)
>
> So defaults should only be changed if there is a pressing need to do so.
>
> If you think there is a compelling case for changing the behaviour of
> JMeter, it should be discussed on this list.
>

No problem, what I would like to know is if there is a process and you
answer me


Before modify a default value, I will ask it before in this list.


>
> But UI compatibility is very important, especially for defaults where
> the user is not forced to choose.
>
> But even where choices have to be made, consistency is important.
> Imagine how awkward it would be if the Yes No Cancel buttons were
> presented in a different order.
>
> >
> > Thanks,
> >
> > Antonio
>

Re: Default value for parameters

[sebb <se...@gmail.com>]

On 24 May 2016 at 12:18, Antonio Gomes Rodrigues <ra...@gmail.com> wrote:
>  Hi,
>
>
> In this issue : *Bug 59152*
> <https://bz.apache.org/bugzilla/show_bug.cgi?id=59152> - Thread Group:
> Change "Action to be taken after a Sample Error" value from "Continue" to
> "Start Next thread loop"
>
> I have modified default value in "Thread Group" component.
>
> Like I have said in the issue, for me the right default value is "Start
> Next thread loop" to :
>
>    - avoid raising too much exception in application and slow it
>    - be more realist (for example if a user failed to login and/or failed
>    to fill out a form, he will stop the "iteration" and not avoid the error
>    and try to execute next action)

Surely it would be more sensible to stop the entire thread/test in this case?

> It's why I think the majority of JMeter users use like it (or not because
> they don't know this feature)
>
>
> What do you think ?
>

We already discussed this and decided against it.

It only makes sense to skip the rest of the samples in a loop if each
is dependent on the previous one completing successfully.
And only if subsequent loops are independent of previous loops,
otherwise it makes more sense to stop the thread or even the entire
test.

That may be true for some test plans but by no means all.

The change would produce unexpected behaviour for existing users.

> There another default value I think it needs to be changed.

Please use separate e-mails for separate issues.

> For example, to "HTTP Cache Manager", I think, "Clear cache each
> Iteration?" must be true and not false like actually
>
> But it's only my opinion and before to check all default value and modify
> them I would like to know the process to avoid wasting your and my time
>
>
> Do you have a process?

People get used to a particular behaviour, and moving things around or
changing defaults causes problems for existing users when creating
test plans.

Imagine how you would feel if the default were changed to Stop Test.
(as noted above, it's a reasonable choice)

So defaults should only be changed if there is a pressing need to do so.

If you think there is a compelling case for changing the behaviour of
JMeter, it should be discussed on this list.

But UI compatibility is very important, especially for defaults where
the user is not forced to choose.

But even where choices have to be made, consistency is important.
Imagine how awkward it would be if the Yes No Cancel buttons were
presented in a different order.

>
> Thanks,
>
> Antonio

Re: Default value for parameters

[Antonio Gomes Rodrigues <ra...@gmail.com>]

Thanks to the precision


2016-05-24 15:47 GMT+02:00 Vladimir Sitnikov <si...@gmail.com>:

> >
> >
> > >
> > > Antonio>For example, to "HTTP Cache Manager", I think, "Clear cache
> each
> > > Antonio>Iteration?" must be true and not false like actually
> > >
> > > Why's that?
> > > Do you clear browser caches each time before opening www.google.com?
> > >
> >
> >
> > No ;-)
> >
> > But in a load test, usually I don't have enough user dataset (login,
> > password) to simulate all the users. The workaround is to use less
> > login/password than "real users" but to use it more time and clear cache
> > each time.
> >
> >
> I claim that everyone would miss to uncheck "clear cache at each
> iteration", thus it would result in "wrong".
>
>
>
> >
> > Another benefit to clear the cache each time is easiest to analyze the
> > results. If the cache is not cleared I have two types of population and
> > need to separate it to analyze them (e.g. 2 threads group, one with cache
> > and another without cache)
> >
> > Note: performance tests should reflect real-life.
> I agree there are cases when you need to know/care of "performance of
> uncached user".
> However, in typical cases, everybody optimizes for "cached" case. Thus
> having "clear cache" is like shooting into one's foot.
>
>
> > Have we got statistics (from stackoverflow...) how JMeter is used?
> > It will help to make the good choices
> >
>
> We might want to collect anonymous statistics somehow.
>
> Vladimir
>

Re: Default value for parameters

[Vladimir Sitnikov <si...@gmail.com>]

>
>
> >
> > Antonio>For example, to "HTTP Cache Manager", I think, "Clear cache each
> > Antonio>Iteration?" must be true and not false like actually
> >
> > Why's that?
> > Do you clear browser caches each time before opening www.google.com?
> >
>
>
> No ;-)
>
> But in a load test, usually I don't have enough user dataset (login,
> password) to simulate all the users. The workaround is to use less
> login/password than "real users" but to use it more time and clear cache
> each time.
>
>
I claim that everyone would miss to uncheck "clear cache at each
iteration", thus it would result in "wrong".



>
> Another benefit to clear the cache each time is easiest to analyze the
> results. If the cache is not cleared I have two types of population and
> need to separate it to analyze them (e.g. 2 threads group, one with cache
> and another without cache)
>
> Note: performance tests should reflect real-life.
I agree there are cases when you need to know/care of "performance of
uncached user".
However, in typical cases, everybody optimizes for "cached" case. Thus
having "clear cache" is like shooting into one's foot.


> Have we got statistics (from stackoverflow...) how JMeter is used?
> It will help to make the good choices
>

We might want to collect anonymous statistics somehow.

Vladimir

Re: Default value for parameters

[Antonio Gomes Rodrigues <ra...@gmail.com>]

Hi Vladimir,

Thanks to you quick answer

My response in your mail

2016-05-24 13:25 GMT+02:00 Vladimir Sitnikov <si...@gmail.com>:

> Antonio>"Action to be taken after a Sample Error" value from "Continue" to
> "Start Next thread loop"
>
> This looks reasonable
>

Thanks,
It will be great to discuss about it to find the best solution (my previous
PR have been rollback but now we have time to discuss it before 3.1 release)



>
> Antonio>For example, to "HTTP Cache Manager", I think, "Clear cache each
> Antonio>Iteration?" must be true and not false like actually
>
> Why's that?
> Do you clear browser caches each time before opening www.google.com?
>


No ;-)

But in a load test, usually I don't have enough user dataset (login,
password) to simulate all the users. The workaround is to use less
login/password than "real users" but to use it more time and clear cache
each time.


With my customers the 2 big challenges are :
mock third party
have a useful and complete dataset

Another benefit to clear the cache each time is easiest to analyze the
results. If the cache is not cleared I have two types of population and
need to separate it to analyze them (e.g. 2 threads group, one with cache
and another without cache)


But if you told me that it's a particular use case, no problem keeping the
default value

Have we got statistics (from stackoverflow...) how JMeter is used?
It will help to make the good choices


Thank
Antonio


>
> Vladimir
>

Re: Default value for parameters

[Vladimir Sitnikov <si...@gmail.com>]

Antonio>"Action to be taken after a Sample Error" value from "Continue" to
"Start Next thread loop"

This looks reasonable

Antonio>For example, to "HTTP Cache Manager", I think, "Clear cache each
Antonio>Iteration?" must be true and not false like actually

Why's that?
Do you clear browser caches each time before opening www.google.com?

Vladimir