Test plan warnings

[Vladimir Sitnikov <si...@gmail.com>]

Hi,

As far as I understand, JMeter UI lacks feedback for "error prone"
configurations. It would be so much better if JMeter would have "spell
checker" integrated, so it warned on "bad test plan patterns".

Was there a discussion/plan to implement that?
I'm sure lack of that checker makes it very hard to integrate new features
since almost every new feature can be misused, so "decline a feature" is
basically the only way to protect user from misuse.

For instance,
1) A proper way to place inter-page timers is to wrap the timer into some
kind of "test action". So, if user has "page, timer, page, timer" kind of
test plan, that deserves a warning "you might probably want to wrap
timers..."
2) If user places constant throughput timer at the very top of the test
plan, that is probably wrong. CTT should likely to be wrapped into some
sampler.
3) If user references non-existing variable, then you don't know that until
you start the test. For instance, JMeter know which variable is produced by
regexp extractor, so it can check if user uses that or not.
For instance, there might be a warning like "unused regexp extractor", etc.
4) "Adjust timers" feature. Even though it might be useful, it might easily
be misused. I think having some warnings would improve UX for this
particular feature as well.

Vladimir

Re: Test plan warnings

[Antonio Gomes Rodrigues <ra...@gmail.com>]

Hi

I think it could be usefull for performance warning

Antonio

2016-08-31 11:20 GMT+02:00 Milamber <mi...@apache.org>:

>
>
> On 31/08/2016 10:11, sebb wrote:
>
>> I think it's far more important to ensure that the features which are
>> added to JMeter are genuinely useful and well documented.
>>
>
> +1
>
>
>

Re: Test plan warnings

[Milamber <mi...@apache.org>]

On 31/08/2016 10:11, sebb wrote:
> I think it's far more important to ensure that the features which are
> added to JMeter are genuinely useful and well documented.

+1

Re: Test plan warnings

[Philippe Mouawad <ph...@gmail.com>]

I find the idea nice.

the reporting could be similar to eclipse problems view at the bottom.
This way if user is not interested he can remove the view. Or we can pake
the view appear only on demand.

I think it could be built in a way that makes checks pluggable.



On Wednesday, August 31, 2016, Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:

> sebb>I think it's far more important to ensure that the features ... and
> well documented.
>
> Note: there are certain peculiarities that are very unexpected by the
> typical user.
> Of course doing some RTFM solves some problems, however I still think
> having a spell checker that allows false positives is way better than
> having no spell checker and forcing everybody to look up dictionary for
> each and every case.
>
> Well, I've converted several engineers from LR to JMeter (they seem to be
> happy with that), however "proper timer placement" did bite them even
> though I did highlight the issue.
>
> sebb>It's not possible in general to detect whether a variable is used or
> not.
>
> Well. As in typical IDEs there might be
> @SupressWarning("I_know_I_use_this_variable_in_extremely_
> obscure_way_so_just_believe_it_exists").
> Note: I'm sure lots of typical regexp-extractor generated variables are
> predictable.
>
> Vladimir
>


-- 
Cordialement.
Philippe Mouawad.

Re: Test plan warnings

[Vladimir Sitnikov <si...@gmail.com>]

sebb>I think it's far more important to ensure that the features ... and
well documented.

Note: there are certain peculiarities that are very unexpected by the
typical user.
Of course doing some RTFM solves some problems, however I still think
having a spell checker that allows false positives is way better than
having no spell checker and forcing everybody to look up dictionary for
each and every case.

Well, I've converted several engineers from LR to JMeter (they seem to be
happy with that), however "proper timer placement" did bite them even
though I did highlight the issue.

sebb>It's not possible in general to detect whether a variable is used or
not.

Well. As in typical IDEs there might be
@SupressWarning("I_know_I_use_this_variable_in_extremely_obscure_way_so_just_believe_it_exists").
Note: I'm sure lots of typical regexp-extractor generated variables are
predictable.

Vladimir

Re: Test plan warnings

[sebb <se...@gmail.com>]

On 31 August 2016 at 09:23, Vladimir Sitnikov
<si...@gmail.com> wrote:
> Hi,
>
> As far as I understand, JMeter UI lacks feedback for "error prone"
> configurations. It would be so much better if JMeter would have "spell
> checker" integrated, so it warned on "bad test plan patterns".
>

Not strictly true.

The GUI prevents certain test elements from being used at some parts
of the test plan.

> Was there a discussion/plan to implement that?
> I'm sure lack of that checker makes it very hard to integrate new features
> since almost every new feature can be misused, so "decline a feature" is
> basically the only way to protect user from misuse.

I disagree that that the lack of a checker makes it hard to integrate
new features.
The problem is that there are lots of potential new features and it's
important to ensure that the costs of a new feature outweigh the
benefits.

I'm not aware of any feature that has been rejected on the basis that
it can be misused.
However several have been rejected on the basis that the potential use
cases are not common enough.

> For instance,
> 1) A proper way to place inter-page timers is to wrap the timer into some
> kind of "test action". So, if user has "page, timer, page, timer" kind of
> test plan, that deserves a warning "you might probably want to wrap
> timers..."
>
> 2) If user places constant throughput timer at the very top of the test
> plan, that is probably wrong. CTT should likely to be wrapped into some
> sampler.
>
> 3) If user references non-existing variable, then you don't know that until
> you start the test. For instance, JMeter know which variable is produced by
> regexp extractor, so it can check if user uses that or not.

It's not possible in general to detect whether a variable is used or not.
For example, it may be used in a script or using $__var() indirection,
so the variable name may not be known until run-time.

> For instance, there might be a warning like "unused regexp extractor", etc.
> 4) "Adjust timers" feature. Even though it might be useful, it might easily
> be misused. I think having some warnings would improve UX for this
> particular feature as well.

 I think it's a nice idea to try an warn about unexpected test element
usage, but implementation is going to be very tricky if one is going
to avoid false positives.

I think it's far more important to ensure that the features which are
added to JMeter are genuinely useful and well documented.

> Vladimir

Re: Test plan warnings

[Philippe Mouawad <ph...@gmail.com>]

On Wednesday, August 31, 2016, Epp, Jeremiah W (Contractor) <je...@cas.org>
wrote:

> > -----Original Message-----
> > From: Vladimir Sitnikov [mailto:sitnikov.vladimir@gmail.com
> <javascript:;>]
> > Sent: Wednesday, August 31, 2016 4:24 AM
> > To: dev@jmeter.apache.org <javascript:;>
> > Subject: Test plan warnings
> >
> > 1) A proper way to place inter-page timers is to wrap the timer into some
> > kind of "test action". So, if user has "page, timer, page, timer" kind of
> > test plan, that deserves a warning "you might probably want to wrap
> > timers..."
>
> Having had to explain this to people repeatedly, I can confirm the scoping
> rules for timers are not easy for many users to understand.
>
> But is this really the solution?
>
> Currently, the TestAction lets one set a constant pause using an internal
> implementation, but the comments in TestAction.java seem to indicate that
> inserting a pause isn't really its "intended" purpose, regardless of how it
> tends to be used in practice.
>
> My view is there should be an obvious "Think Time Sampler" that removes the
> need for end users to set up the wrapper manually


+1

>
> (which is really silly,

when you think about it).  Set it up so you can bind any timer to it
> directly; in the GUI make it a dropdown selector with every available
> AbstractTestElement-implementing Timer.


Maybe just a simple variation around a time is enough.




> (Also, I do still think there's value in timers discriminating on whether
> they apply to Samplers, Controllers, or any TestElement.)
>
> Regards,
> Wyatt
>
> Confidentiality Notice: This electronic message transmission, including
> any attachment(s), may contain confidential, proprietary, or privileged
> information from Chemical Abstracts Service ("CAS"), a division of the
> American Chemical Society ("ACS"). If you have received this transmission
> in error, be advised that any disclosure, copying, distribution, or use of
> the contents of this information is strictly prohibited. Please destroy all
> copies of the message and contact the sender immediately by either replying
> to this message or calling 614-447-3600.
>
>

-- 
Cordialement.
Philippe Mouawad.

RE: Test plan warnings

["Epp, Jeremiah W (Contractor)" <je...@cas.org>]

> -----Original Message-----
> From: Vladimir Sitnikov [mailto:sitnikov.vladimir@gmail.com]
> Sent: Wednesday, August 31, 2016 4:24 AM
> To: dev@jmeter.apache.org
> Subject: Test plan warnings
> 
> 1) A proper way to place inter-page timers is to wrap the timer into some
> kind of "test action". So, if user has "page, timer, page, timer" kind of
> test plan, that deserves a warning "you might probably want to wrap
> timers..."

Having had to explain this to people repeatedly, I can confirm the scoping
rules for timers are not easy for many users to understand.

But is this really the solution?

Currently, the TestAction lets one set a constant pause using an internal
implementation, but the comments in TestAction.java seem to indicate that
inserting a pause isn't really its "intended" purpose, regardless of how it
tends to be used in practice.  

My view is there should be an obvious "Think Time Sampler" that removes the
need for end users to set up the wrapper manually (which is really silly,
when you think about it).  Set it up so you can bind any timer to it
directly; in the GUI make it a dropdown selector with every available
AbstractTestElement-implementing Timer.

(Also, I do still think there's value in timers discriminating on whether
they apply to Samplers, Controllers, or any TestElement.)

Regards,
Wyatt

Confidentiality Notice: This electronic message transmission, including any attachment(s), may contain confidential, proprietary, or privileged information from Chemical Abstracts Service ("CAS"), a division of the American Chemical Society ("ACS"). If you have received this transmission in error, be advised that any disclosure, copying, distribution, or use of the contents of this information is strictly prohibited. Please destroy all copies of the message and contact the sender immediately by either replying to this message or calling 614-447-3600.