You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jmeter.apache.org by Philippe Mouawad <p....@ubik-ingenierie.com> on 2019/01/22 13:02:31 UTC
Pass / Fail criteria in JMeter
Hello,
I am planning to add to next version of JMeter Maven Plugin a pass/fail
criteria based on the JSON file that JMeter 5.1 generates at end of a load
test which contains summary of results:
- https://github.com/jmeter-maven-plugin/jmeter-maven-plugin/issues/303
But thinking about it, I think it could also be a nice addition to JMeter
core where JMeter would return a code based on evaluating those criterions.
Regards
Philippe M.
@philmdot
Re: Pass / Fail criteria in JMeter
Posted by Philippe Mouawad <ph...@gmail.com>.
On Tue, Jan 29, 2019 at 11:43 AM Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:
> Philippe> But while it was rather straightforward to implement through
> Maven, it's a
> Philippe> bit more complex in core JMeter.
>
> I guess the most challenging part is to incorporate "Non Functional
> Requirements" to JMeter UI.
>
> For instance:
> 1) Should it be a separate UI or should each sampler have "NFR" section?
>
No convictions.
For simplicity, I thought about a dedicated config element "SLA Rules":
+ One place for all
- But desynch can happen if you rename transactions / sampler and forget to
update
If we do as you propose, no desync but rules are spread in the whole test
plan, we need to consider also Transaction Controller.
> 2) Should there be a warning in case NFR did not fire (e.g. user
> specified "ligon.pct90 < 1sec" and actual transaction has name of
> "login")?
>
Yes, at least in logs
> 3) Should it warn for conflicting requirements? (e.g. *.pct90<1sec
> + login.pct90<5sec)
>
I would say, login.pct90 is an override of general rule.
But I didn't dig in implementation details yet.
5) Should it allow for rule inherinance? e.g.
> "simple.pages.pct90<1sec, login is a "simple page".
> Someone please stop me :)
>
STOP ! :-)
Let's start and work incrementally
>
> Vladimir
>
--
Cordialement.
Philippe Mouawad.
Re: Pass / Fail criteria in JMeter
Posted by Vladimir Sitnikov <si...@gmail.com>.
Philippe> But while it was rather straightforward to implement through
Maven, it's a
Philippe> bit more complex in core JMeter.
I guess the most challenging part is to incorporate "Non Functional
Requirements" to JMeter UI.
For instance:
1) Should it be a separate UI or should each sampler have "NFR" section?
2) Should there be a warning in case NFR did not fire (e.g. user
specified "ligon.pct90 < 1sec" and actual transaction has name of
"login")?
3) Should it warn for conflicting requirements? (e.g. *.pct90<1sec
+ login.pct90<5sec)
5) Should it allow for rule inherinance? e.g.
"simple.pages.pct90<1sec, login is a "simple page".
Someone please stop me :)
Vladimir
Re: Pass / Fail criteria in JMeter
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello Paulo,
In fact the message intent is to ask whether this feature is not best
suited in core JMeter instead of jmeter-maven-plugin.
So I think if there is a consensus on this, the feature would be in core.
But while it was rather straightforward to implement through Maven, it's a
bit more complex in core JMeter.
Regards
On Tue, Jan 29, 2019 at 10:54 AM Paulo Maia Borges <pm...@temenos.com>
wrote:
> I also started a reply when I first saw this mail-chain, saying:
>
> "It's a really good idea, in fact I have to implement that same «feature»
> manually for a recent automated-testing project I did." Etc.
>
>
> But then I noticed that this was a functionality specific for Maven, which
> I don't use nor know much about, so I refrained from sending the email...
> So, in theory, it looks to me like a very useful functionality but the
> truth is I don't know anything about the likes of Maven or build-tools in
> general (*).
>
>
> * the "automated-testing" I've setup is merely a command-line script
> invoking JMeter (with parameters), triggered by Jenkins. In reality it
> doesn't integrate with any build tool including Jenkins itself, it's merely
> a script that handles everything on its own.
>
>
>
>
> Regards,
> Paulo Augusto Maia Borges
> WWVzLCBJJ20gZ2Vla3kuIEJ1dCBzbyBhcmUgeW91IQ==
>
> -----Original Message-----
> ...
>
> The information in this e-mail and any attachments is confidential and may
> be legally privileged. It is intended solely for the addressee or
> addressees. Any use or disclosure of the contents of this
> e-mail/attachments by a not intended recipient is unauthorized and may be
> unlawful. If you have received this e-mail in error please notify the
> sender. Please note that any views or opinions presented in this e-mail are
> solely those of the author and do not necessarily represent those of
> TEMENOS. We recommend that you check this e-mail and any attachments
> against viruses. TEMENOS accepts no liability for any damage caused by any
> malicious code or virus transmitted by this e-mail.
>
--
Cordialement.
Philippe Mouawad.
RE: Pass / Fail criteria in JMeter
Posted by Paulo Maia Borges <pm...@temenos.com>.
I also started a reply when I first saw this mail-chain, saying:
"It's a really good idea, in fact I have to implement that same «feature» manually for a recent automated-testing project I did." Etc.
But then I noticed that this was a functionality specific for Maven, which I don't use nor know much about, so I refrained from sending the email... So, in theory, it looks to me like a very useful functionality but the truth is I don't know anything about the likes of Maven or build-tools in general (*).
* the "automated-testing" I've setup is merely a command-line script invoking JMeter (with parameters), triggered by Jenkins. In reality it doesn't integrate with any build tool including Jenkins itself, it's merely a script that handles everything on its own.
Regards,
Paulo Augusto Maia Borges
WWVzLCBJJ20gZ2Vla3kuIEJ1dCBzbyBhcmUgeW91IQ==
-----Original Message-----
...
The information in this e-mail and any attachments is confidential and may be legally privileged. It is intended solely for the addressee or addressees. Any use or disclosure of the contents of this e-mail/attachments by a not intended recipient is unauthorized and may be unlawful. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of TEMENOS. We recommend that you check this e-mail and any attachments against viruses. TEMENOS accepts no liability for any damage caused by any malicious code or virus transmitted by this e-mail.
Re: Pass / Fail criteria in JMeter
Posted by Vladimir Sitnikov <si...@gmail.com>.
Philippe> Any feedback ?
Frankly speaking I don't know what to say.
When I perform load tests I use the same metrics that you suggest:
transaction duration and failure rate.
I'm sure it would be really great if "result validation" was in core.
Vladimir
Re: Pass / Fail criteria in JMeter
Posted by Philippe Mouawad <p....@ubik-ingenierie.com>.
Hello,
Any feedback ?
Thanks
On Tue, Jan 22, 2019 at 2:02 PM Philippe Mouawad <
p.mouawad@ubik-ingenierie.com> wrote:
> Hello,
>
> I am planning to add to next version of JMeter Maven Plugin a pass/fail
> criteria based on the JSON file that JMeter 5.1 generates at end of a load
> test which contains summary of results:
>
> - https://github.com/jmeter-maven-plugin/jmeter-maven-plugin/issues/303
>
>
> But thinking about it, I think it could also be a nice addition to JMeter
> core where JMeter would return a code based on evaluating those criterions.
>
> Regards
> Philippe M.
> @philmdot
>
>
--
[image: logo Ubik Ingenierie] <https://www.ubik-ingenierie.com> Philippe
Mouawad
Senior Performance Expert
320914981 <+33320914981> | p.mouawad@ubik-ingenierie.com
[image: ubik-ingenierie.com] ubik-ingenierie.com
<https://www.ubik-ingenierie.com> | [image: 03.20.91.49.81] 03.20.91.49.81
<+33320914981> | [image: 23 rue du chemin de fer , 59100 , Roubaix] 23 rue
du chemin de fer, 59100, Roubaix
<https://www.openstreetmap.org/#map=18/50.69454/3.16455>