You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@jmeter.apache.org by bu...@apache.org on 2016/03/09 13:19:38 UTC
[Bug 59152] New: Change default Thread Group -> Action to be taken
after a Sample Error value from "Continue" to "Start Next thread loop"
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
Bug ID: 59152
Summary: Change default Thread Group -> Action to be taken
after a Sample Error value from "Continue" to "Start
Next thread loop"
Product: JMeter
Version: Nightly (Please specify date)
Hardware: PC
OS: Windows NT
Status: NEW
Severity: enhancement
Priority: P2
Component: Main
Assignee: issues@jmeter.apache.org
Reporter: ra0077@gmail.com
Hi,
This patch change default Thread Group -> Action to be taken after a Sample
Error value from "Continue" to "Start Next thread loop"
I think it's better to start next thread loop instead of continuing without
take into account the error because continue if there is an error makes no
sense (it's not realistic)
Antonio
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 59152] Change default Thread Group -> Action to be taken after
a Sample Error value from "Continue" to "Start Next thread loop"
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
--- Comment #1 from Antonio Gomes Rodrigues <ra...@gmail.com> ---
PR 161 commited
Antonio
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 59152] Thread Group: Change "Action to be taken after a Sample
Error" value from "Continue" to "Start Next thread loop"
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
Philippe Mouawad <p....@ubik-ingenierie.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |FIXED
Status|NEW |RESOLVED
--- Comment #2 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Date: Wed Mar 9 21:09:24 2016
New Revision: 1734313
URL: http://svn.apache.org/viewvc?rev=1734313&view=rev
Log:
Bug 59152 - Thread Group: Change "Action to be taken after a Sample Error"
value from "Continue" to "Start Next thread loop"
#resolve #161
https://github.com/apache/jmeter/pull/161
Bugzilla Id: 59152
Modified:
jmeter/trunk/bin/templates/recording-with-think-time.jmx
jmeter/trunk/bin/templates/recording.jmx
jmeter/trunk/src/core/org/apache/jmeter/threads/gui/AbstractThreadGroupGui.java
jmeter/trunk/xdocs/changes.xml
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 59152] Thread Group: Change "Action to be taken after a Sample
Error" value from "Continue" to "Start Next thread loop"
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
Sebb <se...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |WONTFIX
Status|REOPENED |RESOLVED
--- Comment #9 from Sebb <se...@apache.org> ---
Reverted:
URL: http://svn.apache.org/viewvc?rev=1742333&view=rev
Log:
Revert r1734313 so default action remains as 'Continue'
Bugzilla Id: 59152
Modified:
jmeter/trunk/bin/templates/recording-with-think-time.jmx
jmeter/trunk/bin/templates/recording.jmx
jmeter/trunk/src/core/org/apache/jmeter/threads/gui/AbstractThreadGroupGui.java
jmeter/trunk/xdocs/changes.xml
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 59152] Thread Group: Change "Action to be taken after a Sample
Error" value from "Continue" to "Start Next thread loop"
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
--- Comment #6 from Milamber <mi...@apache.org> ---
Hello,
I think that the revert will be a good thing. I made some load tests with this
new default option. My experience is this new default mask a bad load test.
I explain:
If you are a plan test like this:
1/ Login Form
2/ Login (id/pass)
3/ Home
4/ Search
5/ Results
etc.
With this default option, if the 2/Login failed (target server failed to login
randomly), you see only some Errors (during the load test) on 2/Login, the
other pages have 0 error (because return to 1/ after error)
After, when you show the graph results : all is good
If you show results in the summary listener, you see results like this:
1/ Login form : 2000 samples
2/ Login (id/pass) : 2000 samples with 25% errors
3/ Home : 1500 samples with 0 errors
4/ Search : 1500 samples with 0 errors
etc.
It's very easy to conclude that this load test is successful (only 25% errors
on 1 page), but in reality, this is a bad load test, because my target load is
reduce to 25%, and the target server have been tested with only at 75% of the
load.
I prefer the old option, which force me to stop a load test if the errors
increase on all pages after a error on the login page.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 59152] Thread Group: Change "Action to be taken after a Sample
Error" value from "Continue" to "Start Next thread loop"
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
Sebb <se...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|FIXED |---
--- Comment #3 from Sebb <se...@apache.org> ---
I've just fallen foul of this change.
I set up a test which deliberately had a couple of failing samples, but only
one of them was run.
It took me quite a while to work out what had happened.
I think this change is going to be a nuisance for others as well and should be
reverted.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 59152] Thread Group: Change "Action to be taken after a Sample
Error" value from "Continue" to "Start Next thread loop"
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
Philippe Mouawad <p....@ubik-ingenierie.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Summary|Change default Thread Group |Thread Group: Change
|-> Action to be taken after |"Action to be taken after a
|a Sample Error value from |Sample Error" value from
|"Continue" to "Start Next |"Continue" to "Start Next
|thread loop" |thread loop"
OS|Windows NT |All
Hardware|PC |All
Version|Nightly (Please specify |2.13
|date) |
CC| |p.mouawad@ubik-ingenierie.c
| |om
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 59152] Thread Group: Change "Action to be taken after a Sample
Error" value from "Continue" to "Start Next thread loop"
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
--- Comment #7 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Hi,
Ok for me.
I don't want to delay release for this.
Who reverts it ?
Thanks
Regards
Philippe
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 59152] Thread Group: Change "Action to be taken after a Sample
Error" value from "Continue" to "Start Next thread loop"
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
--- Comment #10 from Vladimir Sitnikov <si...@gmail.com> ---
Milamber>It's very easy to conclude that this load test is successful (only 25%
errors on 1 page), but in reality, this is a bad load test, because my target
load is reduce to 25%, and the target server have been tested with only at 75%
of the load.
Technically speaking, "validation" of a test report should include not only "%
of errors" validation, but "planned throughput vs actual throughput", "planned
response times vs actual response times".
If a test is hard to setup (e.g. lots of steps), then it might be better to
start new iteration to try one's best to achieve "throughput goal".
At the end of the day, real users do restart from scratch in case of failure.
Note: high failrate would be misleading, since it will show "consequence",
while it makes much more sense in knowing "root cause". In that case, it would
be good to restart after the first failure, and %error would show exactly the
step that failed.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 59152] Thread Group: Change "Action to be taken after a Sample
Error" value from "Continue" to "Start Next thread loop"
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
--- Comment #4 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
(In reply to Sebb from comment #3)
> I've just fallen foul of this change.
>
> I set up a test which deliberately had a couple of failing samples, but only
> one of them was run.
>
> It took me quite a while to work out what had happened.
>
> I think this change is going to be a nuisance for others as well and should
> be reverted.
I don't share this opinion and agree with initial motivations of this change.
In my experience (webapps testing), I always set the value to what it is after
the change.
Since change is mentioned in Incompatible changes, it is not a problem for me.
But I can be wrong.
What is the Use case of your plan ?
Thanks
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 59152] Thread Group: Change "Action to be taken after a Sample
Error" value from "Continue" to "Start Next thread loop"
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
--- Comment #5 from Sebb <se...@apache.org> ---
(In reply to Philippe Mouawad from comment #4)
> (In reply to Sebb from comment #3)
> > I've just fallen foul of this change.
> >
> > I set up a test which deliberately had a couple of failing samples, but only
> > one of them was run.
> >
> > It took me quite a while to work out what had happened.
> >
> > I think this change is going to be a nuisance for others as well and should
> > be reverted.
>
> I don't share this opinion and agree with initial motivations of this change.
> In my experience (webapps testing), I always set the value to what it is
> after the change.
>
> Since change is mentioned in Incompatible changes, it is not a problem for
> me.
It is one line amongst lots.
Even though I know it's there the consequences are not immediately obvious.
> But I can be wrong.
> What is the Use case of your plan ?
I was testing the changes for Content-Encoding and wanted to see how error
pages were handled.
I quite often want to use invalid URL or failed samples.
Also when one is developing a test plan, it's easy to make mistakes. With the
new default the first mistake stops the test run, so it can take a long time to
find all the mistakes.
I think this will penalise people who are new to JMeter or who use it
infrequently.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 59152] Thread Group: Change "Action to be taken after a Sample
Error" value from "Continue" to "Start Next thread loop"
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
--- Comment #8 from Sebb <se...@apache.org> ---
I will revert it
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 59152] Thread Group: Change "Action to be taken after a Sample
Error" value from "Continue" to "Start Next thread loop"
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=59152
--- Comment #11 from Sebb <se...@apache.org> ---
(In reply to Vladimir Sitnikov from comment #10)
> Milamber>It's very easy to conclude that this load test is successful (only
> 25% errors on 1 page), but in reality, this is a bad load test, because my
> target load is reduce to 25%, and the target server have been tested with
> only at 75% of the load.
>
> Technically speaking, "validation" of a test report should include not only
> "% of errors" validation, but "planned throughput vs actual throughput",
> "planned response times vs actual response times".
It's also important to know whether any tests were skipped.
>
> If a test is hard to setup (e.g. lots of steps), then it might be better to
> start new iteration to try one's best to achieve "throughput goal".
>
> At the end of the day, real users do restart from scratch in case of failure.
More likely, they will redo the step that failed.
After a couple of such failures they will go away and try another time.
>
> Note: high failrate would be misleading, since it will show "consequence",
> while it makes much more sense in knowing "root cause". In that case, it
> would be good to restart after the first failure, and %error would show
> exactly the step that failed.
That depends on the exact scenario.
Some failures may not be fatal to the loop or the test, e.g. image download
failure.
Only the test designer knows what the correct on error behaviour is, and that
will vary between plans and parts of test plans.
I think the only sensible default is Continue on Error.
Partly because that is the original setting, and partly to ensure that the full
test plan is exercised by default.
--
You are receiving this mail because:
You are the assignee for the bug.