You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Vincent Partington <vp...@xebia.com> on 2004/01/06 10:49:21 UTC
Response data written to results XML file
Hi,
(I see I'm having trouble getting my point across about response data
showing up in the results.jtl XML file, so I made a more comprehensive
test case and included a fix.)
I think there is a bug in JMeter where sometimes response data is
written to the results.jtl XML file when run in non-GUI mode. This
causes problems because it is not correclty encoded (the UTF-8 fix
Jordei implemented solves part of it, but still non-XML data shows up)
and it does not seem to serve a useful purpose.
Let me start at the beginning. Because I did not want to attach files to
a message to a mailing list, please download the file
<http://www.klomp.org/~vinny/jmeter_response_data_problem.zip> and unzip it.
Run the first test plan as such:
> jmeter -n -t response_data_problem_1.jmx
-l response_data_problem_1.jtl
-Jjmeter.save.saveservice.assertion_results=all
and have a look at response_data_problem_1.jtl (or just look at the one
I included). Because both assertions on the Jakarta logo
(/images/jakarta-logo.gif) succeeded, the results file reports the
successes of the assertions. To get those results the
"-Jjmeter.save.saveservice.assertion_results=all" bit is necessary.
Now run the second test plan:
> jmeter -n -t response_data_problem_2.jmx
-l response_data_problem_2.jtl
-Jjmeter.save.saveservice.assertion_results=all
and have a look at the corresponding results file. You can see all kinds
of binary gibberish appearing here. This is the failure message for the
second (response) assertion, even though it was the _first_ (size)
assertion that failed. It failed because I changed "=8584" to "!=8584".
Now contrast that with the the third test plan:
> jmeter -n -t response_data_problem_3.jmx
-l response_data_problem_3.jtl
-Jjmeter.save.saveservice.assertion_results=all
If you look at response_data_problem_3.jtl, you can see that the
response assertion succeeds here, but the size assertion fails.
This behaviour does not make sense to me. It is caused by these lines in
<http://cvs.apache.org/viewcvs.cgi/jakarta-jmeter/src/components/org/apache/jmeter/assertions/ResponseAssertion.java?view=markup>:
public AssertionResult getResult(SampleResult response)
{
AssertionResult result;
if (!response.isSuccessful())
{
result = new AssertionResult();
result.setError(true);
byte [] ba = response.getResponseData();
result.setFailureMessage(
ba == null ? "Unknown Error (responseData is empty)" : new
String(ba)
);
return result;
}
result = evaluateResponse(response);
return result;
}
For some reason a check is first made to see whether the sample was
succesful before the assertion is actually checked. If the sample was
not succesfull (either because there was an error executing the request
or because an earlier assertion failed), the failure message is set to
the response data and a failure is reported.
Now I have two questions:
1) Why is this check made?
2) If this check is necessary, why is the failure message not set to
something like "Cannot assert response when sample was not succesfull"?
Oh yeah, I promised a fix. My fix would be to remove the whole check and
have the method be similar like the one in
<http://cvs.apache.org/viewcvs.cgi/jakarta-jmeter/src/components/org/apache/jmeter/assertions/SizeAssertion.java?view=markup>:
public AssertionResult getResult(SampleResult response)
{
if (response.getResponseData() == null)
{
AssertionResult result = new AssertionResult();
result.setError(false);
result.setFailure(true);
result.setFailureMessage("Response was null");
return result;
}
return evaluateResponse(response);
}
Well, sorry the very long message but there seems to be no shorter way
to explain this and it is an annoying problem for me.
Regards, Vincent.
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Response data written to results XML file
Posted by Vincent Partington <vp...@xebia.com>.
Sebastian Bazley wrote:
> [...]
He, nice to get my point accross and thank you for commenting out those
offending lines. :)
> It seems to me that we need a little rethink on what the Assertion behaviour should be (*) ... meanwhile I will comment out the
> check, so that the Response Assertion behaves like the other assertions.
>
> (*) To consider:
> - should failed samples be subject to assertion checking at all?
I see no harm in it. The assertions will most likely fail, but if they
succeed that can be interesting in itself. I can even imagine wanting to
assert something about an failed sample.
> - or should assertions have a flag to say whether they should be applied to failed samples ?
That sounds better, but I wouldn't need it.
> - should the first assertion failure cause subsequent assertions to be skipped ?
> - or should an assertion have a flag to skip it if a previous assertion failed ?
Not by default. If you have multiple assertions, it is useful to see
which succeed and which fail. A flag could be useful, but again, I
wouldn't use it.
Regards, Vincent.
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Response data written to results XML file
Posted by Sebastian Bazley <Se...@london.sema.slb.com>.
----- Original Message -----
From: "Vincent Partington" <vp...@xebia.com>
To: <jm...@jakarta.apache.org>
Sent: Tuesday, January 06, 2004 9:49 AM
Subject: Response data written to results XML file
> Hi,
>
> (I see I'm having trouble getting my point across about response data
> showing up in the results.jtl XML file, so I made a more comprehensive
> test case and included a fix.)
>
Thanks for recapping and providing the test cases.
[I've been meaning to look at this for a while ...]
> I think there is a bug in JMeter where sometimes response data is
> written to the results.jtl XML file when run in non-GUI mode. This
> causes problems because it is not correclty encoded (the UTF-8 fix
> Jordei implemented solves part of it, but still non-XML data shows up)
> and it does not seem to serve a useful purpose.
I entirely agree. It also messes up the Tree and Assertion Listeners in GUI mode.
[...]
> For some reason a check is first made to see whether the sample was
> succesful before the assertion is actually checked. If the sample was
> not succesfull (either because there was an error executing the request
> or because an earlier assertion failed), the failure message is set to
> the response data and a failure is reported.
>
> Now I have two questions:
> 1) Why is this check made?
I suspect this was done to avoid runtime errors - if the sample failed, it does not make much sense to check the contents.
However, sample "failure" can be caused by either actual sample failure (e.g. 404 error) or a previous Assertion that failed - these
are rather different, but there is only a single flag.
> 2) If this check is necessary, why is the failure message not set to
> something like "Cannot assert response when sample was not succesfull"?
I'm not sure that the check is necessary - or if it is, perhaps it should be done at a higher level (in JMeterThread) - does it make
sense for Response Assertions to take account of sample "failure" when none of the others do?
[...]
Response Assertion treats null reponseData as an empty response, so this does not need to be checked.
==
It seems to me that we need a little rethink on what the Assertion behaviour should be (*) ... meanwhile I will comment out the
check, so that the Response Assertion behaves like the other assertions.
(*) To consider:
- should failed samples be subject to assertion checking at all?
- or should assertions have a flag to say whether they should be applied to failed samples ?
- should the first assertion failure cause subsequent assertions to be skipped ?
- or should an assertion have a flag to skip it if a previous assertion failed ?
S.
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org