You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jmeter.apache.org by Andrey Pokhilko <ap...@ya.ru> on 2018/08/08 08:12:53 UTC
Concerns about change in JTL writing
Hi,
I took the latest snapshot of JMeter and I'm shocked by the change in
CSV JTL writing. It is strange that it writes subsamples now with no
respect to "Save Sub Results" flag. I simply can't turn new behavior off.
It will break all the automated scripts written in the ecosystem. I run
the test with Aggregate Report, I see one number. If I save it into CSV
JTL and re-load it in Aggregate Report, I see completely different
picture. All my response times are counted twice now.
I believe something really breaking has happened, and it will hurt
users. I think CSV JTL writer has to respect the "Save Sub Results" flag
and not have that flag enabled by default. Otherwise, we break any
automation that were reading CSV JTL assuming only top-level samples are
written and math of average response times is consistent.
Or maybe I'm just doing something wrong?
--
Andrey Pokhilko
Re: Concerns about change in JTL writing
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello,
Find my answer inline below.
Regards
On Wed, Aug 8, 2018 at 11:16 AM Andrey Pokhilko <ap...@ya.ru> wrote:
> I tried to use JMeter only like regular user will. I did change through
> UI flag in listener, under "Configure" button. I believe the UI is the
> "final judge" that overrides any property, plus if I have in my JMX the
> flag set to false, it is guaranteed that it will be taken into account.
>
I made the test and the flag is taken into account with View Results Tree
for example and with AggregateReport wether in UI or non UI.
So can you clarify what test you made ?
>
> I looked into code and I see that property default is "true" from many
> years ago. This still leaves us with the problem that behavior of CSV
> JTL writing does change.
>
> If we change property default, it will change result writing for users
> of XML JTL, if we don't - it will change it for users of CSV JTL. JMeter
> use CSV JTL as default format, so I'd suggest to change
> "jmeter.save.saveservice.subresults" default to false as lesser from two
> evils (IMO).
>
The problem is that it is not ok for the HTML report of JMeter which is THE
official JMeter report nowadays and the most complete and modern one.
Other listeners being more for debugging or little tests.
For information , the change made is related to those bugs:
- Bug 62550
- Bug 60917
> --
>
> Andrey Pokhilko
>
> 08.08.2018 11:56, Philippe Mouawad пишет:
> > On Wednesday, August 8, 2018, Andrey Pokhilko <ap...@ya.ru> wrote:
> >
> >> Hi,
> >>
> >> I took the latest snapshot of JMeter and I'm shocked by the change in
> >> CSV JTL writing. It is strange that it writes subsamples now with no
> >> respect to "Save Sub Results" flag. I simply can't turn new behavior
> off.
> >
> > how did you do that ?
> > Through property
> > jmeter.save.saveservice.subresults
> > or in a different way , in the latter case how ?
> >
> >
> >> It will break all the automated scripts written in the ecosystem. I run
> >> the test with Aggregate Report, I see one number. If I save it into CSV
> >> JTL and re-load it in Aggregate Report, I see completely different
> >> picture. All my response times are counted twice now.
> >
> > That would be a bug that we need to fix.
> > The intention of the change was just to respect the
> > jmeter.save.saveservice.subresults
> > flag for csv also.
> >
> > It was only taken into account for xml.
> >
> >> I believe something really breaking has happened, and it will hurt
> >> users. I think CSV JTL writer has to respect the "Save Sub Results" flag
> >> and not have that flag enabled by default. Otherwise, we break any
> >> automation that were reading CSV JTL assuming only top-level samples are
> >> written and math of average response times is consistent.
> >>
> >> Or maybe I'm just doing something wrong?
> >>
> >> --
> >> Andrey Pokhilko
> >>
> >>
>
>
--
Cordialement.
Philippe Mouawad.
Re: Concerns about change in JTL writing
Posted by Philippe Mouawad <ph...@gmail.com>.
On Wednesday, August 8, 2018, Andrey Pokhilko <ap...@ya.ru> wrote:
> I tried to use JMeter only like regular user will. I did change through
> UI flag in listener, under "Configure" button. I believe the UI is the
> "final judge" that overrides any property, plus if I have in my JMX the
> flag set to false, it is guaranteed that it will be taken into account.
>
> I looked into code and I see that property default is "true" from many
> years ago. This still leaves us with the problem that behavior of CSV
> JTL writing does change.
>
> If we change property default, it will change result writing for users
> of XML JTL, if we don't - it will change it for users of CSV JTL. JMeter
> use CSV JTL as default format, so I'd suggest to change
> "jmeter.save.saveservice.subresults" default to false as lesser from two
> evils (IMO).
I ‘ll look deeper into it when I am back from holidays unless somebody
wants to.
Related enhancement is:
Bug 62470
>
> --
>
> Andrey Pokhilko
>
> 08.08.2018 11:56, Philippe Mouawad пишет:
> > On Wednesday, August 8, 2018, Andrey Pokhilko <ap...@ya.ru> wrote:
> >
> >> Hi,
> >>
> >> I took the latest snapshot of JMeter and I'm shocked by the change in
> >> CSV JTL writing. It is strange that it writes subsamples now with no
> >> respect to "Save Sub Results" flag. I simply can't turn new behavior
> off.
> >
> > how did you do that ?
> > Through property
> > jmeter.save.saveservice.subresults
> > or in a different way , in the latter case how ?
> >
> >
> >> It will break all the automated scripts written in the ecosystem. I run
> >> the test with Aggregate Report, I see one number. If I save it into CSV
> >> JTL and re-load it in Aggregate Report, I see completely different
> >> picture. All my response times are counted twice now.
> >
> > That would be a bug that we need to fix.
> > The intention of the change was just to respect the
> > jmeter.save.saveservice.subresults
> > flag for csv also.
> >
> > It was only taken into account for xml.
> >
> >> I believe something really breaking has happened, and it will hurt
> >> users. I think CSV JTL writer has to respect the "Save Sub Results" flag
> >> and not have that flag enabled by default. Otherwise, we break any
> >> automation that were reading CSV JTL assuming only top-level samples are
> >> written and math of average response times is consistent.
> >>
> >> Or maybe I'm just doing something wrong?
> >>
> >> --
> >> Andrey Pokhilko
> >>
> >>
>
>
--
Cordialement.
Philippe Mouawad.
Re: Concerns about change in JTL writing
Posted by Andrey Pokhilko <ap...@ya.ru>.
I tried to use JMeter only like regular user will. I did change through
UI flag in listener, under "Configure" button. I believe the UI is the
"final judge" that overrides any property, plus if I have in my JMX the
flag set to false, it is guaranteed that it will be taken into account.
I looked into code and I see that property default is "true" from many
years ago. This still leaves us with the problem that behavior of CSV
JTL writing does change.
If we change property default, it will change result writing for users
of XML JTL, if we don't - it will change it for users of CSV JTL. JMeter
use CSV JTL as default format, so I'd suggest to change
"jmeter.save.saveservice.subresults" default to false as lesser from two
evils (IMO).
--
Andrey Pokhilko
08.08.2018 11:56, Philippe Mouawad пишет:
> On Wednesday, August 8, 2018, Andrey Pokhilko <ap...@ya.ru> wrote:
>
>> Hi,
>>
>> I took the latest snapshot of JMeter and I'm shocked by the change in
>> CSV JTL writing. It is strange that it writes subsamples now with no
>> respect to "Save Sub Results" flag. I simply can't turn new behavior off.
>
> how did you do that ?
> Through property
> jmeter.save.saveservice.subresults
> or in a different way , in the latter case how ?
>
>
>> It will break all the automated scripts written in the ecosystem. I run
>> the test with Aggregate Report, I see one number. If I save it into CSV
>> JTL and re-load it in Aggregate Report, I see completely different
>> picture. All my response times are counted twice now.
>
> That would be a bug that we need to fix.
> The intention of the change was just to respect the
> jmeter.save.saveservice.subresults
> flag for csv also.
>
> It was only taken into account for xml.
>
>> I believe something really breaking has happened, and it will hurt
>> users. I think CSV JTL writer has to respect the "Save Sub Results" flag
>> and not have that flag enabled by default. Otherwise, we break any
>> automation that were reading CSV JTL assuming only top-level samples are
>> written and math of average response times is consistent.
>>
>> Or maybe I'm just doing something wrong?
>>
>> --
>> Andrey Pokhilko
>>
>>
Re: Concerns about change in JTL writing
Posted by Philippe Mouawad <ph...@gmail.com>.
On Wednesday, August 8, 2018, Andrey Pokhilko <ap...@ya.ru> wrote:
> Hi,
>
> I took the latest snapshot of JMeter and I'm shocked by the change in
> CSV JTL writing. It is strange that it writes subsamples now with no
> respect to "Save Sub Results" flag. I simply can't turn new behavior off.
how did you do that ?
Through property
jmeter.save.saveservice.subresults
or in a different way , in the latter case how ?
> It will break all the automated scripts written in the ecosystem. I run
> the test with Aggregate Report, I see one number. If I save it into CSV
> JTL and re-load it in Aggregate Report, I see completely different
> picture. All my response times are counted twice now.
That would be a bug that we need to fix.
The intention of the change was just to respect the
jmeter.save.saveservice.subresults
flag for csv also.
It was only taken into account for xml.
>
> I believe something really breaking has happened, and it will hurt
> users. I think CSV JTL writer has to respect the "Save Sub Results" flag
> and not have that flag enabled by default. Otherwise, we break any
> automation that were reading CSV JTL assuming only top-level samples are
> written and math of average response times is consistent.
>
> Or maybe I'm just doing something wrong?
>
> --
> Andrey Pokhilko
>
>
--
Cordialement.
Philippe Mouawad.