You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by James Fitzsimons <Ja...@intergen.co.nz> on 2012/11/15 05:13:26 UTC
Duration Assertion and Retrieve All Embedded resources not working
as expected
Hi all,
We are building a test plan where we are attempting to use a duration assertion to validate that responses are completing with 2000ms. This works well without the "Retrieve All Embedded resources from HTML files" option in the HTTP Request Defaults. However if we enable this option then we get a very large number of failures despite browsing manually shows pages loading in less than a second. We have included a cache manager to simulate browser caching behaviour of static resources.
In the duration assertion we have the "Apply to:" option set to "Main sample only". From reading the documentation it seems that using this setting the duration assertion should only be measuring the response time of the primary resource (html page) being requested and not all the embedded resources as well. However, this doesn't match up with what we are seeing in practice. E.g. the main sample might take only 200ms (looking at browser tools and IIS logs) but the 2000ms duration assertion still fails.
Is this a bug, a misunderstanding on our part, or have we just not configured the test plan correctly?
Any thoughts / advice will be much appreciated!
Cheers,
James
James Fitzsimons
Solution Delivery
INTERGEN: LEVEL 7, INTERGEN HOUSE, 126 LAMBTON QUAY. PO BOX 5428, WELLINGTON 6011, NEW ZEALAND.
TEL +64 4 472 2021 FAX +64 4 472 2027 MOB +64 27 273 3328 WWW.INTERGEN.CO.NZ<http://www.intergen.co.nz/>
To catch the latest...
Follow us on Twitter: www.twitter.com/TeamIntergen<http://www.twitter.com/TeamIntergen>
Read our blog: www.intergen.co.nz/blog<http://www.intergen.co.nz/blog>
Find us on LinkedIn: www.linkedin.com/company/Intergen<http://www.linkedin.com/company/Intergen>
2012 Microsoft Country Partner of the Year - New Zealand
________________________________
This electronic message together with any attachments is confidential. If you are not the intended recipient: (i) do not copy, disclose or use the contents in any way (ii) please let us know by return e-mail immediately and then destroy the message. Intergen is not responsible for any changes made to this message and/or any attachments after sending by Intergen.
Re: Duration Assertion and Retrieve All Embedded resources not
working as expected
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello sebb,
Do you have the answer to my question ?
Regards
Philippe
On Sun, Jan 6, 2013 at 2:28 PM, Philippe Mouawad <philippe.mouawad@gmail.com
> wrote:
> Then when Main Sample is checked , don't you think we should transmit the
> first subresult instead of the parent for Sampler that have many subresults
> .
> Of course this is to do if all these kinds of Sampler work the same way:
>
> - Put the Main Sampler result as first SubResults
> - And children as others
>
> Do you know if they all works this way ?
>
>
> Thanks
>
>
>
> On Sat, Jan 5, 2013 at 11:19 PM, sebb <se...@gmail.com> wrote:
>
>> On 5 January 2013 21:59, Philippe Mouawad <ph...@gmail.com>
>> wrote:
>> > Hello sebb,
>> >
>> > Working on
>> >
>> > - https://issues.apache.org/bugzilla/show_bug.cgi?id=54156
>> >
>> > which is related to this question below , there is something I don't
>> > understand.
>> >
>> > Maybe you have the answer so I check with you.
>> >
>> > What is exactly Main Sample for assertions ?
>> >
>> > It seems that in the case of HTTP SAmpler + Retrieve All Embedded
>> resources
>> > from HTML files checked, we have a Sampler where:
>> >
>> > - the first subresult is the main page
>> > - the others are the downloaded resources
>> >
>> > Do you consider this fine ?
>>
>> IIRC, the top-level sample is the overall download for the page, i.e.
>> it includes the times and sizes for initial result plus all embedded
>> resources.
>>
>> Not sure where that info can go if not as the top-level sample.
>>
>> We also want a sample result which contains the indvdidual resources;
>> these are currently added as sub-samples.
>>
>> I agree it's a bit odd, but what else could JMeter do?
>>
>> > Where is it documented ?
>>
>> AFAIK, it's not.
>>
>> > It seems strange to me, you can see the problem with attached test case
>> ,
>> > this is what this user is reporting.
>> >
>> > Do you think this should be fixed in JMeterThread#checkAssertions or
>> > somewhere else ?
>> >
>> >
>> > Regards
>> >
>> > Philippe
>> > On Thu, Nov 15, 2012 at 5:13 AM, James Fitzsimons <
>> > James.Fitzsimons@intergen.co.nz> wrote:
>> >
>> >> Hi all,
>> >>
>> >> We are building a test plan where we are attempting to use a duration
>> >> assertion to validate that responses are completing with 2000ms. This
>> works
>> >> well without the "Retrieve All Embedded resources from HTML files"
>> option
>> >> in the HTTP Request Defaults. However if we enable this option then we
>> get
>> >> a very large number of failures despite browsing manually shows pages
>> >> loading in less than a second. We have included a cache manager to
>> simulate
>> >> browser caching behaviour of static resources.
>> >>
>> >> In the duration assertion we have the "Apply to:" option set to "Main
>> >> sample only". From reading the documentation it seems that using this
>> >> setting the duration assertion should only be measuring the response
>> time
>> >> of the primary resource (html page) being requested and not all the
>> >> embedded resources as well. However, this doesn't match up with what
>> we are
>> >> seeing in practice. E.g. the main sample might take only 200ms
>> (looking at
>> >> browser tools and IIS logs) but the 2000ms duration assertion still
>> fails.
>> >>
>> >> Is this a bug, a misunderstanding on our part, or have we just not
>> >> configured the test plan correctly?
>> >>
>> >> Any thoughts / advice will be much appreciated!
>> >>
>> >> Cheers,
>> >> James
>> >>
>> >>
>> >> James Fitzsimons
>> >> Solution Delivery
>> >>
>> >> INTERGEN: LEVEL 7, INTERGEN HOUSE, 126 LAMBTON QUAY. PO BOX 5428,
>> >> WELLINGTON 6011, NEW ZEALAND.
>> >> TEL +64 4 472 2021 FAX +64 4 472 2027 MOB +64 27 273 3328
>> >> WWW.INTERGEN.CO.NZ<http://www.intergen.co.nz/>
>> >>
>> >>
>> >>
>> >> To catch the latest...
>> >> Follow us on Twitter: www.twitter.com/TeamIntergen<
>> >> http://www.twitter.com/TeamIntergen>
>> >> Read our blog: www.intergen.co.nz/blog<http://www.intergen.co.nz/blog>
>> >> Find us on LinkedIn: www.linkedin.com/company/Intergen<
>> >> http://www.linkedin.com/company/Intergen>
>> >>
>> >> 2012 Microsoft Country Partner of the Year - New Zealand
>> >> ________________________________
>> >> This electronic message together with any attachments is confidential.
>> If
>> >> you are not the intended recipient: (i) do not copy, disclose or use
>> the
>> >> contents in any way (ii) please let us know by return e-mail
>> immediately
>> >> and then destroy the message. Intergen is not responsible for any
>> changes
>> >> made to this message and/or any attachments after sending by Intergen.
>> >>
>> >>
>> >>
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>
--
Cordialement.
Philippe Mouawad.
Re: Duration Assertion and Retrieve All Embedded resources not
working as expected
Posted by sebb <se...@gmail.com>.
On 6 January 2013 13:28, Philippe Mouawad <ph...@gmail.com> wrote:
> Then when Main Sample is checked , don't you think we should transmit the
> first subresult instead of the parent for Sampler that have many subresults
Then the problem is that there is no way to check the aggregated parent sample.
But I agree that the current behaviour is somewhat counter-intuitive.
Currently I think we have:
Main: parent sample
Main + subsamples: parent sample + subsamples (i.e. main+subsamples)
Subsamples: main+subsamples
I think the Main+Subsamples is fairly useless; it effectively contains
the main sample twice (at least for the HTTP case).
What we probably need is:
Parent: parent sample (or only sample)
Main: 1st child (or only sample)
All samples: all children (or only sample)
Sub-samples: all but first child
If we kept the name Main, it would change meaning.
> Of course this is to do if all these kinds of Sampler work the same way:
>
> - Put the Main Sampler result as first SubResults
> - And children as others
>
> Do you know if they all works this way ?
No.
That needs to be checked before we make any changes, in case they make
things worse for non-HTTP samplers.
>
> Thanks
>
>
>
> On Sat, Jan 5, 2013 at 11:19 PM, sebb <se...@gmail.com> wrote:
>
>> On 5 January 2013 21:59, Philippe Mouawad <ph...@gmail.com>
>> wrote:
>> > Hello sebb,
>> >
>> > Working on
>> >
>> > - https://issues.apache.org/bugzilla/show_bug.cgi?id=54156
>> >
>> > which is related to this question below , there is something I don't
>> > understand.
>> >
>> > Maybe you have the answer so I check with you.
>> >
>> > What is exactly Main Sample for assertions ?
>> >
>> > It seems that in the case of HTTP SAmpler + Retrieve All Embedded
>> resources
>> > from HTML files checked, we have a Sampler where:
>> >
>> > - the first subresult is the main page
>> > - the others are the downloaded resources
>> >
>> > Do you consider this fine ?
>>
>> IIRC, the top-level sample is the overall download for the page, i.e.
>> it includes the times and sizes for initial result plus all embedded
>> resources.
>>
>> Not sure where that info can go if not as the top-level sample.
>>
>> We also want a sample result which contains the indvdidual resources;
>> these are currently added as sub-samples.
>>
>> I agree it's a bit odd, but what else could JMeter do?
>>
>> > Where is it documented ?
>>
>> AFAIK, it's not.
>>
>> > It seems strange to me, you can see the problem with attached test case ,
>> > this is what this user is reporting.
>> >
>> > Do you think this should be fixed in JMeterThread#checkAssertions or
>> > somewhere else ?
>> >
>> >
>> > Regards
>> >
>> > Philippe
>> > On Thu, Nov 15, 2012 at 5:13 AM, James Fitzsimons <
>> > James.Fitzsimons@intergen.co.nz> wrote:
>> >
>> >> Hi all,
>> >>
>> >> We are building a test plan where we are attempting to use a duration
>> >> assertion to validate that responses are completing with 2000ms. This
>> works
>> >> well without the "Retrieve All Embedded resources from HTML files"
>> option
>> >> in the HTTP Request Defaults. However if we enable this option then we
>> get
>> >> a very large number of failures despite browsing manually shows pages
>> >> loading in less than a second. We have included a cache manager to
>> simulate
>> >> browser caching behaviour of static resources.
>> >>
>> >> In the duration assertion we have the "Apply to:" option set to "Main
>> >> sample only". From reading the documentation it seems that using this
>> >> setting the duration assertion should only be measuring the response
>> time
>> >> of the primary resource (html page) being requested and not all the
>> >> embedded resources as well. However, this doesn't match up with what we
>> are
>> >> seeing in practice. E.g. the main sample might take only 200ms (looking
>> at
>> >> browser tools and IIS logs) but the 2000ms duration assertion still
>> fails.
>> >>
>> >> Is this a bug, a misunderstanding on our part, or have we just not
>> >> configured the test plan correctly?
>> >>
>> >> Any thoughts / advice will be much appreciated!
>> >>
>> >> Cheers,
>> >> James
>> >>
>> >>
>> >> James Fitzsimons
>> >> Solution Delivery
>> >>
>> >> INTERGEN: LEVEL 7, INTERGEN HOUSE, 126 LAMBTON QUAY. PO BOX 5428,
>> >> WELLINGTON 6011, NEW ZEALAND.
>> >> TEL +64 4 472 2021 FAX +64 4 472 2027 MOB +64 27 273 3328
>> >> WWW.INTERGEN.CO.NZ<http://www.intergen.co.nz/>
>> >>
>> >>
>> >>
>> >> To catch the latest...
>> >> Follow us on Twitter: www.twitter.com/TeamIntergen<
>> >> http://www.twitter.com/TeamIntergen>
>> >> Read our blog: www.intergen.co.nz/blog<http://www.intergen.co.nz/blog>
>> >> Find us on LinkedIn: www.linkedin.com/company/Intergen<
>> >> http://www.linkedin.com/company/Intergen>
>> >>
>> >> 2012 Microsoft Country Partner of the Year - New Zealand
>> >> ________________________________
>> >> This electronic message together with any attachments is confidential.
>> If
>> >> you are not the intended recipient: (i) do not copy, disclose or use the
>> >> contents in any way (ii) please let us know by return e-mail immediately
>> >> and then destroy the message. Intergen is not responsible for any
>> changes
>> >> made to this message and/or any attachments after sending by Intergen.
>> >>
>> >>
>> >>
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
Re: Duration Assertion and Retrieve All Embedded resources not
working as expected
Posted by Philippe Mouawad <ph...@gmail.com>.
Then when Main Sample is checked , don't you think we should transmit the
first subresult instead of the parent for Sampler that have many subresults
.
Of course this is to do if all these kinds of Sampler work the same way:
- Put the Main Sampler result as first SubResults
- And children as others
Do you know if they all works this way ?
Thanks
On Sat, Jan 5, 2013 at 11:19 PM, sebb <se...@gmail.com> wrote:
> On 5 January 2013 21:59, Philippe Mouawad <ph...@gmail.com>
> wrote:
> > Hello sebb,
> >
> > Working on
> >
> > - https://issues.apache.org/bugzilla/show_bug.cgi?id=54156
> >
> > which is related to this question below , there is something I don't
> > understand.
> >
> > Maybe you have the answer so I check with you.
> >
> > What is exactly Main Sample for assertions ?
> >
> > It seems that in the case of HTTP SAmpler + Retrieve All Embedded
> resources
> > from HTML files checked, we have a Sampler where:
> >
> > - the first subresult is the main page
> > - the others are the downloaded resources
> >
> > Do you consider this fine ?
>
> IIRC, the top-level sample is the overall download for the page, i.e.
> it includes the times and sizes for initial result plus all embedded
> resources.
>
> Not sure where that info can go if not as the top-level sample.
>
> We also want a sample result which contains the indvdidual resources;
> these are currently added as sub-samples.
>
> I agree it's a bit odd, but what else could JMeter do?
>
> > Where is it documented ?
>
> AFAIK, it's not.
>
> > It seems strange to me, you can see the problem with attached test case ,
> > this is what this user is reporting.
> >
> > Do you think this should be fixed in JMeterThread#checkAssertions or
> > somewhere else ?
> >
> >
> > Regards
> >
> > Philippe
> > On Thu, Nov 15, 2012 at 5:13 AM, James Fitzsimons <
> > James.Fitzsimons@intergen.co.nz> wrote:
> >
> >> Hi all,
> >>
> >> We are building a test plan where we are attempting to use a duration
> >> assertion to validate that responses are completing with 2000ms. This
> works
> >> well without the "Retrieve All Embedded resources from HTML files"
> option
> >> in the HTTP Request Defaults. However if we enable this option then we
> get
> >> a very large number of failures despite browsing manually shows pages
> >> loading in less than a second. We have included a cache manager to
> simulate
> >> browser caching behaviour of static resources.
> >>
> >> In the duration assertion we have the "Apply to:" option set to "Main
> >> sample only". From reading the documentation it seems that using this
> >> setting the duration assertion should only be measuring the response
> time
> >> of the primary resource (html page) being requested and not all the
> >> embedded resources as well. However, this doesn't match up with what we
> are
> >> seeing in practice. E.g. the main sample might take only 200ms (looking
> at
> >> browser tools and IIS logs) but the 2000ms duration assertion still
> fails.
> >>
> >> Is this a bug, a misunderstanding on our part, or have we just not
> >> configured the test plan correctly?
> >>
> >> Any thoughts / advice will be much appreciated!
> >>
> >> Cheers,
> >> James
> >>
> >>
> >> James Fitzsimons
> >> Solution Delivery
> >>
> >> INTERGEN: LEVEL 7, INTERGEN HOUSE, 126 LAMBTON QUAY. PO BOX 5428,
> >> WELLINGTON 6011, NEW ZEALAND.
> >> TEL +64 4 472 2021 FAX +64 4 472 2027 MOB +64 27 273 3328
> >> WWW.INTERGEN.CO.NZ<http://www.intergen.co.nz/>
> >>
> >>
> >>
> >> To catch the latest...
> >> Follow us on Twitter: www.twitter.com/TeamIntergen<
> >> http://www.twitter.com/TeamIntergen>
> >> Read our blog: www.intergen.co.nz/blog<http://www.intergen.co.nz/blog>
> >> Find us on LinkedIn: www.linkedin.com/company/Intergen<
> >> http://www.linkedin.com/company/Intergen>
> >>
> >> 2012 Microsoft Country Partner of the Year - New Zealand
> >> ________________________________
> >> This electronic message together with any attachments is confidential.
> If
> >> you are not the intended recipient: (i) do not copy, disclose or use the
> >> contents in any way (ii) please let us know by return e-mail immediately
> >> and then destroy the message. Intergen is not responsible for any
> changes
> >> made to this message and/or any attachments after sending by Intergen.
> >>
> >>
> >>
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
>
--
Cordialement.
Philippe Mouawad.
Re: Duration Assertion and Retrieve All Embedded resources not
working as expected
Posted by sebb <se...@gmail.com>.
On 5 January 2013 21:59, Philippe Mouawad <ph...@gmail.com> wrote:
> Hello sebb,
>
> Working on
>
> - https://issues.apache.org/bugzilla/show_bug.cgi?id=54156
>
> which is related to this question below , there is something I don't
> understand.
>
> Maybe you have the answer so I check with you.
>
> What is exactly Main Sample for assertions ?
>
> It seems that in the case of HTTP SAmpler + Retrieve All Embedded resources
> from HTML files checked, we have a Sampler where:
>
> - the first subresult is the main page
> - the others are the downloaded resources
>
> Do you consider this fine ?
IIRC, the top-level sample is the overall download for the page, i.e.
it includes the times and sizes for initial result plus all embedded
resources.
Not sure where that info can go if not as the top-level sample.
We also want a sample result which contains the indvdidual resources;
these are currently added as sub-samples.
I agree it's a bit odd, but what else could JMeter do?
> Where is it documented ?
AFAIK, it's not.
> It seems strange to me, you can see the problem with attached test case ,
> this is what this user is reporting.
>
> Do you think this should be fixed in JMeterThread#checkAssertions or
> somewhere else ?
>
>
> Regards
>
> Philippe
> On Thu, Nov 15, 2012 at 5:13 AM, James Fitzsimons <
> James.Fitzsimons@intergen.co.nz> wrote:
>
>> Hi all,
>>
>> We are building a test plan where we are attempting to use a duration
>> assertion to validate that responses are completing with 2000ms. This works
>> well without the "Retrieve All Embedded resources from HTML files" option
>> in the HTTP Request Defaults. However if we enable this option then we get
>> a very large number of failures despite browsing manually shows pages
>> loading in less than a second. We have included a cache manager to simulate
>> browser caching behaviour of static resources.
>>
>> In the duration assertion we have the "Apply to:" option set to "Main
>> sample only". From reading the documentation it seems that using this
>> setting the duration assertion should only be measuring the response time
>> of the primary resource (html page) being requested and not all the
>> embedded resources as well. However, this doesn't match up with what we are
>> seeing in practice. E.g. the main sample might take only 200ms (looking at
>> browser tools and IIS logs) but the 2000ms duration assertion still fails.
>>
>> Is this a bug, a misunderstanding on our part, or have we just not
>> configured the test plan correctly?
>>
>> Any thoughts / advice will be much appreciated!
>>
>> Cheers,
>> James
>>
>>
>> James Fitzsimons
>> Solution Delivery
>>
>> INTERGEN: LEVEL 7, INTERGEN HOUSE, 126 LAMBTON QUAY. PO BOX 5428,
>> WELLINGTON 6011, NEW ZEALAND.
>> TEL +64 4 472 2021 FAX +64 4 472 2027 MOB +64 27 273 3328
>> WWW.INTERGEN.CO.NZ<http://www.intergen.co.nz/>
>>
>>
>>
>> To catch the latest...
>> Follow us on Twitter: www.twitter.com/TeamIntergen<
>> http://www.twitter.com/TeamIntergen>
>> Read our blog: www.intergen.co.nz/blog<http://www.intergen.co.nz/blog>
>> Find us on LinkedIn: www.linkedin.com/company/Intergen<
>> http://www.linkedin.com/company/Intergen>
>>
>> 2012 Microsoft Country Partner of the Year - New Zealand
>> ________________________________
>> This electronic message together with any attachments is confidential. If
>> you are not the intended recipient: (i) do not copy, disclose or use the
>> contents in any way (ii) please let us know by return e-mail immediately
>> and then destroy the message. Intergen is not responsible for any changes
>> made to this message and/or any attachments after sending by Intergen.
>>
>>
>>
>
>
> --
> Cordialement.
> Philippe Mouawad.
Re: Duration Assertion and Retrieve All Embedded resources not
working as expected
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello sebb,
Working on
- https://issues.apache.org/bugzilla/show_bug.cgi?id=54156
which is related to this question below , there is something I don't
understand.
Maybe you have the answer so I check with you.
What is exactly Main Sample for assertions ?
It seems that in the case of HTTP SAmpler + Retrieve All Embedded resources
from HTML files checked, we have a Sampler where:
- the first subresult is the main page
- the others are the downloaded resources
Do you consider this fine ? Where is it documented ?
It seems strange to me, you can see the problem with attached test case ,
this is what this user is reporting.
Do you think this should be fixed in JMeterThread#checkAssertions or
somewhere else ?
Regards
Philippe
On Thu, Nov 15, 2012 at 5:13 AM, James Fitzsimons <
James.Fitzsimons@intergen.co.nz> wrote:
> Hi all,
>
> We are building a test plan where we are attempting to use a duration
> assertion to validate that responses are completing with 2000ms. This works
> well without the "Retrieve All Embedded resources from HTML files" option
> in the HTTP Request Defaults. However if we enable this option then we get
> a very large number of failures despite browsing manually shows pages
> loading in less than a second. We have included a cache manager to simulate
> browser caching behaviour of static resources.
>
> In the duration assertion we have the "Apply to:" option set to "Main
> sample only". From reading the documentation it seems that using this
> setting the duration assertion should only be measuring the response time
> of the primary resource (html page) being requested and not all the
> embedded resources as well. However, this doesn't match up with what we are
> seeing in practice. E.g. the main sample might take only 200ms (looking at
> browser tools and IIS logs) but the 2000ms duration assertion still fails.
>
> Is this a bug, a misunderstanding on our part, or have we just not
> configured the test plan correctly?
>
> Any thoughts / advice will be much appreciated!
>
> Cheers,
> James
>
>
> James Fitzsimons
> Solution Delivery
>
> INTERGEN: LEVEL 7, INTERGEN HOUSE, 126 LAMBTON QUAY. PO BOX 5428,
> WELLINGTON 6011, NEW ZEALAND.
> TEL +64 4 472 2021 FAX +64 4 472 2027 MOB +64 27 273 3328
> WWW.INTERGEN.CO.NZ<http://www.intergen.co.nz/>
>
>
>
> To catch the latest...
> Follow us on Twitter: www.twitter.com/TeamIntergen<
> http://www.twitter.com/TeamIntergen>
> Read our blog: www.intergen.co.nz/blog<http://www.intergen.co.nz/blog>
> Find us on LinkedIn: www.linkedin.com/company/Intergen<
> http://www.linkedin.com/company/Intergen>
>
> 2012 Microsoft Country Partner of the Year - New Zealand
> ________________________________
> This electronic message together with any attachments is confidential. If
> you are not the intended recipient: (i) do not copy, disclose or use the
> contents in any way (ii) please let us know by return e-mail immediately
> and then destroy the message. Intergen is not responsible for any changes
> made to this message and/or any attachments after sending by Intergen.
>
>
>
--
Cordialement.
Philippe Mouawad.