You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Noel O'Brien <no...@newbay.com> on 2009/03/26 13:29:51 UTC
Accuracy of Elapsed Time
Hi,
I'm running some load testing, nothing too heavy. 10 JMeter threads, ramp up
is 30. JMeter is run from the command line and the test plan has no listeners.
using the -l flag, I capture the results to a CSV file.
I then load the results file into JMeter and look at hem from a Summary
Listener, View Tree Listener etc. I'm a little confused by the elapsed time
results.
The app under test has the ability to log the time that's taken to receive the
request and complete sending the response, which for an auth API call it lists
as taking 114ms for example (ronded to the closest ms)
I've set up tcpdump on the machine running the app (refer to as server
henceforth) and the machine running JMeter (refer to as client henceforth).
tcpdump on the server reports that the time between the request and response
packets to be 113.83ms, while tcpdump on the client reports that the time
between the request and response packets 113.96ms
However for that particular HTTP request, JMeter reports that the load time
(elasped time) is 174ms and that the latency is 174 ms also. How is this
discrepancy explained?
As I understand it, the latency is the time taken to receive the first byte of
the response and since the load time and latency are the same then it
indicated that the response payload was received within 1 ms. Is this correct?
Or is it even possible to achieve this? The payload size for this response is
106 bytes.
From the tcpdump on the client, it's clear that the response packets are
available to JMeter after 113ms so I'm concerned about what's causing the
increase.
FWIW, the auth request sampler has an XPath Extracter Post processor, but
processing time for that's not included in the load time right??. All HTTP
Samplers are Java, not HTTPClient.
Regards,
Noel
Re: Accuracy of Elapsed Time
Posted by Noel O'Brien <no...@newbay.com>.
Just wondering: would adding a think time to a load test help JMeter in terms
of elapsed time accuracy?
I ask because, I configured the app under testing to temporarily eliminate
activity logging which was expensive in terms of time on the server side. Now
the average test times in JMeter are much lower when viewed in the summary
report listener from 2sec to about 300 ms per call.
I looked at the app logs which output the time it took to process the
requests; the longest time was 8sec, but JMeter is reporting the max times for
each call are about 15sec!
I'm thinking that server is super fast (but that's probably just the optimist
in me ;)) and JMeter is struggling to keep up in processing. Again the test
plan has no listeners, outputs only critical results data in CSV, JMeter runs
in non-gui mode, with 10 threads, 30 sec ramp-up.
When I arrived into work this morning, JMeter was at 100% CPU, but the app
server was cruising at about 10%. I remember reading somewhere on the JMeter
site (possibly the wiki) that a faster server will put more load on JMeter to
keep up. What can be done in this situation to have a stable load test?
Will a think time help if I increase the number of threads? Or will a higher
number of threads make things worse?
Regards,
Noel
On Friday 27 March 2009 09:13:06 Noel O'Brien wrote:
> On Friday 27 March 2009 01:39:27 sebb wrote:
> > I've not used it much, but the only problem I can think of with 1.6 is
> > the change to the socket code which causes the "unconnected socket"
> > error when using HTTPS.
> >
> > [This was a bug in JMeter which has been fixed; the nightlies work with
> > 1.6]
>
> The only issue I saw was the bug in Java (BackingStoreException), which did
> not impact testing; it only printed out lots of warnings and resulted in
> the "Recent File List" not being saved.
>
> Solution was to install a version of Java >= 1.6.0_10
>
> Regarding the original issue, I guess I've nothing to lose by trying with
> 1.5. I'll give it a go and report back
>
> Regards,
> Noel
>
> > Are there any other problems with 1.6 you know of?
> >
> > On 27/03/2009, Oliver Erlewein [DATACOM] <Ol...@datacom.co.nz>
>
> wrote:
> > > Hi Noel,
> > >
> > > I'd stay away from using Java 1.6.0 for now. Hasn't yet proven to be
> > > that dependable. Sebb might have some more info on that.
> > >
> > > Cheers Oliver
> > >
> > >
> > > -----Original Message-----
> > > From: Noel O'Brien [mailto:nobrien@newbay.com]
> > > Sent: Friday, March 27, 2009 3:17 AM
> > > To: jmeter-user@jakarta.apache.org
> > > Cc: sebb
> > > Subject: Re: Accuracy of Elapsed Time
> > >
> > > On Thursday 26 March 2009 13:00:02 sebb wrote:
> > > > On 26/03/2009, Noel O'Brien <no...@newbay.com> wrote:
> > > > > Hi,
> > > > >
> > > > > I'm running some load testing, nothing too heavy. 10 JMeter
> > >
> > > threads,
> > >
> > > > > ramp up is 30. JMeter is run from the command line and the test
> > > > > plan
> > >
> > > has
> > >
> > > > > no listeners. using the -l flag, I capture the results to a CSV
> > >
> > > file.
> > >
> > > > Good.
> > > >
> > > > JMeter version?
> > >
> > > JMeter 2.3.2:r665936
> > > Java: Sun 1.6.0_06-b02
> > > OS: RHEL 5.2 running in RHEL hypervisor
> > >
> > > > > I then load the results file into JMeter and look at hem from a
> > >
> > > Summary
> > >
> > > > > Listener, View Tree Listener etc. I'm a little confused by the
> > >
> > > elapsed
> > >
> > > > > time results.
> > > > >
> > > > > The app under test has the ability to log the time that's taken
> > > > > to receive the request and complete sending the response, which
> > > > > for an
> > >
> > > auth
> > >
> > > > > API call it lists as taking 114ms for example (ronded to the
> > > > > closest
> > >
> > > ms)
> > >
> > > > > I've set up tcpdump on the machine running the app (refer to as
> > >
> > > server
> > >
> > > > > henceforth) and the machine running JMeter (refer to as client
> > > > > henceforth).
> > > > >
> > > > > tcpdump on the server reports that the time between the request
> > > > > and response packets to be 113.83ms, while tcpdump on the client
> > > > > reports
> > >
> > > that
> > >
> > > > > the time between the request and response packets 113.96ms
> > > > >
> > > > > However for that particular HTTP request, JMeter reports that the
> > >
> > > load
> > >
> > > > > time (elasped time) is 174ms and that the latency is 174 ms also.
> > >
> > > How is
> > >
> > > > > this discrepancy explained?
> > > >
> > > > There is overhead in the OS, Java and JMeter on both sending and
> > >
> > > receiving.
> > >
> > > Some of the other API calls I make only show a difference of a few ms
> > > in JMeter
> > > compared to the app/tcpdump
> > >
> > > > Note also that the timer resolution will affect the reported elapsed
> > >
> > > time.
> > >
> > > > However if you are running 2.3.2+ on Java 1.5+ it will use a higher
> > > > resolution timer.
> > >
> > > Does it matter that the auth API call is the first call for each
> > > thread? (Is
> > > the thread doing extra work at it's start-up that would impact the
> > > time?)
> > >
> > > > > As I understand it, the latency is the time taken to receive the
> > >
> > > first
> > >
> > > > > byte of the response and since the load time and latency are the
> > >
> > > same
> > >
> > > > > then it indicated that the response payload was received within 1
> > >
> > > ms. Is
> > >
> > > > > this correct? Or is it even possible to achieve this? The payload
> > >
> > > size
> > >
> > > > > for this response is 106 bytes.
> > > >
> > > > Latency is time to first response.
> > > > This may be the entire response, especially for small payloads.
> > >
> > > Ah, I see :)
> > >
> > > > > From the tcpdump on the client, it's clear that the response
> > >
> > > packets are
> > >
> > > > > available to JMeter after 113ms (*) so I'm concerned about what's
> > > > > causing the increase.
> > > >
> > > > (*) The OS (and Java) have to process the request before tcpdump
> > > > sees it, and likewise after tcpdump sees the response.
> > >
> > > Hmmm, see above re: other API calls.
> > >
> > > > > FWIW, the auth request sampler has an XPath Extracter Post
> > >
> > > processor,
> > >
> > > > > but processing time for that's not included in the load time
> > >
> > > right??. All
> > >
> > > > > HTTP Samplers are Java, not HTTPClient.
> > > >
> > > > Post-Processors are not included in individual sample-times.
> > >
> > > Good :)
> > >
> > > > The HttpSampler does as little processing as possible whilst timing
> > > > the sample, but it has to issue the connect - retrying if necessary
> > > > - then send the data and process the response.
> > > >
> > > > The connection time is not currently measured separately.
> > > >
> > > > I don't know why the times differ by as much as you are seeing.
> > > > Is this the case for all samples, or only a few?
> > >
> > > Only a few when the tests are run for a short period of time. Even
> > > with a
> > > single threaded test the times are off a little for some calls.
> > >
> > > When the test is left for more than a few hours the times seem to
> > > become more
> > > in-accurate.
> > >
> > > > And does it really matter, so long as the server can handle the load
> > > > it's supposed to?
> > >
> > > Part of our tests are to test how many "users" the app can handle
> > > before a
> > > degradation of quality beyond a certain point (e.g. 2 seconds), so the
> > > inaccuracies I'm seeing are not helping to determine that number :(
> > > because
> > > the average time per call for 10 threads grows over time
> > >
> > > > > Regards,
> > > > >
> > > > > Noel
> > > >
> > > > --------------------------------------------------------------------
> > > >- To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> > > > For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Accuracy of Elapsed Time
Posted by Noel O'Brien <no...@newbay.com>.
On Friday 27 March 2009 01:39:27 sebb wrote:
> I've not used it much, but the only problem I can think of with 1.6 is
> the change to the socket code which causes the "unconnected socket"
> error when using HTTPS.
>
> [This was a bug in JMeter which has been fixed; the nightlies work with
> 1.6]
The only issue I saw was the bug in Java (BackingStoreException), which did
not impact testing; it only printed out lots of warnings and resulted in the
"Recent File List" not being saved.
Solution was to install a version of Java >= 1.6.0_10
Regarding the original issue, I guess I've nothing to lose by trying with 1.5.
I'll give it a go and report back
Regards,
Noel
> Are there any other problems with 1.6 you know of?
>
> On 27/03/2009, Oliver Erlewein [DATACOM] <Ol...@datacom.co.nz>
wrote:
> > Hi Noel,
> >
> > I'd stay away from using Java 1.6.0 for now. Hasn't yet proven to be
> > that dependable. Sebb might have some more info on that.
> >
> > Cheers Oliver
> >
> >
> > -----Original Message-----
> > From: Noel O'Brien [mailto:nobrien@newbay.com]
> > Sent: Friday, March 27, 2009 3:17 AM
> > To: jmeter-user@jakarta.apache.org
> > Cc: sebb
> > Subject: Re: Accuracy of Elapsed Time
> >
> > On Thursday 26 March 2009 13:00:02 sebb wrote:
> > > On 26/03/2009, Noel O'Brien <no...@newbay.com> wrote:
> > > > Hi,
> > > >
> > > > I'm running some load testing, nothing too heavy. 10 JMeter
> >
> > threads,
> >
> > > > ramp up is 30. JMeter is run from the command line and the test plan
> >
> > has
> >
> > > > no listeners. using the -l flag, I capture the results to a CSV
> >
> > file.
> >
> > > Good.
> > >
> > > JMeter version?
> >
> > JMeter 2.3.2:r665936
> > Java: Sun 1.6.0_06-b02
> > OS: RHEL 5.2 running in RHEL hypervisor
> >
> > > > I then load the results file into JMeter and look at hem from a
> >
> > Summary
> >
> > > > Listener, View Tree Listener etc. I'm a little confused by the
> >
> > elapsed
> >
> > > > time results.
> > > >
> > > > The app under test has the ability to log the time that's taken to
> > > > receive the request and complete sending the response, which for an
> >
> > auth
> >
> > > > API call it lists as taking 114ms for example (ronded to the closest
> >
> > ms)
> >
> > > > I've set up tcpdump on the machine running the app (refer to as
> >
> > server
> >
> > > > henceforth) and the machine running JMeter (refer to as client
> > > > henceforth).
> > > >
> > > > tcpdump on the server reports that the time between the request and
> > > > response packets to be 113.83ms, while tcpdump on the client reports
> >
> > that
> >
> > > > the time between the request and response packets 113.96ms
> > > >
> > > > However for that particular HTTP request, JMeter reports that the
> >
> > load
> >
> > > > time (elasped time) is 174ms and that the latency is 174 ms also.
> >
> > How is
> >
> > > > this discrepancy explained?
> > >
> > > There is overhead in the OS, Java and JMeter on both sending and
> >
> > receiving.
> >
> > Some of the other API calls I make only show a difference of a few ms in
> > JMeter
> > compared to the app/tcpdump
> >
> > > Note also that the timer resolution will affect the reported elapsed
> >
> > time.
> >
> > > However if you are running 2.3.2+ on Java 1.5+ it will use a higher
> > > resolution timer.
> >
> > Does it matter that the auth API call is the first call for each thread?
> > (Is
> > the thread doing extra work at it's start-up that would impact the
> > time?)
> >
> > > > As I understand it, the latency is the time taken to receive the
> >
> > first
> >
> > > > byte of the response and since the load time and latency are the
> >
> > same
> >
> > > > then it indicated that the response payload was received within 1
> >
> > ms. Is
> >
> > > > this correct? Or is it even possible to achieve this? The payload
> >
> > size
> >
> > > > for this response is 106 bytes.
> > >
> > > Latency is time to first response.
> > > This may be the entire response, especially for small payloads.
> >
> > Ah, I see :)
> >
> > > > From the tcpdump on the client, it's clear that the response
> >
> > packets are
> >
> > > > available to JMeter after 113ms (*) so I'm concerned about what's
> > > > causing the increase.
> > >
> > > (*) The OS (and Java) have to process the request before tcpdump sees
> > > it, and likewise after tcpdump sees the response.
> >
> > Hmmm, see above re: other API calls.
> >
> > > > FWIW, the auth request sampler has an XPath Extracter Post
> >
> > processor,
> >
> > > > but processing time for that's not included in the load time
> >
> > right??. All
> >
> > > > HTTP Samplers are Java, not HTTPClient.
> > >
> > > Post-Processors are not included in individual sample-times.
> >
> > Good :)
> >
> > > The HttpSampler does as little processing as possible whilst timing
> > > the sample, but it has to issue the connect - retrying if necessary -
> > > then send the data and process the response.
> > >
> > > The connection time is not currently measured separately.
> > >
> > > I don't know why the times differ by as much as you are seeing.
> > > Is this the case for all samples, or only a few?
> >
> > Only a few when the tests are run for a short period of time. Even with
> > a
> > single threaded test the times are off a little for some calls.
> >
> > When the test is left for more than a few hours the times seem to become
> > more
> > in-accurate.
> >
> > > And does it really matter, so long as the server can handle the load
> > > it's supposed to?
> >
> > Part of our tests are to test how many "users" the app can handle before
> > a
> > degradation of quality beyond a certain point (e.g. 2 seconds), so the
> > inaccuracies I'm seeing are not helping to determine that number :(
> > because
> > the average time per call for 10 threads grows over time
> >
> > > > Regards,
> > > >
> > > > Noel
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Accuracy of Elapsed Time
Posted by sebb <se...@gmail.com>.
I've not used it much, but the only problem I can think of with 1.6 is
the change to the socket code which causes the "unconnected socket"
error when using HTTPS.
[This was a bug in JMeter which has been fixed; the nightlies work with 1.6]
Are there any other problems with 1.6 you know of?
On 27/03/2009, Oliver Erlewein [DATACOM] <Ol...@datacom.co.nz> wrote:
> Hi Noel,
>
> I'd stay away from using Java 1.6.0 for now. Hasn't yet proven to be
> that dependable. Sebb might have some more info on that.
>
> Cheers Oliver
>
>
> -----Original Message-----
> From: Noel O'Brien [mailto:nobrien@newbay.com]
> Sent: Friday, March 27, 2009 3:17 AM
> To: jmeter-user@jakarta.apache.org
> Cc: sebb
> Subject: Re: Accuracy of Elapsed Time
>
>
> On Thursday 26 March 2009 13:00:02 sebb wrote:
> > On 26/03/2009, Noel O'Brien <no...@newbay.com> wrote:
> > > Hi,
> > >
> > > I'm running some load testing, nothing too heavy. 10 JMeter
> threads,
> > > ramp up is 30. JMeter is run from the command line and the test plan
> has
> > > no listeners. using the -l flag, I capture the results to a CSV
> file.
> >
> > Good.
> >
> > JMeter version?
>
> JMeter 2.3.2:r665936
> Java: Sun 1.6.0_06-b02
> OS: RHEL 5.2 running in RHEL hypervisor
>
> > > I then load the results file into JMeter and look at hem from a
> Summary
> > > Listener, View Tree Listener etc. I'm a little confused by the
> elapsed
> > > time results.
> > >
> > > The app under test has the ability to log the time that's taken to
> > > receive the request and complete sending the response, which for an
> auth
> > > API call it lists as taking 114ms for example (ronded to the closest
> ms)
> > >
> > > I've set up tcpdump on the machine running the app (refer to as
> server
> > > henceforth) and the machine running JMeter (refer to as client
> > > henceforth).
> > >
> > > tcpdump on the server reports that the time between the request and
> > > response packets to be 113.83ms, while tcpdump on the client reports
> that
> > > the time between the request and response packets 113.96ms
> > >
> > > However for that particular HTTP request, JMeter reports that the
> load
> > > time (elasped time) is 174ms and that the latency is 174 ms also.
> How is
> > > this discrepancy explained?
> >
> > There is overhead in the OS, Java and JMeter on both sending and
> receiving.
>
> Some of the other API calls I make only show a difference of a few ms in
> JMeter
> compared to the app/tcpdump
>
> > Note also that the timer resolution will affect the reported elapsed
> time.
> > However if you are running 2.3.2+ on Java 1.5+ it will use a higher
> > resolution timer.
>
> Does it matter that the auth API call is the first call for each thread?
> (Is
> the thread doing extra work at it's start-up that would impact the
> time?)
>
> > > As I understand it, the latency is the time taken to receive the
> first
> > > byte of the response and since the load time and latency are the
> same
> > > then it indicated that the response payload was received within 1
> ms. Is
> > > this correct? Or is it even possible to achieve this? The payload
> size
> > > for this response is 106 bytes.
> >
> > Latency is time to first response.
> > This may be the entire response, especially for small payloads.
>
> Ah, I see :)
>
> > > From the tcpdump on the client, it's clear that the response
> packets are
> > > available to JMeter after 113ms (*) so I'm concerned about what's
> > > causing the increase.
> >
> > (*) The OS (and Java) have to process the request before tcpdump sees
> > it, and likewise after tcpdump sees the response.
>
> Hmmm, see above re: other API calls.
>
> > > FWIW, the auth request sampler has an XPath Extracter Post
> processor,
> > > but processing time for that's not included in the load time
> right??. All
> > > HTTP Samplers are Java, not HTTPClient.
> >
> > Post-Processors are not included in individual sample-times.
>
> Good :)
>
> > The HttpSampler does as little processing as possible whilst timing
> > the sample, but it has to issue the connect - retrying if necessary -
> > then send the data and process the response.
> >
> > The connection time is not currently measured separately.
> >
> > I don't know why the times differ by as much as you are seeing.
> > Is this the case for all samples, or only a few?
>
> Only a few when the tests are run for a short period of time. Even with
> a
> single threaded test the times are off a little for some calls.
>
> When the test is left for more than a few hours the times seem to become
> more
> in-accurate.
>
> > And does it really matter, so long as the server can handle the load
> > it's supposed to?
>
> Part of our tests are to test how many "users" the app can handle before
> a
> degradation of quality beyond a certain point (e.g. 2 seconds), so the
> inaccuracies I'm seeing are not helping to determine that number :(
> because
> the average time per call for 10 threads grows over time
>
> > > Regards,
> > >
> > > Noel
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
RE: Accuracy of Elapsed Time
Posted by "Oliver Erlewein [DATACOM]" <Ol...@datacom.co.nz>.
Hi Noel,
I'd stay away from using Java 1.6.0 for now. Hasn't yet proven to be
that dependable. Sebb might have some more info on that.
Cheers Oliver
-----Original Message-----
From: Noel O'Brien [mailto:nobrien@newbay.com]
Sent: Friday, March 27, 2009 3:17 AM
To: jmeter-user@jakarta.apache.org
Cc: sebb
Subject: Re: Accuracy of Elapsed Time
On Thursday 26 March 2009 13:00:02 sebb wrote:
> On 26/03/2009, Noel O'Brien <no...@newbay.com> wrote:
> > Hi,
> >
> > I'm running some load testing, nothing too heavy. 10 JMeter
threads,
> > ramp up is 30. JMeter is run from the command line and the test plan
has
> > no listeners. using the -l flag, I capture the results to a CSV
file.
>
> Good.
>
> JMeter version?
JMeter 2.3.2:r665936
Java: Sun 1.6.0_06-b02
OS: RHEL 5.2 running in RHEL hypervisor
> > I then load the results file into JMeter and look at hem from a
Summary
> > Listener, View Tree Listener etc. I'm a little confused by the
elapsed
> > time results.
> >
> > The app under test has the ability to log the time that's taken to
> > receive the request and complete sending the response, which for an
auth
> > API call it lists as taking 114ms for example (ronded to the closest
ms)
> >
> > I've set up tcpdump on the machine running the app (refer to as
server
> > henceforth) and the machine running JMeter (refer to as client
> > henceforth).
> >
> > tcpdump on the server reports that the time between the request and
> > response packets to be 113.83ms, while tcpdump on the client reports
that
> > the time between the request and response packets 113.96ms
> >
> > However for that particular HTTP request, JMeter reports that the
load
> > time (elasped time) is 174ms and that the latency is 174 ms also.
How is
> > this discrepancy explained?
>
> There is overhead in the OS, Java and JMeter on both sending and
receiving.
Some of the other API calls I make only show a difference of a few ms in
JMeter
compared to the app/tcpdump
> Note also that the timer resolution will affect the reported elapsed
time.
> However if you are running 2.3.2+ on Java 1.5+ it will use a higher
> resolution timer.
Does it matter that the auth API call is the first call for each thread?
(Is
the thread doing extra work at it's start-up that would impact the
time?)
> > As I understand it, the latency is the time taken to receive the
first
> > byte of the response and since the load time and latency are the
same
> > then it indicated that the response payload was received within 1
ms. Is
> > this correct? Or is it even possible to achieve this? The payload
size
> > for this response is 106 bytes.
>
> Latency is time to first response.
> This may be the entire response, especially for small payloads.
Ah, I see :)
> > From the tcpdump on the client, it's clear that the response
packets are
> > available to JMeter after 113ms (*) so I'm concerned about what's
> > causing the increase.
>
> (*) The OS (and Java) have to process the request before tcpdump sees
> it, and likewise after tcpdump sees the response.
Hmmm, see above re: other API calls.
> > FWIW, the auth request sampler has an XPath Extracter Post
processor,
> > but processing time for that's not included in the load time
right??. All
> > HTTP Samplers are Java, not HTTPClient.
>
> Post-Processors are not included in individual sample-times.
Good :)
> The HttpSampler does as little processing as possible whilst timing
> the sample, but it has to issue the connect - retrying if necessary -
> then send the data and process the response.
>
> The connection time is not currently measured separately.
>
> I don't know why the times differ by as much as you are seeing.
> Is this the case for all samples, or only a few?
Only a few when the tests are run for a short period of time. Even with
a
single threaded test the times are off a little for some calls.
When the test is left for more than a few hours the times seem to become
more
in-accurate.
> And does it really matter, so long as the server can handle the load
> it's supposed to?
Part of our tests are to test how many "users" the app can handle before
a
degradation of quality beyond a certain point (e.g. 2 seconds), so the
inaccuracies I'm seeing are not helping to determine that number :(
because
the average time per call for 10 threads grows over time
> > Regards,
> >
> > Noel
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Accuracy of Elapsed Time
Posted by Noel O'Brien <no...@newbay.com>.
On Thursday 26 March 2009 13:00:02 sebb wrote:
> On 26/03/2009, Noel O'Brien <no...@newbay.com> wrote:
> > Hi,
> >
> > I'm running some load testing, nothing too heavy. 10 JMeter threads,
> > ramp up is 30. JMeter is run from the command line and the test plan has
> > no listeners. using the -l flag, I capture the results to a CSV file.
>
> Good.
>
> JMeter version?
JMeter 2.3.2:r665936
Java: Sun 1.6.0_06-b02
OS: RHEL 5.2 running in RHEL hypervisor
> > I then load the results file into JMeter and look at hem from a Summary
> > Listener, View Tree Listener etc. I'm a little confused by the elapsed
> > time results.
> >
> > The app under test has the ability to log the time that's taken to
> > receive the request and complete sending the response, which for an auth
> > API call it lists as taking 114ms for example (ronded to the closest ms)
> >
> > I've set up tcpdump on the machine running the app (refer to as server
> > henceforth) and the machine running JMeter (refer to as client
> > henceforth).
> >
> > tcpdump on the server reports that the time between the request and
> > response packets to be 113.83ms, while tcpdump on the client reports that
> > the time between the request and response packets 113.96ms
> >
> > However for that particular HTTP request, JMeter reports that the load
> > time (elasped time) is 174ms and that the latency is 174 ms also. How is
> > this discrepancy explained?
>
> There is overhead in the OS, Java and JMeter on both sending and receiving.
Some of the other API calls I make only show a difference of a few ms in JMeter
compared to the app/tcpdump
> Note also that the timer resolution will affect the reported elapsed time.
> However if you are running 2.3.2+ on Java 1.5+ it will use a higher
> resolution timer.
Does it matter that the auth API call is the first call for each thread? (Is
the thread doing extra work at it's start-up that would impact the time?)
> > As I understand it, the latency is the time taken to receive the first
> > byte of the response and since the load time and latency are the same
> > then it indicated that the response payload was received within 1 ms. Is
> > this correct? Or is it even possible to achieve this? The payload size
> > for this response is 106 bytes.
>
> Latency is time to first response.
> This may be the entire response, especially for small payloads.
Ah, I see :)
> > From the tcpdump on the client, it's clear that the response packets are
> > available to JMeter after 113ms (*) so I'm concerned about what's
> > causing the increase.
>
> (*) The OS (and Java) have to process the request before tcpdump sees
> it, and likewise after tcpdump sees the response.
Hmmm, see above re: other API calls.
> > FWIW, the auth request sampler has an XPath Extracter Post processor,
> > but processing time for that's not included in the load time right??. All
> > HTTP Samplers are Java, not HTTPClient.
>
> Post-Processors are not included in individual sample-times.
Good :)
> The HttpSampler does as little processing as possible whilst timing
> the sample, but it has to issue the connect - retrying if necessary -
> then send the data and process the response.
>
> The connection time is not currently measured separately.
>
> I don't know why the times differ by as much as you are seeing.
> Is this the case for all samples, or only a few?
Only a few when the tests are run for a short period of time. Even with a
single threaded test the times are off a little for some calls.
When the test is left for more than a few hours the times seem to become more
in-accurate.
> And does it really matter, so long as the server can handle the load
> it's supposed to?
Part of our tests are to test how many "users" the app can handle before a
degradation of quality beyond a certain point (e.g. 2 seconds), so the
inaccuracies I'm seeing are not helping to determine that number :( because
the average time per call for 10 threads grows over time
> > Regards,
> >
> > Noel
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Accuracy of Elapsed Time
Posted by sebb <se...@gmail.com>.
On 26/03/2009, Noel O'Brien <no...@newbay.com> wrote:
> Hi,
>
> I'm running some load testing, nothing too heavy. 10 JMeter threads, ramp up
> is 30. JMeter is run from the command line and the test plan has no listeners.
> using the -l flag, I capture the results to a CSV file.
Good.
JMeter version?
> I then load the results file into JMeter and look at hem from a Summary
> Listener, View Tree Listener etc. I'm a little confused by the elapsed time
> results.
>
> The app under test has the ability to log the time that's taken to receive the
> request and complete sending the response, which for an auth API call it lists
> as taking 114ms for example (ronded to the closest ms)
>
> I've set up tcpdump on the machine running the app (refer to as server
> henceforth) and the machine running JMeter (refer to as client henceforth).
>
> tcpdump on the server reports that the time between the request and response
> packets to be 113.83ms, while tcpdump on the client reports that the time
> between the request and response packets 113.96ms
>
> However for that particular HTTP request, JMeter reports that the load time
> (elasped time) is 174ms and that the latency is 174 ms also. How is this
> discrepancy explained?
There is overhead in the OS, Java and JMeter on both sending and receiving.
Note also that the timer resolution will affect the reported elapsed time.
However if you are running 2.3.2+ on Java 1.5+ it will use a higher
resolution timer.
> As I understand it, the latency is the time taken to receive the first byte of
> the response and since the load time and latency are the same then it
> indicated that the response payload was received within 1 ms. Is this correct?
> Or is it even possible to achieve this? The payload size for this response is
> 106 bytes.
Latency is time to first response.
This may be the entire response, especially for small payloads.
> From the tcpdump on the client, it's clear that the response packets are
> available to JMeter after 113ms (*) so I'm concerned about what's causing the
> increase.
(*) The OS (and Java) have to process the request before tcpdump sees
it, and likewise after tcpdump sees the response.
> FWIW, the auth request sampler has an XPath Extracter Post processor, but
> processing time for that's not included in the load time right??. All HTTP
> Samplers are Java, not HTTPClient.
Post-Processors are not included in individual sample-times.
The HttpSampler does as little processing as possible whilst timing
the sample, but it has to issue the connect - retrying if necessary -
then send the data and process the response.
The connection time is not currently measured separately.
I don't know why the times differ by as much as you are seeing.
Is this the case for all samples, or only a few?
And does it really matter, so long as the server can handle the load
it's supposed to?
> Regards,
>
> Noel
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org