You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@jmeter.apache.org by "Robin D. Wilson" <rw...@gmail.com> on 2011/12/14 22:35:54 UTC

Strange 'pause' activity during testing

I have a marginally complicated test case that performs a 'registration' on
my site. It gets the home page, POSTS a form, gets the response, POSTS
another form, gets that response, and then completes. The test runs fine
with 100 threads, and 30000 iterations - IF I don't "Retrieve All Embedded
Resources from HTML Files". In this mode, I am really testing the throughput
of my 'tomcat' application, not the other elements of my system. (I'm
assuming that the other elements are being retrieved from our Content Data
Network instead of our main system in this case.)

If I enable the "Retrieve All Embedded Resources from HTML Files" flag, and
tune the test down to 10 threads with 3000 total iterations, I notice a very
strange behavior. The test runs along at a pretty good clip for the first
~600 iterations (about 1 min, 25 seconds into the run), and then it just
stops making requests for about 35 seconds. Then, it picks back up again and
runs for another 1 m 25s, and then stops again for 35 seconds... (NOTE: with
the 10 threads, 3000 total iterations - but with "Retrieve All Embedded..."
disabled - I don't see the 'stop' behavior either - so it isn't caused by
tuning it down...)

I recently added the "Perfmon Metrics Collector" to the test script, so I
could see if one of the servers was maxed out - but it looks like all the
servers are idle during the 'stop' period. Likewise, I added the Perfmon for
the "localhost" (running the JMeter test) to see if it was swamped - but it
too is idle during the 'stop'. I swapped out our network switch (the test
environment is on an isolated network switch) with a _much_ higher capacity
switch - in case there was a network issue, still no change.

I'm running out of ideas for things to check - so I thought I'd ask you guys
if you have any suggestions of things I should look at.

My system consists of:

	WinXP - running JMeter 2.4.1 - driving the test script in GUI mode
	Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as the web
server - using AJP Proxy to Server 2
	Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the App
Server - connecting through Hibernate to Server 3
	Server 3 - running Red Hat Linux with MySQL 5.x as the DB Server

All 4 machines are running on a private switched network (32Gbs backplane).

The requests are downloading about 3MB total (per thread per iteration) over
4 main URL requests, and 30+ 'Retrieve All Embedded' requests.

At first I thought it was the network - but the new switch seemed to deny
that thought (the old switch had a much slower backplane). Also, I'm having
no trouble collecting the PerfMon data during the 'stop' period - so the
network is still functioning just fine...
Then I thought it might be garbage collection on the tomcat - but I watched
the gc.log - and it doesn't do any GCs during the 'stop' period.
Then I thought it might be the garbage collection on the JMeter side, so I
started the JMeter.bat from a 'cmd' prompt with gc logging enabled - it
doesn't do any GCs during the 'stop' period either.

The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no network
I/O, no disk I/O, etc.) during the 'stop' period.

The JMeter box (WinXP) is doing very little during that time too (I
attribute the little bit of activity to displaying the PerfMon graphs, and
Remote Desktop display to my desktop computer)...

--
Robin D. Wilson
Sr. Director of Web Development
KingsIsle Entertainment, Inc.
VOICE: 512-777-1861
www.KingsIsle.com




---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by Shmuel Krakower <sh...@gmail.com>.

I had something like that too when proxy configuration was not set
correctly on the other box.
What is the result of the embeded resources on the linux machine? Did they
got the right response or an error?
בתאריך 2012 4 9 13:56, מאת "ravi" <ra...@gmail.com>:

> I disabled iptables and ip6tables.
> But still I am seeing this issue only on linux.
>
> Can you let me know if there is anything else I need to do.
>
>
> Thanks
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>

RE: Strange 'pause' activity during testing

Posted by "Robin D. Wilson" <rw...@gmail.com>.

To debug my issue, I had to strip the JMeter test down to the bare minimum that would still reproduce the issue. Then I tried to
separate out parts that might affect the issue. In my case, going straight to the Tomcat java app server (bypassing the Apache web
server 'proxy') eliminated the problem. This allowed me to focus on the Apache config - and that's when my system admin noticed that
someone had enabled the iptables and ip6tables on the Apache server.

For your problem, the debugging will be much the same - try to isolate the JMeter and server portions of the test to the bare
minimum, then eliminate parts that might affect the test until you identify the part that is causing the delay.

Also, just out of curiosity, what sort of network connection are you using for your test? We had an isolated 1000Mbps switch with a
12Gbps backplane running our tests (so only the test boxes and JMeter were on the network). Keeping your test traffic isolated from
other network activity is crucial to a reliable (and repeatable) test environment. If you have a 1Gbps switch, that's probably good
- but even so, if you have strange delays on the communications - make sure you look at the switch to see if is throwing errors or
running out of bandwidth on the switch backplane.

--
Robin D. Wilson
Sr. Director of Web Development
KingsIsle Entertainment, Inc.
VOICE: 512-777-1861
www.KingsIsle.com


-----Original Message-----
From: ravi [mailto:ravireliable@gmail.com] 
Sent: Monday, April 09, 2012 5:56 AM
To: user@jmeter.apache.org
Subject: Re: Strange 'pause' activity during testing

I disabled iptables and ip6tables.
But still I am seeing this issue only on linux.

Can you let me know if there is anything else I need to do.


Thanks





---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by ravi <ra...@gmail.com>.

I disabled iptables and ip6tables.
But still I am seeing this issue only on linux.

Can you let me know if there is anything else I need to do.


Thanks





---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by "Robin D. Wilson" <rw...@gmail.com>.

When I experienced this previously, it was from a Windows XP Jmeter box to my web app running on a Linux server setup (Apache web server talking to a Tomcat app server).

My problem was 'iptables' (Linux firewall) running on the Apache web server.

But I was running 100 simultaneous threads (users), not just 2... (But, like you, I only saw the pause when I enabled the "retrieve all embedded resources" flag.)

Make sure you disable iptables and ip6tables...

--
Robin D. Wilson
VOICE: 512-777-1861



On Apr 8, 2012, at 12:15 PM, ravi <ra...@gmail.com> wrote:

I am seeing this issue in linux box and not in windows.
I am using "Retrieve embedded resource html" 
option and 2 connections. I am running the test with 1user i.e., 
1 thread. It works fine when run from 
Windows machine. But it has aproblem when the same test runs on Linux 
and the calls that have 
"embedded resources html" 
option checked are taking >300sec in linux while it is  < 10sec in windows.
Can somebody explain why is 
this behaviour. I even turned off Firewalls
on the linux box and the webserver.



---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Strange 'pause' activity during testing

Posted by ravi <ra...@gmail.com>.

I am seeing this issue in linux box and not in windows.
 I am using "Retrieve embedded resource html" 
option and 2 connections. I am running the test with 1user i.e., 
1 thread. It works fine when run from 
Windows machine. But it has aproblem when the same test runs on Linux 
and the calls that have 
"embedded resources html" 
option checked are taking >300sec in linux while it is  < 10sec in windows.
 Can somebody explain why is 
this behaviour. I even turned off Firewalls
 on the linux box and the webserver.



---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by Kirk <ki...@gmail.com>.

jpid -l to get the process id of JMeter and then jstack <pid> to get the thread dump. Works on all systems the same way.

If GC isn't running than what you're seeing is that JMeter threads are hung up making server requests. It's a result of the threading model used in JMeter. If you go back to the beginning of summer you can see the Sebb and I had a discussion regarding the challenges of JMeter's threading model.. and why LoadRunner, SilkRunner.. etc use an event driven thread pool based model instead of one thread per user.

Regards,
Kirk

On Dec 14, 2011, at 11:32 PM, Robin D. Wilson wrote:

> I can take a thread dump if someone can tell me how to do it... (Go easy on
> me - I'm a manager, no longer a coder...)
> 
> And you were correct, there are no GCs during the 'stop' period on either
> the tomcat servers, or the JMeter client box.
> 
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> VOICE: 512-777-1861
> www.KingsIsle.com
> 
> 
> -----Original Message-----
> From: Deepak Shetty [mailto:shettyd@gmail.com] 
> Sent: Wednesday, December 14, 2011 4:20 PM
> To: JMeter Users List
> Subject: Re: Strange 'pause' activity during testing
> 
> I think the original post mentioned that GC was not running.
> can you take a thread dump when there is a pause and see what the threads
> are waiting on?
> 
> 
> 
> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com> wrote:
> 
>> might be GC.. me be that JMeter's threads are all hung up in your 
>> server doing stuff.
>> 
>> Regards,
>> Kirk
>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>> 
>>> I have a marginally complicated test case that performs a 'registration'
>> on
>>> my site. It gets the home page, POSTS a form, gets the response, 
>>> POSTS another form, gets that response, and then completes. The test 
>>> runs fine with 100 threads, and 30000 iterations - IF I don't 
>>> "Retrieve All
>> Embedded
>>> Resources from HTML Files". In this mode, I am really testing the
>> throughput
>>> of my 'tomcat' application, not the other elements of my system. 
>>> (I'm assuming that the other elements are being retrieved from our 
>>> Content
>> Data
>>> Network instead of our main system in this case.)
>>> 
>>> If I enable the "Retrieve All Embedded Resources from HTML Files" 
>>> flag,
>> and
>>> tune the test down to 10 threads with 3000 total iterations, I 
>>> notice a
>> very
>>> strange behavior. The test runs along at a pretty good clip for the 
>>> first
>>> ~600 iterations (about 1 min, 25 seconds into the run), and then it 
>>> just stops making requests for about 35 seconds. Then, it picks back 
>>> up again
>> and
>>> runs for another 1 m 25s, and then stops again for 35 seconds... (NOTE:
>> with
>>> the 10 threads, 3000 total iterations - but with "Retrieve All
>> Embedded..."
>>> disabled - I don't see the 'stop' behavior either - so it isn't 
>>> caused by tuning it down...)
>>> 
>>> I recently added the "Perfmon Metrics Collector" to the test script, 
>>> so I could see if one of the servers was maxed out - but it looks 
>>> like all the servers are idle during the 'stop' period. Likewise, I 
>>> added the Perfmon
>> for
>>> the "localhost" (running the JMeter test) to see if it was swamped - 
>>> but
>> it
>>> too is idle during the 'stop'. I swapped out our network switch (the 
>>> test environment is on an isolated network switch) with a _much_ 
>>> higher
>> capacity
>>> switch - in case there was a network issue, still no change.
>>> 
>>> I'm running out of ideas for things to check - so I thought I'd ask 
>>> you
>> guys
>>> if you have any suggestions of things I should look at.
>>> 
>>> My system consists of:
>>> 
>>>      WinXP - running JMeter 2.4.1 - driving the test script in GUI mode
>>>      Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as 
>>> the web server - using AJP Proxy to Server 2
>>>      Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the 
>>> App Server - connecting through Hibernate to Server 3
>>>      Server 3 - running Red Hat Linux with MySQL 5.x as the DB 
>>> Server
>>> 
>>> All 4 machines are running on a private switched network (32Gbs
>> backplane).
>>> 
>>> The requests are downloading about 3MB total (per thread per 
>>> iteration)
>> over
>>> 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
>>> 
>>> At first I thought it was the network - but the new switch seemed to 
>>> deny that thought (the old switch had a much slower backplane). 
>>> Also, I'm
>> having
>>> no trouble collecting the PerfMon data during the 'stop' period - so 
>>> the network is still functioning just fine...
>>> Then I thought it might be garbage collection on the tomcat - but I
>> watched
>>> the gc.log - and it doesn't do any GCs during the 'stop' period.
>>> Then I thought it might be the garbage collection on the JMeter 
>>> side, so
>> I
>>> started the JMeter.bat from a 'cmd' prompt with gc logging enabled - 
>>> it doesn't do any GCs during the 'stop' period either.
>>> 
>>> The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no 
>>> network I/O, no disk I/O, etc.) during the 'stop' period.
>>> 
>>> The JMeter box (WinXP) is doing very little during that time too (I 
>>> attribute the little bit of activity to displaying the PerfMon 
>>> graphs,
>> and
>>> Remote Desktop display to my desktop computer)...
>>> 
>>> --
>>> Robin D. Wilson
>>> Sr. Director of Web Development
>>> KingsIsle Entertainment, Inc.
>>> VOICE: 512-777-1861
>>> www.KingsIsle.com
>>> 
>>> 
>>> 
>>> 
>>> --------------------------------------------------------------------
>>> - To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>> 
>> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by Deepak Shetty <sh...@gmail.com>.

Hi
On *nix systems you can use kill -3 processid
On windows you run in console mode (jmeter.bat instead of jmeterw) and on
that window you use ctrl + pause/break.
Should print to system.out.

You can also use things like jconsole (but you also need to change the
startup for each jmeter) , connect to each instance and see the threads -
http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html. Almost
all VM monitoring  tools can do this as well) - but all usually add a good
deal of overhead.

I wonder if the pause is because the slaves are returning data to the
master.


regards
deepak

On Wed, Dec 14, 2011 at 2:32 PM, Robin D. Wilson <rw...@gmail.com> wrote:

> I can take a thread dump if someone can tell me how to do it... (Go easy on
> me - I'm a manager, no longer a coder...)
>
> And you were correct, there are no GCs during the 'stop' period on either
> the tomcat servers, or the JMeter client box.
>
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> VOICE: 512-777-1861
> www.KingsIsle.com
>
>
> -----Original Message-----
> From: Deepak Shetty [mailto:shettyd@gmail.com]
> Sent: Wednesday, December 14, 2011 4:20 PM
> To: JMeter Users List
> Subject: Re: Strange 'pause' activity during testing
>
> I think the original post mentioned that GC was not running.
> can you take a thread dump when there is a pause and see what the threads
> are waiting on?
>
>
>
> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com> wrote:
>
> > might be GC.. me be that JMeter's threads are all hung up in your
> > server doing stuff.
> >
> > Regards,
> > Kirk
> > On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
> >
> > > I have a marginally complicated test case that performs a
> 'registration'
> > on
> > > my site. It gets the home page, POSTS a form, gets the response,
> > > POSTS another form, gets that response, and then completes. The test
> > > runs fine with 100 threads, and 30000 iterations - IF I don't
> > > "Retrieve All
> > Embedded
> > > Resources from HTML Files". In this mode, I am really testing the
> > throughput
> > > of my 'tomcat' application, not the other elements of my system.
> > > (I'm assuming that the other elements are being retrieved from our
> > > Content
> > Data
> > > Network instead of our main system in this case.)
> > >
> > > If I enable the "Retrieve All Embedded Resources from HTML Files"
> > > flag,
> > and
> > > tune the test down to 10 threads with 3000 total iterations, I
> > > notice a
> > very
> > > strange behavior. The test runs along at a pretty good clip for the
> > > first
> > > ~600 iterations (about 1 min, 25 seconds into the run), and then it
> > > just stops making requests for about 35 seconds. Then, it picks back
> > > up again
> > and
> > > runs for another 1 m 25s, and then stops again for 35 seconds... (NOTE:
> > with
> > > the 10 threads, 3000 total iterations - but with "Retrieve All
> > Embedded..."
> > > disabled - I don't see the 'stop' behavior either - so it isn't
> > > caused by tuning it down...)
> > >
> > > I recently added the "Perfmon Metrics Collector" to the test script,
> > > so I could see if one of the servers was maxed out - but it looks
> > > like all the servers are idle during the 'stop' period. Likewise, I
> > > added the Perfmon
> > for
> > > the "localhost" (running the JMeter test) to see if it was swamped -
> > > but
> > it
> > > too is idle during the 'stop'. I swapped out our network switch (the
> > > test environment is on an isolated network switch) with a _much_
> > > higher
> > capacity
> > > switch - in case there was a network issue, still no change.
> > >
> > > I'm running out of ideas for things to check - so I thought I'd ask
> > > you
> > guys
> > > if you have any suggestions of things I should look at.
> > >
> > > My system consists of:
> > >
> > >       WinXP - running JMeter 2.4.1 - driving the test script in GUI
> mode
> > >       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as
> > > the web server - using AJP Proxy to Server 2
> > >       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the
> > > App Server - connecting through Hibernate to Server 3
> > >       Server 3 - running Red Hat Linux with MySQL 5.x as the DB
> > > Server
> > >
> > > All 4 machines are running on a private switched network (32Gbs
> > backplane).
> > >
> > > The requests are downloading about 3MB total (per thread per
> > > iteration)
> > over
> > > 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
> > >
> > > At first I thought it was the network - but the new switch seemed to
> > > deny that thought (the old switch had a much slower backplane).
> > > Also, I'm
> > having
> > > no trouble collecting the PerfMon data during the 'stop' period - so
> > > the network is still functioning just fine...
> > > Then I thought it might be garbage collection on the tomcat - but I
> > watched
> > > the gc.log - and it doesn't do any GCs during the 'stop' period.
> > > Then I thought it might be the garbage collection on the JMeter
> > > side, so
> > I
> > > started the JMeter.bat from a 'cmd' prompt with gc logging enabled -
> > > it doesn't do any GCs during the 'stop' period either.
> > >
> > > The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no
> > > network I/O, no disk I/O, etc.) during the 'stop' period.
> > >
> > > The JMeter box (WinXP) is doing very little during that time too (I
> > > attribute the little bit of activity to displaying the PerfMon
> > > graphs,
> > and
> > > Remote Desktop display to my desktop computer)...
> > >
> > > --
> > > Robin D. Wilson
> > > Sr. Director of Web Development
> > > KingsIsle Entertainment, Inc.
> > > VOICE: 512-777-1861
> > > www.KingsIsle.com
> > >
> > >
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > > For additional commands, e-mail: user-help@jmeter.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > For additional commands, e-mail: user-help@jmeter.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>

Re: Strange 'pause' activity during testing

Posted by Philippe Mouawad <ph...@gmail.com>.

On Linux do kill -3 <pid>

Philippe
Http://www.Ubik-Ingenierie.com
On Wednesday, December 14, 2011, Robin D. Wilson <rw...@gmail.com> wrote:
> I can take a thread dump if someone can tell me how to do it... (Go easy
on
> me - I'm a manager, no longer a coder...)
>
> And you were correct, there are no GCs during the 'stop' period on either
> the tomcat servers, or the JMeter client box.
>
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> VOICE: 512-777-1861
> www.KingsIsle.com
>
>
> -----Original Message-----
> From: Deepak Shetty [mailto:shettyd@gmail.com]
> Sent: Wednesday, December 14, 2011 4:20 PM
> To: JMeter Users List
> Subject: Re: Strange 'pause' activity during testing
>
> I think the original post mentioned that GC was not running.
> can you take a thread dump when there is a pause and see what the threads
> are waiting on?
>
>
>
> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com> wrote:
>
>> might be GC.. me be that JMeter's threads are all hung up in your
>> server doing stuff.
>>
>> Regards,
>> Kirk
>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>>
>> > I have a marginally complicated test case that performs a
'registration'
>> on
>> > my site. It gets the home page, POSTS a form, gets the response,
>> > POSTS another form, gets that response, and then completes. The test
>> > runs fine with 100 threads, and 30000 iterations - IF I don't
>> > "Retrieve All
>> Embedded
>> > Resources from HTML Files". In this mode, I am really testing the
>> throughput
>> > of my 'tomcat' application, not the other elements of my system.
>> > (I'm assuming that the other elements are being retrieved from our
>> > Content
>> Data
>> > Network instead of our main system in this case.)
>> >
>> > If I enable the "Retrieve All Embedded Resources from HTML Files"
>> > flag,
>> and
>> > tune the test down to 10 threads with 3000 total iterations, I
>> > notice a
>> very
>> > strange behavior. The test runs along at a pretty good clip for the
>> > first
>> > ~600 iterations (about 1 min, 25 seconds into the run), and then it
>> > just stops making requests for about 35 seconds. Then, it picks back
>> > up again
>> and
>> > runs for another 1 m 25s, and then stops again for 35 seconds... (NOTE:
>> with
>> > the 10 threads, 3000 total iterations - but with "Retrieve All
>> Embedded..."
>> > disabled - I don't see the 'stop' behavior either - so it isn't
>> > caused by tuning it down...)
>> >
>> > I recently added the "Perfmon Metrics Collector" to the test script,
>> > so I could see if one of the servers was maxed out - but it looks
>> > like all the servers are idle during the 'stop' period. Likewise, I
>> > added the Perfmon
>> for
>> > the "localhost" (running the JMeter test) to see if it was swamped -
>> > but
>> it
>> > too is idle during the 'stop'. I swapped out our network switch (the
>> > test environment is on an isolated network switch) with a _much_
>> > higher
>> capacity
>> > switch - in case there was a network issue, still no change.
>> >
>> > I'm running out of ideas for things to check - so I thought I'd ask
>> > you
>> guys
>> > if you have any suggestions of things I should look at.
>> >
>> > My system consists of:
>> >
>> >       WinXP - running JMeter 2.4.1 - driving the test script in GUI
mode
>> >       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as
>> > the web server - using AJP Proxy to Server 2
>> >       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the
>> > App Server - connecting through Hibernate to Server 3
>> >       Server 3 - running Red Hat Linux with MySQL 5.x as the DB
>> > Server
>> >
>> > All 4 machines are running on a private switched network (32Gbs
>> backplane).
>> >
>> > The requests are downloading about 3MB total (per thread per
>> > iteration)
>> over
>> > 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
>> >
>> > At first I thought it was the network - but the new switch seemed to
>> > deny that thought (the old switch had a much slower backplane).
>> > Also, I'm
>> having
>> > no trouble collecting the PerfMon data during the 'stop' period - so
>> > the network is still functioning just fine...
>> > Then I thought it might be garbage collection on the tomcat - but I
>> watched
>> > the gc.log - and it doesn't do any GCs during the 'stop' period.
>> > Then I thought it might be the garbage > > - To unsubscribe, e-mail:
user-unsubscribe@jmeter.apache.org
>> > For additional commands, e-mail: user-help@jmeter.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>

RE: Strange 'pause' activity during testing

Posted by "Robin D. Wilson" <rw...@gmail.com>.

I can take a thread dump if someone can tell me how to do it... (Go easy on
me - I'm a manager, no longer a coder...)

And you were correct, there are no GCs during the 'stop' period on either
the tomcat servers, or the JMeter client box.

--
Robin D. Wilson
Sr. Director of Web Development
KingsIsle Entertainment, Inc.
VOICE: 512-777-1861
www.KingsIsle.com


-----Original Message-----
From: Deepak Shetty [mailto:shettyd@gmail.com] 
Sent: Wednesday, December 14, 2011 4:20 PM
To: JMeter Users List
Subject: Re: Strange 'pause' activity during testing

I think the original post mentioned that GC was not running.
can you take a thread dump when there is a pause and see what the threads
are waiting on?



On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com> wrote:

> might be GC.. me be that JMeter's threads are all hung up in your 
> server doing stuff.
>
> Regards,
> Kirk
> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>
> > I have a marginally complicated test case that performs a 'registration'
> on
> > my site. It gets the home page, POSTS a form, gets the response, 
> > POSTS another form, gets that response, and then completes. The test 
> > runs fine with 100 threads, and 30000 iterations - IF I don't 
> > "Retrieve All
> Embedded
> > Resources from HTML Files". In this mode, I am really testing the
> throughput
> > of my 'tomcat' application, not the other elements of my system. 
> > (I'm assuming that the other elements are being retrieved from our 
> > Content
> Data
> > Network instead of our main system in this case.)
> >
> > If I enable the "Retrieve All Embedded Resources from HTML Files" 
> > flag,
> and
> > tune the test down to 10 threads with 3000 total iterations, I 
> > notice a
> very
> > strange behavior. The test runs along at a pretty good clip for the 
> > first
> > ~600 iterations (about 1 min, 25 seconds into the run), and then it 
> > just stops making requests for about 35 seconds. Then, it picks back 
> > up again
> and
> > runs for another 1 m 25s, and then stops again for 35 seconds... (NOTE:
> with
> > the 10 threads, 3000 total iterations - but with "Retrieve All
> Embedded..."
> > disabled - I don't see the 'stop' behavior either - so it isn't 
> > caused by tuning it down...)
> >
> > I recently added the "Perfmon Metrics Collector" to the test script, 
> > so I could see if one of the servers was maxed out - but it looks 
> > like all the servers are idle during the 'stop' period. Likewise, I 
> > added the Perfmon
> for
> > the "localhost" (running the JMeter test) to see if it was swamped - 
> > but
> it
> > too is idle during the 'stop'. I swapped out our network switch (the 
> > test environment is on an isolated network switch) with a _much_ 
> > higher
> capacity
> > switch - in case there was a network issue, still no change.
> >
> > I'm running out of ideas for things to check - so I thought I'd ask 
> > you
> guys
> > if you have any suggestions of things I should look at.
> >
> > My system consists of:
> >
> >       WinXP - running JMeter 2.4.1 - driving the test script in GUI mode
> >       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as 
> > the web server - using AJP Proxy to Server 2
> >       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the 
> > App Server - connecting through Hibernate to Server 3
> >       Server 3 - running Red Hat Linux with MySQL 5.x as the DB 
> > Server
> >
> > All 4 machines are running on a private switched network (32Gbs
> backplane).
> >
> > The requests are downloading about 3MB total (per thread per 
> > iteration)
> over
> > 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
> >
> > At first I thought it was the network - but the new switch seemed to 
> > deny that thought (the old switch had a much slower backplane). 
> > Also, I'm
> having
> > no trouble collecting the PerfMon data during the 'stop' period - so 
> > the network is still functioning just fine...
> > Then I thought it might be garbage collection on the tomcat - but I
> watched
> > the gc.log - and it doesn't do any GCs during the 'stop' period.
> > Then I thought it might be the garbage collection on the JMeter 
> > side, so
> I
> > started the JMeter.bat from a 'cmd' prompt with gc logging enabled - 
> > it doesn't do any GCs during the 'stop' period either.
> >
> > The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no 
> > network I/O, no disk I/O, etc.) during the 'stop' period.
> >
> > The JMeter box (WinXP) is doing very little during that time too (I 
> > attribute the little bit of activity to displaying the PerfMon 
> > graphs,
> and
> > Remote Desktop display to my desktop computer)...
> >
> > --
> > Robin D. Wilson
> > Sr. Director of Web Development
> > KingsIsle Entertainment, Inc.
> > VOICE: 512-777-1861
> > www.KingsIsle.com
> >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > For additional commands, e-mail: user-help@jmeter.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by Kirk <ki...@gmail.com>.

On Dec 15, 2011, at 1:22 AM, Robin D. Wilson wrote:

> I can use non-GUI mode for testing out the problem, but with the exact same
> test - just switching the 'Retrieve All Embedded...' disabled, the problem
> goes away. 
> 
> The only listeners I have enabled are the summary listener - the PerfMon
> listeners (1 for each server) - and a 'tree' listener, but it only reports
> 'errors'.

I've rarely seen JMeter performance problems during a load test. 

> 
> I can run the test with 'Retrieve All Embedded...' enabled, I see the
> problem every time.

My guess is that getting all embedded is causing the performance problems on your server.


Regards,
Kirk
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by Kirk <ki...@gmail.com>.

a garbage collection log is in order. Use -Xloggc:gc.log -XX:+PrintGCDetails

Again, I suspect that this is the behavior outside of JMeter and JMeter's threads are hung up waiting from something coming back from the web application (not just tomcat). Try the thread dump instructions I previously gave when that happens and post the results here. No point in guessing when a couple of good pieces of data can solve the mystery ;-)

Regards,
Kirk

On Dec 15, 2011, at 8:38 PM, Robin D. Wilson wrote:

> I don't think that's what I'm seeing.
> 
> The test does 600+ iterations (sorry, I miscommunicated earlier - not
> 'samples', but iterations) - that's ~2400 samples with each requisite
> embedded resource set per sample - so ~24000 GET and POST requests are
> fulfilled before the first pause.
> 
> Also, if it were my application pausing, I would expect to see similar
> behavior when I run without the embedded resources (all of the embedded
> resources are being served as static files directly from the Apache server,
> they never even hit the tomcat). Especially considering I can increase the
> thread count 10X and still not see an application pause.
> 
> Without the embedded files, each sampler downloads 50-70KB. With the
> embedded files, each sampler downloads 500-900KB. So there is definitely a
> _lot_ more data (more than 10X) being transferred.
> 
> HOWEVER, when the 'pause' occurs - the network is idle on all machines in
> the environment - so if its busy downloading stuff, it's doing it without
> any bandwidth being used.
> 
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> VOICE: 512-777-1861
> www.KingsIsle.com
> 
> 
> -----Original Message-----
> From: Deepak Goel [mailto:deicool@gmail.com] 
> Sent: Thursday, December 15, 2011 10:35 AM
> To: JMeter Users List
> Subject: Re: Strange 'pause' activity during testing
> 
> Hey
> 
> It looks like common sense...
> 
> The embedded files in the html page are about 6 odd per original page.
> So if you have 10 threads firing the load, Jmeter has to make 60 odd
> connections to the web server to complete these transactions. The embedded
> download from the web server to Jmeter is taking about 300ms.
> 
> So your test makes a pause of 35 odd seconds and then continues because it
> has to download all the embedded stuff and then refire the next 10
> transactions. The test hardly pauses, it is completing the embedded
> download....
> 
> :)
> Deepak
> 
> On 12/15/11, Adrian Speteanu <as...@gmail.com> wrote:
>> I agree that jmeter rarely performs poorly in non-GUI mode, unless you 
>> allocate to much memory to it.
>> 
>> On Thu, Dec 15, 2011 at 1:12 PM, Adrian Speteanu
> <as...@gmail.com>wrote:
>> 
>>> In my experience, "stange pauses" usually points to the application.
>>> 
>>> Of course, you have to make sure its not you (overlook in the test 
>>> script, jmeter's configuration), or the test client environment (OS, 
>>> network). BUT after all that is exhausted is usually the application 
>>> or one of the layers of the system under test (like the database, 
>>> usually more trivial stuff like a small connection pool to the 
>>> database, or GC of the application under test).
>>> 
>>> If you suspect its GC, divide the load you generate with the script 
>>> in 3, and instead of testing with one jmeter instance, use 3 (if you 
>>> can test like this on 3 different machines - the better). Also 
>>> allocate less memory to jmeter now. What's the probability that all 
>>> instances get paused by GC at the same time? Also, if you allocate 
>>> less memory, stop-the-world GC takes less time. If you still have 
>>> response times pauses in the same intervals - it has to be the system
> under test. Note:
>>> 
>>> On Thu, Dec 15, 2011 at 3:51 AM, Robin D. Wilson
>>> <rw...@gmail.com>wrote:
>>> 
>>>> Yes it is. I can turn it off if you think that matters.
>>>> 
>>>> --
>>>> Robin D. Wilson
>>>> 
>>>> 
>>>> 
>>>> On Dec 14, 2011, at 6:42 PM, basim@baassiri.ca wrote:
>>>> 
>>>>> I'm curious to know if the system under test is https
>>>>> 
>>>>> 
>>>>> -----Original Message-----
>>>>> From: sebb <se...@gmail.com>
>>>>> Date: Thu, 15 Dec 2011 00:35:39
>>>>> To: JMeter Users List<us...@jmeter.apache.org>
>>>>> Reply-To: "JMeter Users List" <us...@jmeter.apache.org>
>>>>> Subject: Re: Strange 'pause' activity during testing
>>>>> 
>>>>> On 15 December 2011 00:22, Robin D. Wilson <rw...@gmail.com> wrote:
>>>>>> I can use non-GUI mode for testing out the problem, but with the 
>>>>>> exact
>>>> same
>>>>>> test - just switching the 'Retrieve All Embedded...' disabled, 
>>>>>> the
>>>> problem
>>>>>> goes away.
>>>>>> 
>>>>>> The only listeners I have enabled are the summary listener - the
>>>> PerfMon
>>>>>> listeners (1 for each server) - and a 'tree' listener, but it 
>>>>>> only
>>>> reports
>>>>>> 'errors'.
>>>>>> 
>>>>>> I can run the test with 'Retrieve All Embedded...' enabled, I see 
>>>>>> the problem every time.
>>>>> 
>>>>> Maybe then there is a problem with the scanning of the HTML to 
>>>>> extract the embedded resources, or maybe one of the embedded 
>>>>> resources is a tar-pit.
>>>>> 
>>>>> Do all the expected embedded resources get downloaded?
>>>>> 
>>>>> Are there any unusually long elapsed times for embedded resources?
>>>>> 
>>>>> Or large gaps between the parent sample download completion and 
>>>>> the start of the first embedded resource?
>>>>> 
>>>>> That would suggest the page took a while to parse.
>>>>> 
>>>>> You'll need to select the optiion to "save subresults" in order to 
>>>>> see the embedded samples.
>>>>> 
>>>>>> If I just uncheck the box in the Request Defaults and re-run the 
>>>>>> test,
>>>> it
>>>>>> never happens... So are you suggesting that I've hit some 
>>>>>> arbitrary
>>>> limit of
>>>>>> the GUI mode?
>>>>> 
>>>>> No, just that GUI mode is inherently more resource intensive.
>>>>> 
>>>>> BTW, you wrote you were running JMeter 2.4.1 - that does not 
>>>>> exist, so perhaps you meant the current version, 2.5.1?
>>>>> 
>>>>> 
>>>>>> What I'm wondering is why:
>>>>>> 
>>>>>> 1) It is so consistent - the 'stop' happens at nearly the exact 
>>>>>> same
>>>> point
>>>>>> in every test run, and for nearly the exact same duration - and 
>>>>>> it
>>>> happens
>>>>>> multiple times during the test run, in the same pattern.
>>>>>> 
>>>>>> 2) I can increase the load 10X (even more if I want), but without 
>>>>>> the 'Retrieve All Embedded...' enabled, and it doesn't happen.
>>>>>> 
>>>>>> If it were resource related on the JMeter box, I would guess that
>>>> changing
>>>>>> the JVM memory config would show significant changes in behavior 
>>>>>> (like
>>>> it
>>>>>> would take longer to encounter the first 'stop'). If it were 
>>>>>> resource related on the servers, I would expect to see some sort 
>>>>>> of contention
>>>> on the
>>>>>> servers.
>>>>>> 
>>>>>> Tomorrow I will try the thread dump stuff - to see if there is 
>>>>>> anything obvious in that output.
>>>>>> 
>>>>>> From an "external" view - it looks like JMeter has some sort of 
>>>>>> 'pause'
>>>>>> built into it. Since all machines involved are nearly completely
>>>> 'idle' when
>>>>>> the stop happens - it seems unlikely that I've run out of resources.
>>>> The
>>>>>> other thought is that there is a deadlock somewhere - and 
>>>>>> something
>>>> has to
>>>>>> timeout before it can free up the deadlock and start again. But 
>>>>>> if
>>>> that were
>>>>>> the case, I would expect it to happen at slightly different 
>>>>>> intervals
>>>> in
>>>>>> each test run - and not repeat in the same consistent pattern 
>>>>>> during
>>>> the
>>>>>> test run.
>>>>>> 
>>>>>> Some thoughts on what I think I've ruled out:
>>>>>> 
>>>>>> 1) If it were the network 'backing off' I would expect the 
>>>>>> PerfMon
>>>> listener
>>>>>> to stop updating during the 'stop' - but it trundles happily along.
>>>>>> 2) If it were JVM garbage collections - I would have expected to 
>>>>>> see
>>>> them
>>>>>> running (or at least a 'stop-the-world' gc right before/after the
>>>> 'stop'
>>>>>> period) during the 'stop' period - but there are no gc events at 
>>>>>> all
>>>> during
>>>>>> the 'stop' period.
>>>>>> 3) If it were WinXP resources, I would have expected the PerfMon
>>>> listener to
>>>>>> show some activity during the 'stop' period on the WinXP (JMeter
>>>> client) box
>>>>>> - nothing like that appears.
>>>>>> 
>>>>>> I'll look at the thread dumps tomorrow and get back to you guys.
>>>> Thanks for
>>>>>> helping me ponder this.
>>>>>> 
>>>>>> --
>>>>>> Robin D. Wilson
>>>>>> Sr. Director of Web Development
>>>>>> KingsIsle Entertainment, Inc.
>>>>>> VOICE: 512-777-1861
>>>>>> www.KingsIsle.com
>>>>>> 
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: sebb [mailto:sebbaz@gmail.com]
>>>>>> Sent: Wednesday, December 14, 2011 5:27 PM
>>>>>> To: JMeter Users List
>>>>>> Subject: Re: Strange 'pause' activity during testing
>>>>>> 
>>>>>> On 14 December 2011 22:20, Deepak Shetty <sh...@gmail.com> wrote:
>>>>>>> I think the original post mentioned that GC was not running.
>>>>>>> 
>>>>>>> can you take a thread dump when there is a pause and see what 
>>>>>>> the threads are waiting on?
>>>>>>> 
>>>>>> 
>>>>>> I would also try running JMeter in non-GUI mode; GUI mode is very
>>>> expensive
>>>>>> in resources.
>>>>>> 
>>>>>> Ensure all non-essential Listeners are disabled, see:
>>>>>> 
>>>>>> http://jmeter.apache.org/usermanual/best-practices.html#lean_mean
>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Dec 14, 2011 at 1:56 PM, Kirk 
>>>>>>> <ki...@gmail.com>
>>>> wrote:
>>>>>>> 
>>>>>>>> might be GC.. me be that JMeter's threads are all hung up in 
>>>>>>>> your server doing stuff.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Kirk
>>>>>>>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>>>>>>>> 
>>>>>>>>> I have a marginally complicated test case that performs a
>>>>>> 'registration'
>>>>>>>> on
>>>>>>>>> my site. It gets the home page, POSTS a form, gets the 
>>>>>>>>> response, POSTS another form, gets that response, and then 
>>>>>>>>> completes. The test runs fine with 100 threads, and 30000 
>>>>>>>>> iterations - IF I don't "Retrieve All
>>>>>>>> Embedded
>>>>>>>>> Resources from HTML Files". In this mode, I am really testing 
>>>>>>>>> the
>>>>>>>> throughput
>>>>>>>>> of my 'tomcat' application, not the other elements of my system.
>>>>>>>>> (I'm assuming that the other elements are being retrieved from 
>>>>>>>>> our Content
>>>>>>>> Data
>>>>>>>>> Network instead of our main system in this case.)
>>>>>>>>> 
>>>>>>>>> If I enable the "Retrieve All Embedded Resources from HTML Files"
>>>>>>>>> flag,
>>>>>>>> and
>>>>>>>>> tune the test down to 10 threads with 3000 total iterations, I 
>>>>>>>>> notice a
>>>>>>>> very
>>>>>>>>> strange behavior. The test runs along at a pretty good clip 
>>>>>>>>> for the first
>>>>>>>>> ~600 iterations (about 1 min, 25 seconds into the run), and 
>>>>>>>>> then it just stops making requests for about 35 seconds. Then, 
>>>>>>>>> it picks back up again
>>>>>>>> and
>>>>>>>>> runs for another 1 m 25s, and then stops again for 35 seconds...
>>>> (NOTE:
>>>>>>>> with
>>>>>>>>> the 10 threads, 3000 total iterations - but with "Retrieve All
>>>>>>>> Embedded..."
>>>>>>>>> disabled - I don't see the 'stop' behavior either - so it 
>>>>>>>>> isn't caused by tuning it down...)
>>>>>>>>> 
>>>>>>>>> I recently added the "Perfmon Metrics Collector" to the test 
>>>>>>>>> script, so I could see if one of the servers was maxed out - 
>>>>>>>>> but it looks like all the servers are idle during the 'stop'
> period.
>>>>>>>>> Likewise, I added the Perfmon
>>>>>>>> for
>>>>>>>>> the "localhost" (running the JMeter test) to see if it was 
>>>>>>>>> swamped
>>>>>>>>> - but
>>>>>>>> it
>>>>>>>>> too is idle during the 'stop'. I swapped out our network 
>>>>>>>>> switch (the test environment is on an isolated network switch) 
>>>>>>>>> with a _much_ higher
>>>>>>>> capacity
>>>>>>>>> switch - in case there was a network issue, still no change.
>>>>>>>>> 
>>>>>>>>> I'm running out of ideas for things to check - so I thought 
>>>>>>>>> I'd ask you
>>>>>>>> guys
>>>>>>>>> if you have any suggestions of things I should look at.
>>>>>>>>> 
>>>>>>>>> My system consists of:
>>>>>>>>> 
>>>>>>>>>      WinXP - running JMeter 2.4.1 - driving the test script 
>>>>>>>>> in GUI mode
>>>>>>>>>      Server 1 - running Red Hat Linux, with "Apache (2.2.21)" 
>>>>>>>>> as the web server - using AJP Proxy to Server 2
>>>>>>>>>      Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as 
>>>>>>>>> the App Server - connecting through Hibernate to Server 3
>>>>>>>>>      Server 3 - running Red Hat Linux with MySQL 5.x as the 
>>>>>>>>> DB Server
>>>>>>>>> 
>>>>>>>>> All 4 machines are running on a private switched network 
>>>>>>>>> (32Gbs
>>>>>>>> backplane).
>>>>>>>>> 
>>>>>>>>> The requests are downloading about 3MB total (per thread per
>>>>>>>>> iteration)
>>>>>>>> over
>>>>>>>>> 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
>>>>>>>>> 
>>>>>>>>> At first I thought it was the network - but the new switch 
>>>>>>>>> seemed to deny that thought (the old switch had a much slower
> backplane).
>>>>>>>>> Also, I'm
>>>>>>>> having
>>>>>>>>> no trouble collecting the PerfMon data during the 'stop' 
>>>>>>>>> period - so the network is still functioning just fine...
>>>>>>>>> Then I thought it might be garbage collection on the tomcat - 
>>>>>>>>> but I
>>>>>>>> watched
>>>>>>>>> the gc.log - and it doesn't do any GCs during the 'stop' period.
>>>>>>>>> Then I thought it might be the garbage collection on the 
>>>>>>>>> JMeter side, so
>>>>>>>> I
>>>>>>>>> started the JMeter.bat from a 'cmd' prompt with gc logging 
>>>>>>>>> enabled
>>>>>>>>> - it doesn't do any GCs during the 'stop' period either.
>>>>>>>>> 
>>>>>>>>> The apache, tomcat, and DB are all 'idle' (no CPU to speak of, 
>>>>>>>>> no network I/O, no disk I/O, etc.) during the 'stop' period.
>>>>>>>>> 
>>>>>>>>> The JMeter box (WinXP) is doing very little during that time 
>>>>>>>>> too (I attribute the little bit of activity to displaying the 
>>>>>>>>> PerfMon graphs,
>>>>>>>> and
>>>>>>>>> Remote Desktop display to my desktop computer)...
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Robin D. Wilson
>>>>>>>>> Sr. Director of Web Development KingsIsle Entertainment, Inc.
>>>>>>>>> VOICE: 512-777-1861
>>>>>>>>> www.KingsIsle.com
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
> -------------------------------------------------------------------
>>>>>>>>> -- To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>>> 
>>>>>> 
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>> 
>>>> 
>>> 
>> 
> 
> 
> -- 
> Namaskara~Nalama~Guten Tag~Bonjour
> 
> 
>   --
> Keigu
> 
> Deepak
> +91-9765089593
> deicool@gmail.com
> http://www.simtree.net
> 
> Skype: thumsupdeicool
> Google talk: deicool
> Blog: http://loveandfearless.wordpress.com
> Facebook: http://www.facebook.com/deicool
> 
> "Contribute to the world, environment and more : http://www.gridrepublic.org
> "
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by Deepak Goel <de...@gmail.com>.

Hey

Namaskara~Nalama~Guten Tag~Bonjour

Hmmm...Looks like it is just aggregating and confirming all data which has
been sent to it (The 10X data). It has to join all the previous packets
sent, do a checksum and other checks on it to verify the size and then pass
on the result to be recorded.

Deepak
   --
Keigu

Deepak
+91-9765089593
deicool@gmail.com
http://www.simtree.net

Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool

"Contribute to the world, environment and more : http://www.gridrepublic.org
"



On Fri, Dec 16, 2011 at 1:08 AM, Robin D. Wilson <rw...@gmail.com> wrote:

> I don't think that's what I'm seeing.
>
> The test does 600+ iterations (sorry, I miscommunicated earlier - not
> 'samples', but iterations) - that's ~2400 samples with each requisite
> embedded resource set per sample - so ~24000 GET and POST requests are
> fulfilled before the first pause.
>
> Also, if it were my application pausing, I would expect to see similar
> behavior when I run without the embedded resources (all of the embedded
> resources are being served as static files directly from the Apache server,
> they never even hit the tomcat). Especially considering I can increase the
> thread count 10X and still not see an application pause.
>
> Without the embedded files, each sampler downloads 50-70KB. With the
> embedded files, each sampler downloads 500-900KB. So there is definitely a
> _lot_ more data (more than 10X) being transferred.
>
> HOWEVER, when the 'pause' occurs - the network is idle on all machines in
> the environment - so if its busy downloading stuff, it's doing it without
> any bandwidth being used.
>
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> VOICE: 512-777-1861
> www.KingsIsle.com
>
>
> -----Original Message-----
> From: Deepak Goel [mailto:deicool@gmail.com]
> Sent: Thursday, December 15, 2011 10:35 AM
> To: JMeter Users List
> Subject: Re: Strange 'pause' activity during testing
>
> Hey
>
> It looks like common sense...
>
> The embedded files in the html page are about 6 odd per original page.
> So if you have 10 threads firing the load, Jmeter has to make 60 odd
> connections to the web server to complete these transactions. The embedded
> download from the web server to Jmeter is taking about 300ms.
>
> So your test makes a pause of 35 odd seconds and then continues because it
> has to download all the embedded stuff and then refire the next 10
> transactions. The test hardly pauses, it is completing the embedded
> download....
>
> :)
> Deepak
>
> On 12/15/11, Adrian Speteanu <as...@gmail.com> wrote:
> > I agree that jmeter rarely performs poorly in non-GUI mode, unless you
> > allocate to much memory to it.
> >
> > On Thu, Dec 15, 2011 at 1:12 PM, Adrian Speteanu
> <as...@gmail.com>wrote:
> >
> >> In my experience, "stange pauses" usually points to the application.
> >>
> >> Of course, you have to make sure its not you (overlook in the test
> >> script, jmeter's configuration), or the test client environment (OS,
> >> network). BUT after all that is exhausted is usually the application
> >> or one of the layers of the system under test (like the database,
> >> usually more trivial stuff like a small connection pool to the
> >> database, or GC of the application under test).
> >>
> >> If you suspect its GC, divide the load you generate with the script
> >> in 3, and instead of testing with one jmeter instance, use 3 (if you
> >> can test like this on 3 different machines - the better). Also
> >> allocate less memory to jmeter now. What's the probability that all
> >> instances get paused by GC at the same time? Also, if you allocate
> >> less memory, stop-the-world GC takes less time. If you still have
> >> response times pauses in the same intervals - it has to be the system
> under test. Note:
> >>
> >> On Thu, Dec 15, 2011 at 3:51 AM, Robin D. Wilson
> >> <rw...@gmail.com>wrote:
> >>
> >>> Yes it is. I can turn it off if you think that matters.
> >>>
> >>> --
> >>> Robin D. Wilson
> >>>
> >>>
> >>>
> >>> On Dec 14, 2011, at 6:42 PM, basim@baassiri.ca wrote:
> >>>
> >>> > I'm curious to know if the system under test is https
> >>> >
> >>> >
> >>> > -----Original Message-----
> >>> > From: sebb <se...@gmail.com>
> >>> > Date: Thu, 15 Dec 2011 00:35:39
> >>> > To: JMeter Users List<us...@jmeter.apache.org>
> >>> > Reply-To: "JMeter Users List" <us...@jmeter.apache.org>
> >>> > Subject: Re: Strange 'pause' activity during testing
> >>> >
> >>> > On 15 December 2011 00:22, Robin D. Wilson <rw...@gmail.com>
> wrote:
> >>> >> I can use non-GUI mode for testing out the problem, but with the
> >>> >> exact
> >>> same
> >>> >> test - just switching the 'Retrieve All Embedded...' disabled,
> >>> >> the
> >>> problem
> >>> >> goes away.
> >>> >>
> >>> >> The only listeners I have enabled are the summary listener - the
> >>> PerfMon
> >>> >> listeners (1 for each server) - and a 'tree' listener, but it
> >>> >> only
> >>> reports
> >>> >> 'errors'.
> >>> >>
> >>> >> I can run the test with 'Retrieve All Embedded...' enabled, I see
> >>> >> the problem every time.
> >>> >
> >>> > Maybe then there is a problem with the scanning of the HTML to
> >>> > extract the embedded resources, or maybe one of the embedded
> >>> > resources is a tar-pit.
> >>> >
> >>> > Do all the expected embedded resources get downloaded?
> >>> >
> >>> > Are there any unusually long elapsed times for embedded resources?
> >>> >
> >>> > Or large gaps between the parent sample download completion and
> >>> > the start of the first embedded resource?
> >>> >
> >>> > That would suggest the page took a while to parse.
> >>> >
> >>> > You'll need to select the optiion to "save subresults" in order to
> >>> > see the embedded samples.
> >>> >
> >>> >> If I just uncheck the box in the Request Defaults and re-run the
> >>> >> test,
> >>> it
> >>> >> never happens... So are you suggesting that I've hit some
> >>> >> arbitrary
> >>> limit of
> >>> >> the GUI mode?
> >>> >
> >>> > No, just that GUI mode is inherently more resource intensive.
> >>> >
> >>> > BTW, you wrote you were running JMeter 2.4.1 - that does not
> >>> > exist, so perhaps you meant the current version, 2.5.1?
> >>> >
> >>> >
> >>> >> What I'm wondering is why:
> >>> >>
> >>> >> 1) It is so consistent - the 'stop' happens at nearly the exact
> >>> >> same
> >>> point
> >>> >> in every test run, and for nearly the exact same duration - and
> >>> >> it
> >>> happens
> >>> >> multiple times during the test run, in the same pattern.
> >>> >>
> >>> >> 2) I can increase the load 10X (even more if I want), but without
> >>> >> the 'Retrieve All Embedded...' enabled, and it doesn't happen.
> >>> >>
> >>> >> If it were resource related on the JMeter box, I would guess that
> >>> changing
> >>> >> the JVM memory config would show significant changes in behavior
> >>> >> (like
> >>> it
> >>> >> would take longer to encounter the first 'stop'). If it were
> >>> >> resource related on the servers, I would expect to see some sort
> >>> >> of contention
> >>> on the
> >>> >> servers.
> >>> >>
> >>> >> Tomorrow I will try the thread dump stuff - to see if there is
> >>> >> anything obvious in that output.
> >>> >>
> >>> >> From an "external" view - it looks like JMeter has some sort of
> >>> >> 'pause'
> >>> >> built into it. Since all machines involved are nearly completely
> >>> 'idle' when
> >>> >> the stop happens - it seems unlikely that I've run out of resources.
> >>> The
> >>> >> other thought is that there is a deadlock somewhere - and
> >>> >> something
> >>> has to
> >>> >> timeout before it can free up the deadlock and start again. But
> >>> >> if
> >>> that were
> >>> >> the case, I would expect it to happen at slightly different
> >>> >> intervals
> >>> in
> >>> >> each test run - and not repeat in the same consistent pattern
> >>> >> during
> >>> the
> >>> >> test run.
> >>> >>
> >>> >> Some thoughts on what I think I've ruled out:
> >>> >>
> >>> >> 1) If it were the network 'backing off' I would expect the
> >>> >> PerfMon
> >>> listener
> >>> >> to stop updating during the 'stop' - but it trundles happily along.
> >>> >> 2) If it were JVM garbage collections - I would have expected to
> >>> >> see
> >>> them
> >>> >> running (or at least a 'stop-the-world' gc right before/after the
> >>> 'stop'
> >>> >> period) during the 'stop' period - but there are no gc events at
> >>> >> all
> >>> during
> >>> >> the 'stop' period.
> >>> >> 3) If it were WinXP resources, I would have expected the PerfMon
> >>> listener to
> >>> >> show some activity during the 'stop' period on the WinXP (JMeter
> >>> client) box
> >>> >> - nothing like that appears.
> >>> >>
> >>> >> I'll look at the thread dumps tomorrow and get back to you guys.
> >>> Thanks for
> >>> >> helping me ponder this.
> >>> >>
> >>> >> --
> >>> >> Robin D. Wilson
> >>> >> Sr. Director of Web Development
> >>> >> KingsIsle Entertainment, Inc.
> >>> >> VOICE: 512-777-1861
> >>> >> www.KingsIsle.com
> >>> >>
> >>> >>
> >>> >> -----Original Message-----
> >>> >> From: sebb [mailto:sebbaz@gmail.com]
> >>> >> Sent: Wednesday, December 14, 2011 5:27 PM
> >>> >> To: JMeter Users List
> >>> >> Subject: Re: Strange 'pause' activity during testing
> >>> >>
> >>> >> On 14 December 2011 22:20, Deepak Shetty <sh...@gmail.com> wrote:
> >>> >>> I think the original post mentioned that GC was not running.
> >>> >>>
> >>> >>> can you take a thread dump when there is a pause and see what
> >>> >>> the threads are waiting on?
> >>> >>>
> >>> >>
> >>> >> I would also try running JMeter in non-GUI mode; GUI mode is very
> >>> expensive
> >>> >> in resources.
> >>> >>
> >>> >> Ensure all non-essential Listeners are disabled, see:
> >>> >>
> >>> >> http://jmeter.apache.org/usermanual/best-practices.html#lean_mean
> >>> >>
> >>> >>>
> >>> >>> On Wed, Dec 14, 2011 at 1:56 PM, Kirk
> >>> >>> <ki...@gmail.com>
> >>> wrote:
> >>> >>>
> >>> >>>> might be GC.. me be that JMeter's threads are all hung up in
> >>> >>>> your server doing stuff.
> >>> >>>>
> >>> >>>> Regards,
> >>> >>>> Kirk
> >>> >>>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
> >>> >>>>
> >>> >>>>> I have a marginally complicated test case that performs a
> >>> >> 'registration'
> >>> >>>> on
> >>> >>>>> my site. It gets the home page, POSTS a form, gets the
> >>> >>>>> response, POSTS another form, gets that response, and then
> >>> >>>>> completes. The test runs fine with 100 threads, and 30000
> >>> >>>>> iterations - IF I don't "Retrieve All
> >>> >>>> Embedded
> >>> >>>>> Resources from HTML Files". In this mode, I am really testing
> >>> >>>>> the
> >>> >>>> throughput
> >>> >>>>> of my 'tomcat' application, not the other elements of my system.
> >>> >>>>> (I'm assuming that the other elements are being retrieved from
> >>> >>>>> our Content
> >>> >>>> Data
> >>> >>>>> Network instead of our main system in this case.)
> >>> >>>>>
> >>> >>>>> If I enable the "Retrieve All Embedded Resources from HTML Files"
> >>> >>>>> flag,
> >>> >>>> and
> >>> >>>>> tune the test down to 10 threads with 3000 total iterations, I
> >>> >>>>> notice a
> >>> >>>> very
> >>> >>>>> strange behavior. The test runs along at a pretty good clip
> >>> >>>>> for the first
> >>> >>>>> ~600 iterations (about 1 min, 25 seconds into the run), and
> >>> >>>>> then it just stops making requests for about 35 seconds. Then,
> >>> >>>>> it picks back up again
> >>> >>>> and
> >>> >>>>> runs for another 1 m 25s, and then stops again for 35 seconds...
> >>> (NOTE:
> >>> >>>> with
> >>> >>>>> the 10 threads, 3000 total iterations - but with "Retrieve All
> >>> >>>> Embedded..."
> >>> >>>>> disabled - I don't see the 'stop' behavior either - so it
> >>> >>>>> isn't caused by tuning it down...)
> >>> >>>>>
> >>> >>>>> I recently added the "Perfmon Metrics Collector" to the test
> >>> >>>>> script, so I could see if one of the servers was maxed out -
> >>> >>>>> but it looks like all the servers are idle during the 'stop'
> period.
> >>> >>>>> Likewise, I added the Perfmon
> >>> >>>> for
> >>> >>>>> the "localhost" (running the JMeter test) to see if it was
> >>> >>>>> swamped
> >>> >>>>> - but
> >>> >>>> it
> >>> >>>>> too is idle during the 'stop'. I swapped out our network
> >>> >>>>> switch (the test environment is on an isolated network switch)
> >>> >>>>> with a _much_ higher
> >>> >>>> capacity
> >>> >>>>> switch - in case there was a network issue, still no change.
> >>> >>>>>
> >>> >>>>> I'm running out of ideas for things to check - so I thought
> >>> >>>>> I'd ask you
> >>> >>>> guys
> >>> >>>>> if you have any suggestions of things I should look at.
> >>> >>>>>
> >>> >>>>> My system consists of:
> >>> >>>>>
> >>> >>>>>       WinXP - running JMeter 2.4.1 - driving the test script
> >>> >>>>> in GUI mode
> >>> >>>>>       Server 1 - running Red Hat Linux, with "Apache (2.2.21)"
> >>> >>>>> as the web server - using AJP Proxy to Server 2
> >>> >>>>>       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as
> >>> >>>>> the App Server - connecting through Hibernate to Server 3
> >>> >>>>>       Server 3 - running Red Hat Linux with MySQL 5.x as the
> >>> >>>>> DB Server
> >>> >>>>>
> >>> >>>>> All 4 machines are running on a private switched network
> >>> >>>>> (32Gbs
> >>> >>>> backplane).
> >>> >>>>>
> >>> >>>>> The requests are downloading about 3MB total (per thread per
> >>> >>>>> iteration)
> >>> >>>> over
> >>> >>>>> 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
> >>> >>>>>
> >>> >>>>> At first I thought it was the network - but the new switch
> >>> >>>>> seemed to deny that thought (the old switch had a much slower
> backplane).
> >>> >>>>> Also, I'm
> >>> >>>> having
> >>> >>>>> no trouble collecting the PerfMon data during the 'stop'
> >>> >>>>> period - so the network is still functioning just fine...
> >>> >>>>> Then I thought it might be garbage collection on the tomcat -
> >>> >>>>> but I
> >>> >>>> watched
> >>> >>>>> the gc.log - and it doesn't do any GCs during the 'stop' period.
> >>> >>>>> Then I thought it might be the garbage collection on the
> >>> >>>>> JMeter side, so
> >>> >>>> I
> >>> >>>>> started the JMeter.bat from a 'cmd' prompt with gc logging
> >>> >>>>> enabled
> >>> >>>>> - it doesn't do any GCs during the 'stop' period either.
> >>> >>>>>
> >>> >>>>> The apache, tomcat, and DB are all 'idle' (no CPU to speak of,
> >>> >>>>> no network I/O, no disk I/O, etc.) during the 'stop' period.
> >>> >>>>>
> >>> >>>>> The JMeter box (WinXP) is doing very little during that time
> >>> >>>>> too (I attribute the little bit of activity to displaying the
> >>> >>>>> PerfMon graphs,
> >>> >>>> and
> >>> >>>>> Remote Desktop display to my desktop computer)...
> >>> >>>>>
> >>> >>>>> --
> >>> >>>>> Robin D. Wilson
> >>> >>>>> Sr. Director of Web Development KingsIsle Entertainment, Inc.
> >>> >>>>> VOICE: 512-777-1861
> >>> >>>>> www.KingsIsle.com
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> -------------------------------------------------------------------
> >>> >>>>> -- To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >>> >>>>> For additional commands, e-mail: user-help@jmeter.apache.org
> >>> >>>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> ---------------------------------------------------------------------
> >>> >>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >>> >>>> For additional commands, e-mail: user-help@jmeter.apache.org
> >>> >>>>
> >>> >>>>
> >>> >>
> >>> >>
> ---------------------------------------------------------------------
> >>> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >>> >> For additional commands, e-mail: user-help@jmeter.apache.org
> >>> >>
> >>> >>
> >>> >>
> ---------------------------------------------------------------------
> >>> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >>> >> For additional commands, e-mail: user-help@jmeter.apache.org
> >>> >>
> >>> >
> >>> > ---------------------------------------------------------------------
> >>> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >>> > For additional commands, e-mail: user-help@jmeter.apache.org
> >>> >
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >>> For additional commands, e-mail: user-help@jmeter.apache.org
> >>>
> >>>
> >>
> >
>
>
> --
> Namaskara~Nalama~Guten Tag~Bonjour
>
>
>   --
> Keigu
>
> Deepak
> +91-9765089593
> deicool@gmail.com
> http://www.simtree.net
>
> Skype: thumsupdeicool
> Google talk: deicool
> Blog: http://loveandfearless.wordpress.com
> Facebook: http://www.facebook.com/deicool
>
> "Contribute to the world, environment and more :
> http://www.gridrepublic.org
> "
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>

RE: Strange 'pause' activity during testing

Posted by "Robin D. Wilson" <rw...@gmail.com>.

I don't think that's what I'm seeing.

The test does 600+ iterations (sorry, I miscommunicated earlier - not
'samples', but iterations) - that's ~2400 samples with each requisite
embedded resource set per sample - so ~24000 GET and POST requests are
fulfilled before the first pause.

Also, if it were my application pausing, I would expect to see similar
behavior when I run without the embedded resources (all of the embedded
resources are being served as static files directly from the Apache server,
they never even hit the tomcat). Especially considering I can increase the
thread count 10X and still not see an application pause.

Without the embedded files, each sampler downloads 50-70KB. With the
embedded files, each sampler downloads 500-900KB. So there is definitely a
_lot_ more data (more than 10X) being transferred.

HOWEVER, when the 'pause' occurs - the network is idle on all machines in
the environment - so if its busy downloading stuff, it's doing it without
any bandwidth being used.

--
Robin D. Wilson
Sr. Director of Web Development
KingsIsle Entertainment, Inc.
VOICE: 512-777-1861
www.KingsIsle.com


-----Original Message-----
From: Deepak Goel [mailto:deicool@gmail.com] 
Sent: Thursday, December 15, 2011 10:35 AM
To: JMeter Users List
Subject: Re: Strange 'pause' activity during testing

Hey

It looks like common sense...

The embedded files in the html page are about 6 odd per original page.
So if you have 10 threads firing the load, Jmeter has to make 60 odd
connections to the web server to complete these transactions. The embedded
download from the web server to Jmeter is taking about 300ms.

So your test makes a pause of 35 odd seconds and then continues because it
has to download all the embedded stuff and then refire the next 10
transactions. The test hardly pauses, it is completing the embedded
download....

:)
Deepak

On 12/15/11, Adrian Speteanu <as...@gmail.com> wrote:
> I agree that jmeter rarely performs poorly in non-GUI mode, unless you 
> allocate to much memory to it.
>
> On Thu, Dec 15, 2011 at 1:12 PM, Adrian Speteanu
<as...@gmail.com>wrote:
>
>> In my experience, "stange pauses" usually points to the application.
>>
>> Of course, you have to make sure its not you (overlook in the test 
>> script, jmeter's configuration), or the test client environment (OS, 
>> network). BUT after all that is exhausted is usually the application 
>> or one of the layers of the system under test (like the database, 
>> usually more trivial stuff like a small connection pool to the 
>> database, or GC of the application under test).
>>
>> If you suspect its GC, divide the load you generate with the script 
>> in 3, and instead of testing with one jmeter instance, use 3 (if you 
>> can test like this on 3 different machines - the better). Also 
>> allocate less memory to jmeter now. What's the probability that all 
>> instances get paused by GC at the same time? Also, if you allocate 
>> less memory, stop-the-world GC takes less time. If you still have 
>> response times pauses in the same intervals - it has to be the system
under test. Note:
>>
>> On Thu, Dec 15, 2011 at 3:51 AM, Robin D. Wilson
>> <rw...@gmail.com>wrote:
>>
>>> Yes it is. I can turn it off if you think that matters.
>>>
>>> --
>>> Robin D. Wilson
>>>
>>>
>>>
>>> On Dec 14, 2011, at 6:42 PM, basim@baassiri.ca wrote:
>>>
>>> > I'm curious to know if the system under test is https
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: sebb <se...@gmail.com>
>>> > Date: Thu, 15 Dec 2011 00:35:39
>>> > To: JMeter Users List<us...@jmeter.apache.org>
>>> > Reply-To: "JMeter Users List" <us...@jmeter.apache.org>
>>> > Subject: Re: Strange 'pause' activity during testing
>>> >
>>> > On 15 December 2011 00:22, Robin D. Wilson <rw...@gmail.com> wrote:
>>> >> I can use non-GUI mode for testing out the problem, but with the 
>>> >> exact
>>> same
>>> >> test - just switching the 'Retrieve All Embedded...' disabled, 
>>> >> the
>>> problem
>>> >> goes away.
>>> >>
>>> >> The only listeners I have enabled are the summary listener - the
>>> PerfMon
>>> >> listeners (1 for each server) - and a 'tree' listener, but it 
>>> >> only
>>> reports
>>> >> 'errors'.
>>> >>
>>> >> I can run the test with 'Retrieve All Embedded...' enabled, I see 
>>> >> the problem every time.
>>> >
>>> > Maybe then there is a problem with the scanning of the HTML to 
>>> > extract the embedded resources, or maybe one of the embedded 
>>> > resources is a tar-pit.
>>> >
>>> > Do all the expected embedded resources get downloaded?
>>> >
>>> > Are there any unusually long elapsed times for embedded resources?
>>> >
>>> > Or large gaps between the parent sample download completion and 
>>> > the start of the first embedded resource?
>>> >
>>> > That would suggest the page took a while to parse.
>>> >
>>> > You'll need to select the optiion to "save subresults" in order to 
>>> > see the embedded samples.
>>> >
>>> >> If I just uncheck the box in the Request Defaults and re-run the 
>>> >> test,
>>> it
>>> >> never happens... So are you suggesting that I've hit some 
>>> >> arbitrary
>>> limit of
>>> >> the GUI mode?
>>> >
>>> > No, just that GUI mode is inherently more resource intensive.
>>> >
>>> > BTW, you wrote you were running JMeter 2.4.1 - that does not 
>>> > exist, so perhaps you meant the current version, 2.5.1?
>>> >
>>> >
>>> >> What I'm wondering is why:
>>> >>
>>> >> 1) It is so consistent - the 'stop' happens at nearly the exact 
>>> >> same
>>> point
>>> >> in every test run, and for nearly the exact same duration - and 
>>> >> it
>>> happens
>>> >> multiple times during the test run, in the same pattern.
>>> >>
>>> >> 2) I can increase the load 10X (even more if I want), but without 
>>> >> the 'Retrieve All Embedded...' enabled, and it doesn't happen.
>>> >>
>>> >> If it were resource related on the JMeter box, I would guess that
>>> changing
>>> >> the JVM memory config would show significant changes in behavior 
>>> >> (like
>>> it
>>> >> would take longer to encounter the first 'stop'). If it were 
>>> >> resource related on the servers, I would expect to see some sort 
>>> >> of contention
>>> on the
>>> >> servers.
>>> >>
>>> >> Tomorrow I will try the thread dump stuff - to see if there is 
>>> >> anything obvious in that output.
>>> >>
>>> >> From an "external" view - it looks like JMeter has some sort of 
>>> >> 'pause'
>>> >> built into it. Since all machines involved are nearly completely
>>> 'idle' when
>>> >> the stop happens - it seems unlikely that I've run out of resources.
>>> The
>>> >> other thought is that there is a deadlock somewhere - and 
>>> >> something
>>> has to
>>> >> timeout before it can free up the deadlock and start again. But 
>>> >> if
>>> that were
>>> >> the case, I would expect it to happen at slightly different 
>>> >> intervals
>>> in
>>> >> each test run - and not repeat in the same consistent pattern 
>>> >> during
>>> the
>>> >> test run.
>>> >>
>>> >> Some thoughts on what I think I've ruled out:
>>> >>
>>> >> 1) If it were the network 'backing off' I would expect the 
>>> >> PerfMon
>>> listener
>>> >> to stop updating during the 'stop' - but it trundles happily along.
>>> >> 2) If it were JVM garbage collections - I would have expected to 
>>> >> see
>>> them
>>> >> running (or at least a 'stop-the-world' gc right before/after the
>>> 'stop'
>>> >> period) during the 'stop' period - but there are no gc events at 
>>> >> all
>>> during
>>> >> the 'stop' period.
>>> >> 3) If it were WinXP resources, I would have expected the PerfMon
>>> listener to
>>> >> show some activity during the 'stop' period on the WinXP (JMeter
>>> client) box
>>> >> - nothing like that appears.
>>> >>
>>> >> I'll look at the thread dumps tomorrow and get back to you guys.
>>> Thanks for
>>> >> helping me ponder this.
>>> >>
>>> >> --
>>> >> Robin D. Wilson
>>> >> Sr. Director of Web Development
>>> >> KingsIsle Entertainment, Inc.
>>> >> VOICE: 512-777-1861
>>> >> www.KingsIsle.com
>>> >>
>>> >>
>>> >> -----Original Message-----
>>> >> From: sebb [mailto:sebbaz@gmail.com]
>>> >> Sent: Wednesday, December 14, 2011 5:27 PM
>>> >> To: JMeter Users List
>>> >> Subject: Re: Strange 'pause' activity during testing
>>> >>
>>> >> On 14 December 2011 22:20, Deepak Shetty <sh...@gmail.com> wrote:
>>> >>> I think the original post mentioned that GC was not running.
>>> >>>
>>> >>> can you take a thread dump when there is a pause and see what 
>>> >>> the threads are waiting on?
>>> >>>
>>> >>
>>> >> I would also try running JMeter in non-GUI mode; GUI mode is very
>>> expensive
>>> >> in resources.
>>> >>
>>> >> Ensure all non-essential Listeners are disabled, see:
>>> >>
>>> >> http://jmeter.apache.org/usermanual/best-practices.html#lean_mean
>>> >>
>>> >>>
>>> >>> On Wed, Dec 14, 2011 at 1:56 PM, Kirk 
>>> >>> <ki...@gmail.com>
>>> wrote:
>>> >>>
>>> >>>> might be GC.. me be that JMeter's threads are all hung up in 
>>> >>>> your server doing stuff.
>>> >>>>
>>> >>>> Regards,
>>> >>>> Kirk
>>> >>>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>>> >>>>
>>> >>>>> I have a marginally complicated test case that performs a
>>> >> 'registration'
>>> >>>> on
>>> >>>>> my site. It gets the home page, POSTS a form, gets the 
>>> >>>>> response, POSTS another form, gets that response, and then 
>>> >>>>> completes. The test runs fine with 100 threads, and 30000 
>>> >>>>> iterations - IF I don't "Retrieve All
>>> >>>> Embedded
>>> >>>>> Resources from HTML Files". In this mode, I am really testing 
>>> >>>>> the
>>> >>>> throughput
>>> >>>>> of my 'tomcat' application, not the other elements of my system.
>>> >>>>> (I'm assuming that the other elements are being retrieved from 
>>> >>>>> our Content
>>> >>>> Data
>>> >>>>> Network instead of our main system in this case.)
>>> >>>>>
>>> >>>>> If I enable the "Retrieve All Embedded Resources from HTML Files"
>>> >>>>> flag,
>>> >>>> and
>>> >>>>> tune the test down to 10 threads with 3000 total iterations, I 
>>> >>>>> notice a
>>> >>>> very
>>> >>>>> strange behavior. The test runs along at a pretty good clip 
>>> >>>>> for the first
>>> >>>>> ~600 iterations (about 1 min, 25 seconds into the run), and 
>>> >>>>> then it just stops making requests for about 35 seconds. Then, 
>>> >>>>> it picks back up again
>>> >>>> and
>>> >>>>> runs for another 1 m 25s, and then stops again for 35 seconds...
>>> (NOTE:
>>> >>>> with
>>> >>>>> the 10 threads, 3000 total iterations - but with "Retrieve All
>>> >>>> Embedded..."
>>> >>>>> disabled - I don't see the 'stop' behavior either - so it 
>>> >>>>> isn't caused by tuning it down...)
>>> >>>>>
>>> >>>>> I recently added the "Perfmon Metrics Collector" to the test 
>>> >>>>> script, so I could see if one of the servers was maxed out - 
>>> >>>>> but it looks like all the servers are idle during the 'stop'
period.
>>> >>>>> Likewise, I added the Perfmon
>>> >>>> for
>>> >>>>> the "localhost" (running the JMeter test) to see if it was 
>>> >>>>> swamped
>>> >>>>> - but
>>> >>>> it
>>> >>>>> too is idle during the 'stop'. I swapped out our network 
>>> >>>>> switch (the test environment is on an isolated network switch) 
>>> >>>>> with a _much_ higher
>>> >>>> capacity
>>> >>>>> switch - in case there was a network issue, still no change.
>>> >>>>>
>>> >>>>> I'm running out of ideas for things to check - so I thought 
>>> >>>>> I'd ask you
>>> >>>> guys
>>> >>>>> if you have any suggestions of things I should look at.
>>> >>>>>
>>> >>>>> My system consists of:
>>> >>>>>
>>> >>>>>       WinXP - running JMeter 2.4.1 - driving the test script 
>>> >>>>> in GUI mode
>>> >>>>>       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" 
>>> >>>>> as the web server - using AJP Proxy to Server 2
>>> >>>>>       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as 
>>> >>>>> the App Server - connecting through Hibernate to Server 3
>>> >>>>>       Server 3 - running Red Hat Linux with MySQL 5.x as the 
>>> >>>>> DB Server
>>> >>>>>
>>> >>>>> All 4 machines are running on a private switched network 
>>> >>>>> (32Gbs
>>> >>>> backplane).
>>> >>>>>
>>> >>>>> The requests are downloading about 3MB total (per thread per
>>> >>>>> iteration)
>>> >>>> over
>>> >>>>> 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
>>> >>>>>
>>> >>>>> At first I thought it was the network - but the new switch 
>>> >>>>> seemed to deny that thought (the old switch had a much slower
backplane).
>>> >>>>> Also, I'm
>>> >>>> having
>>> >>>>> no trouble collecting the PerfMon data during the 'stop' 
>>> >>>>> period - so the network is still functioning just fine...
>>> >>>>> Then I thought it might be garbage collection on the tomcat - 
>>> >>>>> but I
>>> >>>> watched
>>> >>>>> the gc.log - and it doesn't do any GCs during the 'stop' period.
>>> >>>>> Then I thought it might be the garbage collection on the 
>>> >>>>> JMeter side, so
>>> >>>> I
>>> >>>>> started the JMeter.bat from a 'cmd' prompt with gc logging 
>>> >>>>> enabled
>>> >>>>> - it doesn't do any GCs during the 'stop' period either.
>>> >>>>>
>>> >>>>> The apache, tomcat, and DB are all 'idle' (no CPU to speak of, 
>>> >>>>> no network I/O, no disk I/O, etc.) during the 'stop' period.
>>> >>>>>
>>> >>>>> The JMeter box (WinXP) is doing very little during that time 
>>> >>>>> too (I attribute the little bit of activity to displaying the 
>>> >>>>> PerfMon graphs,
>>> >>>> and
>>> >>>>> Remote Desktop display to my desktop computer)...
>>> >>>>>
>>> >>>>> --
>>> >>>>> Robin D. Wilson
>>> >>>>> Sr. Director of Web Development KingsIsle Entertainment, Inc.
>>> >>>>> VOICE: 512-777-1861
>>> >>>>> www.KingsIsle.com
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
-------------------------------------------------------------------
>>> >>>>> -- To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> >>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>>
---------------------------------------------------------------------
>>> >>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> >>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>> >>>>
>>> >>>>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> >> For additional commands, e-mail: user-help@jmeter.apache.org
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> >> For additional commands, e-mail: user-help@jmeter.apache.org
>>> >>
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> > For additional commands, e-mail: user-help@jmeter.apache.org
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>
>>>
>>
>


-- 
Namaskara~Nalama~Guten Tag~Bonjour


   --
Keigu

Deepak
+91-9765089593
deicool@gmail.com
http://www.simtree.net

Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool

"Contribute to the world, environment and more : http://www.gridrepublic.org
"

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by Deepak Goel <de...@gmail.com>.

Hey

It looks like common sense...

The embedded files in the html page are about 6 odd per original page.
So if you have 10 threads firing the load, Jmeter has to make 60 odd
connections to the web server to complete these transactions. The
embedded download from the web server to Jmeter is taking about 300ms.

So your test makes a pause of 35 odd seconds and then continues
because it has to download all the embedded stuff and then refire the
next 10 transactions. The test hardly pauses, it is completing the
embedded download....

:)
Deepak

On 12/15/11, Adrian Speteanu <as...@gmail.com> wrote:
> I agree that jmeter rarely performs poorly in non-GUI mode, unless you
> allocate to much memory to it.
>
> On Thu, Dec 15, 2011 at 1:12 PM, Adrian Speteanu <as...@gmail.com>wrote:
>
>> In my experience, "stange pauses" usually points to the application.
>>
>> Of course, you have to make sure its not you (overlook in the test script,
>> jmeter's configuration), or the test client environment (OS, network). BUT
>> after all that is exhausted is usually the application or one of the
>> layers
>> of the system under test (like the database, usually more trivial stuff
>> like a small connection pool to the database, or GC of the application
>> under test).
>>
>> If you suspect its GC, divide the load you generate with the script in 3,
>> and instead of testing with one jmeter instance, use 3 (if you can test
>> like this on 3 different machines - the better). Also allocate less memory
>> to jmeter now. What's the probability that all instances get paused by GC
>> at the same time? Also, if you allocate less memory, stop-the-world GC
>> takes less time. If you still have response times pauses in the same
>> intervals - it has to be the system under test. Note:
>>
>> On Thu, Dec 15, 2011 at 3:51 AM, Robin D. Wilson
>> <rw...@gmail.com>wrote:
>>
>>> Yes it is. I can turn it off if you think that matters.
>>>
>>> --
>>> Robin D. Wilson
>>>
>>>
>>>
>>> On Dec 14, 2011, at 6:42 PM, basim@baassiri.ca wrote:
>>>
>>> > I'm curious to know if the system under test is https
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: sebb <se...@gmail.com>
>>> > Date: Thu, 15 Dec 2011 00:35:39
>>> > To: JMeter Users List<us...@jmeter.apache.org>
>>> > Reply-To: "JMeter Users List" <us...@jmeter.apache.org>
>>> > Subject: Re: Strange 'pause' activity during testing
>>> >
>>> > On 15 December 2011 00:22, Robin D. Wilson <rw...@gmail.com> wrote:
>>> >> I can use non-GUI mode for testing out the problem, but with the exact
>>> same
>>> >> test - just switching the 'Retrieve All Embedded...' disabled, the
>>> problem
>>> >> goes away.
>>> >>
>>> >> The only listeners I have enabled are the summary listener - the
>>> PerfMon
>>> >> listeners (1 for each server) - and a 'tree' listener, but it only
>>> reports
>>> >> 'errors'.
>>> >>
>>> >> I can run the test with 'Retrieve All Embedded...' enabled, I see the
>>> >> problem every time.
>>> >
>>> > Maybe then there is a problem with the scanning of the HTML to extract
>>> > the embedded resources, or maybe one of the embedded resources is a
>>> > tar-pit.
>>> >
>>> > Do all the expected embedded resources get downloaded?
>>> >
>>> > Are there any unusually long elapsed times for embedded resources?
>>> >
>>> > Or large gaps between the parent sample download completion and the
>>> > start of the first embedded resource?
>>> >
>>> > That would suggest the page took a while to parse.
>>> >
>>> > You'll need to select the optiion to "save subresults" in order to see
>>> > the embedded samples.
>>> >
>>> >> If I just uncheck the box in the Request Defaults and re-run the test,
>>> it
>>> >> never happens... So are you suggesting that I've hit some arbitrary
>>> limit of
>>> >> the GUI mode?
>>> >
>>> > No, just that GUI mode is inherently more resource intensive.
>>> >
>>> > BTW, you wrote you were running JMeter 2.4.1 - that does not exist, so
>>> > perhaps you meant the current version, 2.5.1?
>>> >
>>> >
>>> >> What I'm wondering is why:
>>> >>
>>> >> 1) It is so consistent - the 'stop' happens at nearly the exact same
>>> point
>>> >> in every test run, and for nearly the exact same duration - and it
>>> happens
>>> >> multiple times during the test run, in the same pattern.
>>> >>
>>> >> 2) I can increase the load 10X (even more if I want), but without the
>>> >> 'Retrieve All Embedded...' enabled, and it doesn’t happen.
>>> >>
>>> >> If it were resource related on the JMeter box, I would guess that
>>> changing
>>> >> the JVM memory config would show significant changes in behavior (like
>>> it
>>> >> would take longer to encounter the first 'stop'). If it were resource
>>> >> related on the servers, I would expect to see some sort of contention
>>> on the
>>> >> servers.
>>> >>
>>> >> Tomorrow I will try the thread dump stuff - to see if there is
>>> >> anything
>>> >> obvious in that output.
>>> >>
>>> >> From an "external" view - it looks like JMeter has some sort of
>>> >> 'pause'
>>> >> built into it. Since all machines involved are nearly completely
>>> 'idle' when
>>> >> the stop happens - it seems unlikely that I've run out of resources.
>>> The
>>> >> other thought is that there is a deadlock somewhere - and something
>>> has to
>>> >> timeout before it can free up the deadlock and start again. But if
>>> that were
>>> >> the case, I would expect it to happen at slightly different intervals
>>> in
>>> >> each test run - and not repeat in the same consistent pattern during
>>> the
>>> >> test run.
>>> >>
>>> >> Some thoughts on what I think I've ruled out:
>>> >>
>>> >> 1) If it were the network 'backing off' I would expect the PerfMon
>>> listener
>>> >> to stop updating during the 'stop' - but it trundles happily along.
>>> >> 2) If it were JVM garbage collections - I would have expected to see
>>> them
>>> >> running (or at least a 'stop-the-world' gc right before/after the
>>> 'stop'
>>> >> period) during the 'stop' period - but there are no gc events at all
>>> during
>>> >> the 'stop' period.
>>> >> 3) If it were WinXP resources, I would have expected the PerfMon
>>> listener to
>>> >> show some activity during the 'stop' period on the WinXP (JMeter
>>> client) box
>>> >> - nothing like that appears.
>>> >>
>>> >> I'll look at the thread dumps tomorrow and get back to you guys.
>>> Thanks for
>>> >> helping me ponder this.
>>> >>
>>> >> --
>>> >> Robin D. Wilson
>>> >> Sr. Director of Web Development
>>> >> KingsIsle Entertainment, Inc.
>>> >> VOICE: 512-777-1861
>>> >> www.KingsIsle.com
>>> >>
>>> >>
>>> >> -----Original Message-----
>>> >> From: sebb [mailto:sebbaz@gmail.com]
>>> >> Sent: Wednesday, December 14, 2011 5:27 PM
>>> >> To: JMeter Users List
>>> >> Subject: Re: Strange 'pause' activity during testing
>>> >>
>>> >> On 14 December 2011 22:20, Deepak Shetty <sh...@gmail.com> wrote:
>>> >>> I think the original post mentioned that GC was not running.
>>> >>>
>>> >>> can you take a thread dump when there is a pause and see what the
>>> >>> threads are waiting on?
>>> >>>
>>> >>
>>> >> I would also try running JMeter in non-GUI mode; GUI mode is very
>>> expensive
>>> >> in resources.
>>> >>
>>> >> Ensure all non-essential Listeners are disabled, see:
>>> >>
>>> >> http://jmeter.apache.org/usermanual/best-practices.html#lean_mean
>>> >>
>>> >>>
>>> >>> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com>
>>> wrote:
>>> >>>
>>> >>>> might be GC.. me be that JMeter's threads are all hung up in your
>>> >>>> server doing stuff.
>>> >>>>
>>> >>>> Regards,
>>> >>>> Kirk
>>> >>>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>>> >>>>
>>> >>>>> I have a marginally complicated test case that performs a
>>> >> 'registration'
>>> >>>> on
>>> >>>>> my site. It gets the home page, POSTS a form, gets the response,
>>> >>>>> POSTS another form, gets that response, and then completes. The
>>> >>>>> test runs fine with 100 threads, and 30000 iterations - IF I don't
>>> >>>>> "Retrieve All
>>> >>>> Embedded
>>> >>>>> Resources from HTML Files". In this mode, I am really testing the
>>> >>>> throughput
>>> >>>>> of my 'tomcat' application, not the other elements of my system.
>>> >>>>> (I'm assuming that the other elements are being retrieved from our
>>> >>>>> Content
>>> >>>> Data
>>> >>>>> Network instead of our main system in this case.)
>>> >>>>>
>>> >>>>> If I enable the "Retrieve All Embedded Resources from HTML Files"
>>> >>>>> flag,
>>> >>>> and
>>> >>>>> tune the test down to 10 threads with 3000 total iterations, I
>>> >>>>> notice a
>>> >>>> very
>>> >>>>> strange behavior. The test runs along at a pretty good clip for the
>>> >>>>> first
>>> >>>>> ~600 iterations (about 1 min, 25 seconds into the run), and then it
>>> >>>>> just stops making requests for about 35 seconds. Then, it picks
>>> >>>>> back up again
>>> >>>> and
>>> >>>>> runs for another 1 m 25s, and then stops again for 35 seconds...
>>> (NOTE:
>>> >>>> with
>>> >>>>> the 10 threads, 3000 total iterations - but with "Retrieve All
>>> >>>> Embedded..."
>>> >>>>> disabled - I don't see the 'stop' behavior either - so it isn't
>>> >>>>> caused by tuning it down...)
>>> >>>>>
>>> >>>>> I recently added the "Perfmon Metrics Collector" to the test
>>> >>>>> script, so I could see if one of the servers was maxed out - but it
>>> >>>>> looks like all the servers are idle during the 'stop' period.
>>> >>>>> Likewise, I added the Perfmon
>>> >>>> for
>>> >>>>> the "localhost" (running the JMeter test) to see if it was swamped
>>> >>>>> - but
>>> >>>> it
>>> >>>>> too is idle during the 'stop'. I swapped out our network switch
>>> >>>>> (the test environment is on an isolated network switch) with a
>>> >>>>> _much_ higher
>>> >>>> capacity
>>> >>>>> switch - in case there was a network issue, still no change.
>>> >>>>>
>>> >>>>> I'm running out of ideas for things to check - so I thought I'd ask
>>> >>>>> you
>>> >>>> guys
>>> >>>>> if you have any suggestions of things I should look at.
>>> >>>>>
>>> >>>>> My system consists of:
>>> >>>>>
>>> >>>>>       WinXP - running JMeter 2.4.1 - driving the test script in GUI
>>> >>>>> mode
>>> >>>>>       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as
>>> >>>>> the web server - using AJP Proxy to Server 2
>>> >>>>>       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the
>>> >>>>> App Server - connecting through Hibernate to Server 3
>>> >>>>>       Server 3 - running Red Hat Linux with MySQL 5.x as the DB
>>> >>>>> Server
>>> >>>>>
>>> >>>>> All 4 machines are running on a private switched network (32Gbs
>>> >>>> backplane).
>>> >>>>>
>>> >>>>> The requests are downloading about 3MB total (per thread per
>>> >>>>> iteration)
>>> >>>> over
>>> >>>>> 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
>>> >>>>>
>>> >>>>> At first I thought it was the network - but the new switch seemed
>>> >>>>> to deny that thought (the old switch had a much slower backplane).
>>> >>>>> Also, I'm
>>> >>>> having
>>> >>>>> no trouble collecting the PerfMon data during the 'stop' period -
>>> >>>>> so the network is still functioning just fine...
>>> >>>>> Then I thought it might be garbage collection on the tomcat - but I
>>> >>>> watched
>>> >>>>> the gc.log - and it doesn't do any GCs during the 'stop' period.
>>> >>>>> Then I thought it might be the garbage collection on the JMeter
>>> >>>>> side, so
>>> >>>> I
>>> >>>>> started the JMeter.bat from a 'cmd' prompt with gc logging enabled
>>> >>>>> - it doesn't do any GCs during the 'stop' period either.
>>> >>>>>
>>> >>>>> The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no
>>> >>>>> network I/O, no disk I/O, etc.) during the 'stop' period.
>>> >>>>>
>>> >>>>> The JMeter box (WinXP) is doing very little during that time too (I
>>> >>>>> attribute the little bit of activity to displaying the PerfMon
>>> >>>>> graphs,
>>> >>>> and
>>> >>>>> Remote Desktop display to my desktop computer)...
>>> >>>>>
>>> >>>>> --
>>> >>>>> Robin D. Wilson
>>> >>>>> Sr. Director of Web Development
>>> >>>>> KingsIsle Entertainment, Inc.
>>> >>>>> VOICE: 512-777-1861
>>> >>>>> www.KingsIsle.com
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> -------------------------------------------------------------------
>>> >>>>> -- To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> >>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>> ---------------------------------------------------------------------
>>> >>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> >>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>> >>>>
>>> >>>>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> >> For additional commands, e-mail: user-help@jmeter.apache.org
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> >> For additional commands, e-mail: user-help@jmeter.apache.org
>>> >>
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> > For additional commands, e-mail: user-help@jmeter.apache.org
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>
>>>
>>
>


-- 
Namaskara~Nalama~Guten Tag~Bonjour


   --
Keigu

Deepak
+91-9765089593
deicool@gmail.com
http://www.simtree.net

Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool

"Contribute to the world, environment and more : http://www.gridrepublic.org
"

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by Adrian Speteanu <as...@gmail.com>.

I agree that jmeter rarely performs poorly in non-GUI mode, unless you
allocate to much memory to it.

On Thu, Dec 15, 2011 at 1:12 PM, Adrian Speteanu <as...@gmail.com>wrote:

> In my experience, "stange pauses" usually points to the application.
>
> Of course, you have to make sure its not you (overlook in the test script,
> jmeter's configuration), or the test client environment (OS, network). BUT
> after all that is exhausted is usually the application or one of the layers
> of the system under test (like the database, usually more trivial stuff
> like a small connection pool to the database, or GC of the application
> under test).
>
> If you suspect its GC, divide the load you generate with the script in 3,
> and instead of testing with one jmeter instance, use 3 (if you can test
> like this on 3 different machines - the better). Also allocate less memory
> to jmeter now. What's the probability that all instances get paused by GC
> at the same time? Also, if you allocate less memory, stop-the-world GC
> takes less time. If you still have response times pauses in the same
> intervals - it has to be the system under test. Note:
>
> On Thu, Dec 15, 2011 at 3:51 AM, Robin D. Wilson <rw...@gmail.com>wrote:
>
>> Yes it is. I can turn it off if you think that matters.
>>
>> --
>> Robin D. Wilson
>>
>>
>>
>> On Dec 14, 2011, at 6:42 PM, basim@baassiri.ca wrote:
>>
>> > I'm curious to know if the system under test is https
>> >
>> >
>> > -----Original Message-----
>> > From: sebb <se...@gmail.com>
>> > Date: Thu, 15 Dec 2011 00:35:39
>> > To: JMeter Users List<us...@jmeter.apache.org>
>> > Reply-To: "JMeter Users List" <us...@jmeter.apache.org>
>> > Subject: Re: Strange 'pause' activity during testing
>> >
>> > On 15 December 2011 00:22, Robin D. Wilson <rw...@gmail.com> wrote:
>> >> I can use non-GUI mode for testing out the problem, but with the exact
>> same
>> >> test - just switching the 'Retrieve All Embedded...' disabled, the
>> problem
>> >> goes away.
>> >>
>> >> The only listeners I have enabled are the summary listener - the
>> PerfMon
>> >> listeners (1 for each server) - and a 'tree' listener, but it only
>> reports
>> >> 'errors'.
>> >>
>> >> I can run the test with 'Retrieve All Embedded...' enabled, I see the
>> >> problem every time.
>> >
>> > Maybe then there is a problem with the scanning of the HTML to extract
>> > the embedded resources, or maybe one of the embedded resources is a
>> > tar-pit.
>> >
>> > Do all the expected embedded resources get downloaded?
>> >
>> > Are there any unusually long elapsed times for embedded resources?
>> >
>> > Or large gaps between the parent sample download completion and the
>> > start of the first embedded resource?
>> >
>> > That would suggest the page took a while to parse.
>> >
>> > You'll need to select the optiion to "save subresults" in order to see
>> > the embedded samples.
>> >
>> >> If I just uncheck the box in the Request Defaults and re-run the test,
>> it
>> >> never happens... So are you suggesting that I've hit some arbitrary
>> limit of
>> >> the GUI mode?
>> >
>> > No, just that GUI mode is inherently more resource intensive.
>> >
>> > BTW, you wrote you were running JMeter 2.4.1 - that does not exist, so
>> > perhaps you meant the current version, 2.5.1?
>> >
>> >
>> >> What I'm wondering is why:
>> >>
>> >> 1) It is so consistent - the 'stop' happens at nearly the exact same
>> point
>> >> in every test run, and for nearly the exact same duration - and it
>> happens
>> >> multiple times during the test run, in the same pattern.
>> >>
>> >> 2) I can increase the load 10X (even more if I want), but without the
>> >> 'Retrieve All Embedded...' enabled, and it doesn’t happen.
>> >>
>> >> If it were resource related on the JMeter box, I would guess that
>> changing
>> >> the JVM memory config would show significant changes in behavior (like
>> it
>> >> would take longer to encounter the first 'stop'). If it were resource
>> >> related on the servers, I would expect to see some sort of contention
>> on the
>> >> servers.
>> >>
>> >> Tomorrow I will try the thread dump stuff - to see if there is anything
>> >> obvious in that output.
>> >>
>> >> From an "external" view - it looks like JMeter has some sort of 'pause'
>> >> built into it. Since all machines involved are nearly completely
>> 'idle' when
>> >> the stop happens - it seems unlikely that I've run out of resources.
>> The
>> >> other thought is that there is a deadlock somewhere - and something
>> has to
>> >> timeout before it can free up the deadlock and start again. But if
>> that were
>> >> the case, I would expect it to happen at slightly different intervals
>> in
>> >> each test run - and not repeat in the same consistent pattern during
>> the
>> >> test run.
>> >>
>> >> Some thoughts on what I think I've ruled out:
>> >>
>> >> 1) If it were the network 'backing off' I would expect the PerfMon
>> listener
>> >> to stop updating during the 'stop' - but it trundles happily along.
>> >> 2) If it were JVM garbage collections - I would have expected to see
>> them
>> >> running (or at least a 'stop-the-world' gc right before/after the
>> 'stop'
>> >> period) during the 'stop' period - but there are no gc events at all
>> during
>> >> the 'stop' period.
>> >> 3) If it were WinXP resources, I would have expected the PerfMon
>> listener to
>> >> show some activity during the 'stop' period on the WinXP (JMeter
>> client) box
>> >> - nothing like that appears.
>> >>
>> >> I'll look at the thread dumps tomorrow and get back to you guys.
>> Thanks for
>> >> helping me ponder this.
>> >>
>> >> --
>> >> Robin D. Wilson
>> >> Sr. Director of Web Development
>> >> KingsIsle Entertainment, Inc.
>> >> VOICE: 512-777-1861
>> >> www.KingsIsle.com
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: sebb [mailto:sebbaz@gmail.com]
>> >> Sent: Wednesday, December 14, 2011 5:27 PM
>> >> To: JMeter Users List
>> >> Subject: Re: Strange 'pause' activity during testing
>> >>
>> >> On 14 December 2011 22:20, Deepak Shetty <sh...@gmail.com> wrote:
>> >>> I think the original post mentioned that GC was not running.
>> >>>
>> >>> can you take a thread dump when there is a pause and see what the
>> >>> threads are waiting on?
>> >>>
>> >>
>> >> I would also try running JMeter in non-GUI mode; GUI mode is very
>> expensive
>> >> in resources.
>> >>
>> >> Ensure all non-essential Listeners are disabled, see:
>> >>
>> >> http://jmeter.apache.org/usermanual/best-practices.html#lean_mean
>> >>
>> >>>
>> >>> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com>
>> wrote:
>> >>>
>> >>>> might be GC.. me be that JMeter's threads are all hung up in your
>> >>>> server doing stuff.
>> >>>>
>> >>>> Regards,
>> >>>> Kirk
>> >>>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>> >>>>
>> >>>>> I have a marginally complicated test case that performs a
>> >> 'registration'
>> >>>> on
>> >>>>> my site. It gets the home page, POSTS a form, gets the response,
>> >>>>> POSTS another form, gets that response, and then completes. The
>> >>>>> test runs fine with 100 threads, and 30000 iterations - IF I don't
>> >>>>> "Retrieve All
>> >>>> Embedded
>> >>>>> Resources from HTML Files". In this mode, I am really testing the
>> >>>> throughput
>> >>>>> of my 'tomcat' application, not the other elements of my system.
>> >>>>> (I'm assuming that the other elements are being retrieved from our
>> >>>>> Content
>> >>>> Data
>> >>>>> Network instead of our main system in this case.)
>> >>>>>
>> >>>>> If I enable the "Retrieve All Embedded Resources from HTML Files"
>> >>>>> flag,
>> >>>> and
>> >>>>> tune the test down to 10 threads with 3000 total iterations, I
>> >>>>> notice a
>> >>>> very
>> >>>>> strange behavior. The test runs along at a pretty good clip for the
>> >>>>> first
>> >>>>> ~600 iterations (about 1 min, 25 seconds into the run), and then it
>> >>>>> just stops making requests for about 35 seconds. Then, it picks
>> >>>>> back up again
>> >>>> and
>> >>>>> runs for another 1 m 25s, and then stops again for 35 seconds...
>> (NOTE:
>> >>>> with
>> >>>>> the 10 threads, 3000 total iterations - but with "Retrieve All
>> >>>> Embedded..."
>> >>>>> disabled - I don't see the 'stop' behavior either - so it isn't
>> >>>>> caused by tuning it down...)
>> >>>>>
>> >>>>> I recently added the "Perfmon Metrics Collector" to the test
>> >>>>> script, so I could see if one of the servers was maxed out - but it
>> >>>>> looks like all the servers are idle during the 'stop' period.
>> >>>>> Likewise, I added the Perfmon
>> >>>> for
>> >>>>> the "localhost" (running the JMeter test) to see if it was swamped
>> >>>>> - but
>> >>>> it
>> >>>>> too is idle during the 'stop'. I swapped out our network switch
>> >>>>> (the test environment is on an isolated network switch) with a
>> >>>>> _much_ higher
>> >>>> capacity
>> >>>>> switch - in case there was a network issue, still no change.
>> >>>>>
>> >>>>> I'm running out of ideas for things to check - so I thought I'd ask
>> >>>>> you
>> >>>> guys
>> >>>>> if you have any suggestions of things I should look at.
>> >>>>>
>> >>>>> My system consists of:
>> >>>>>
>> >>>>>       WinXP - running JMeter 2.4.1 - driving the test script in GUI
>> >>>>> mode
>> >>>>>       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as
>> >>>>> the web server - using AJP Proxy to Server 2
>> >>>>>       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the
>> >>>>> App Server - connecting through Hibernate to Server 3
>> >>>>>       Server 3 - running Red Hat Linux with MySQL 5.x as the DB
>> >>>>> Server
>> >>>>>
>> >>>>> All 4 machines are running on a private switched network (32Gbs
>> >>>> backplane).
>> >>>>>
>> >>>>> The requests are downloading about 3MB total (per thread per
>> >>>>> iteration)
>> >>>> over
>> >>>>> 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
>> >>>>>
>> >>>>> At first I thought it was the network - but the new switch seemed
>> >>>>> to deny that thought (the old switch had a much slower backplane).
>> >>>>> Also, I'm
>> >>>> having
>> >>>>> no trouble collecting the PerfMon data during the 'stop' period -
>> >>>>> so the network is still functioning just fine...
>> >>>>> Then I thought it might be garbage collection on the tomcat - but I
>> >>>> watched
>> >>>>> the gc.log - and it doesn't do any GCs during the 'stop' period.
>> >>>>> Then I thought it might be the garbage collection on the JMeter
>> >>>>> side, so
>> >>>> I
>> >>>>> started the JMeter.bat from a 'cmd' prompt with gc logging enabled
>> >>>>> - it doesn't do any GCs during the 'stop' period either.
>> >>>>>
>> >>>>> The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no
>> >>>>> network I/O, no disk I/O, etc.) during the 'stop' period.
>> >>>>>
>> >>>>> The JMeter box (WinXP) is doing very little during that time too (I
>> >>>>> attribute the little bit of activity to displaying the PerfMon
>> >>>>> graphs,
>> >>>> and
>> >>>>> Remote Desktop display to my desktop computer)...
>> >>>>>
>> >>>>> --
>> >>>>> Robin D. Wilson
>> >>>>> Sr. Director of Web Development
>> >>>>> KingsIsle Entertainment, Inc.
>> >>>>> VOICE: 512-777-1861
>> >>>>> www.KingsIsle.com
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> -------------------------------------------------------------------
>> >>>>> -- To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> >>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>> >>>>>
>> >>>>
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> >>>> For additional commands, e-mail: user-help@jmeter.apache.org
>> >>>>
>> >>>>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> >> For additional commands, e-mail: user-help@jmeter.apache.org
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> >> For additional commands, e-mail: user-help@jmeter.apache.org
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> > For additional commands, e-mail: user-help@jmeter.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>
>

Re: Strange 'pause' activity during testing

Posted by Adrian Speteanu <as...@gmail.com>.

In my experience, "stange pauses" usually points to the application.

Of course, you have to make sure its not you (overlook in the test script,
jmeter's configuration), or the test client environment (OS, network). BUT
after all that is exhausted is usually the application or one of the layers
of the system under test (like the database, usually more trivial stuff
like a small connection pool to the database, or GC of the application
under test).

If you suspect its GC, divide the load you generate with the script in 3,
and instead of testing with one jmeter instance, use 3 (if you can test
like this on 3 different machines - the better). Also allocate less memory
to jmeter now. What's the probability that all instances get paused by GC
at the same time? Also, if you allocate less memory, stop-the-world GC
takes less time. If you still have response times pauses in the same
intervals - it has to be the system under test. Note:

On Thu, Dec 15, 2011 at 3:51 AM, Robin D. Wilson <rw...@gmail.com> wrote:

> Yes it is. I can turn it off if you think that matters.
>
> --
> Robin D. Wilson
>
>
>
> On Dec 14, 2011, at 6:42 PM, basim@baassiri.ca wrote:
>
> > I'm curious to know if the system under test is https
> >
> >
> > -----Original Message-----
> > From: sebb <se...@gmail.com>
> > Date: Thu, 15 Dec 2011 00:35:39
> > To: JMeter Users List<us...@jmeter.apache.org>
> > Reply-To: "JMeter Users List" <us...@jmeter.apache.org>
> > Subject: Re: Strange 'pause' activity during testing
> >
> > On 15 December 2011 00:22, Robin D. Wilson <rw...@gmail.com> wrote:
> >> I can use non-GUI mode for testing out the problem, but with the exact
> same
> >> test - just switching the 'Retrieve All Embedded...' disabled, the
> problem
> >> goes away.
> >>
> >> The only listeners I have enabled are the summary listener - the PerfMon
> >> listeners (1 for each server) - and a 'tree' listener, but it only
> reports
> >> 'errors'.
> >>
> >> I can run the test with 'Retrieve All Embedded...' enabled, I see the
> >> problem every time.
> >
> > Maybe then there is a problem with the scanning of the HTML to extract
> > the embedded resources, or maybe one of the embedded resources is a
> > tar-pit.
> >
> > Do all the expected embedded resources get downloaded?
> >
> > Are there any unusually long elapsed times for embedded resources?
> >
> > Or large gaps between the parent sample download completion and the
> > start of the first embedded resource?
> >
> > That would suggest the page took a while to parse.
> >
> > You'll need to select the optiion to "save subresults" in order to see
> > the embedded samples.
> >
> >> If I just uncheck the box in the Request Defaults and re-run the test,
> it
> >> never happens... So are you suggesting that I've hit some arbitrary
> limit of
> >> the GUI mode?
> >
> > No, just that GUI mode is inherently more resource intensive.
> >
> > BTW, you wrote you were running JMeter 2.4.1 - that does not exist, so
> > perhaps you meant the current version, 2.5.1?
> >
> >
> >> What I'm wondering is why:
> >>
> >> 1) It is so consistent - the 'stop' happens at nearly the exact same
> point
> >> in every test run, and for nearly the exact same duration - and it
> happens
> >> multiple times during the test run, in the same pattern.
> >>
> >> 2) I can increase the load 10X (even more if I want), but without the
> >> 'Retrieve All Embedded...' enabled, and it doesn’t happen.
> >>
> >> If it were resource related on the JMeter box, I would guess that
> changing
> >> the JVM memory config would show significant changes in behavior (like
> it
> >> would take longer to encounter the first 'stop'). If it were resource
> >> related on the servers, I would expect to see some sort of contention
> on the
> >> servers.
> >>
> >> Tomorrow I will try the thread dump stuff - to see if there is anything
> >> obvious in that output.
> >>
> >> From an "external" view - it looks like JMeter has some sort of 'pause'
> >> built into it. Since all machines involved are nearly completely 'idle'
> when
> >> the stop happens - it seems unlikely that I've run out of resources. The
> >> other thought is that there is a deadlock somewhere - and something has
> to
> >> timeout before it can free up the deadlock and start again. But if that
> were
> >> the case, I would expect it to happen at slightly different intervals in
> >> each test run - and not repeat in the same consistent pattern during the
> >> test run.
> >>
> >> Some thoughts on what I think I've ruled out:
> >>
> >> 1) If it were the network 'backing off' I would expect the PerfMon
> listener
> >> to stop updating during the 'stop' - but it trundles happily along.
> >> 2) If it were JVM garbage collections - I would have expected to see
> them
> >> running (or at least a 'stop-the-world' gc right before/after the 'stop'
> >> period) during the 'stop' period - but there are no gc events at all
> during
> >> the 'stop' period.
> >> 3) If it were WinXP resources, I would have expected the PerfMon
> listener to
> >> show some activity during the 'stop' period on the WinXP (JMeter
> client) box
> >> - nothing like that appears.
> >>
> >> I'll look at the thread dumps tomorrow and get back to you guys. Thanks
> for
> >> helping me ponder this.
> >>
> >> --
> >> Robin D. Wilson
> >> Sr. Director of Web Development
> >> KingsIsle Entertainment, Inc.
> >> VOICE: 512-777-1861
> >> www.KingsIsle.com
> >>
> >>
> >> -----Original Message-----
> >> From: sebb [mailto:sebbaz@gmail.com]
> >> Sent: Wednesday, December 14, 2011 5:27 PM
> >> To: JMeter Users List
> >> Subject: Re: Strange 'pause' activity during testing
> >>
> >> On 14 December 2011 22:20, Deepak Shetty <sh...@gmail.com> wrote:
> >>> I think the original post mentioned that GC was not running.
> >>>
> >>> can you take a thread dump when there is a pause and see what the
> >>> threads are waiting on?
> >>>
> >>
> >> I would also try running JMeter in non-GUI mode; GUI mode is very
> expensive
> >> in resources.
> >>
> >> Ensure all non-essential Listeners are disabled, see:
> >>
> >> http://jmeter.apache.org/usermanual/best-practices.html#lean_mean
> >>
> >>>
> >>> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com>
> wrote:
> >>>
> >>>> might be GC.. me be that JMeter's threads are all hung up in your
> >>>> server doing stuff.
> >>>>
> >>>> Regards,
> >>>> Kirk
> >>>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
> >>>>
> >>>>> I have a marginally complicated test case that performs a
> >> 'registration'
> >>>> on
> >>>>> my site. It gets the home page, POSTS a form, gets the response,
> >>>>> POSTS another form, gets that response, and then completes. The
> >>>>> test runs fine with 100 threads, and 30000 iterations - IF I don't
> >>>>> "Retrieve All
> >>>> Embedded
> >>>>> Resources from HTML Files". In this mode, I am really testing the
> >>>> throughput
> >>>>> of my 'tomcat' application, not the other elements of my system.
> >>>>> (I'm assuming that the other elements are being retrieved from our
> >>>>> Content
> >>>> Data
> >>>>> Network instead of our main system in this case.)
> >>>>>
> >>>>> If I enable the "Retrieve All Embedded Resources from HTML Files"
> >>>>> flag,
> >>>> and
> >>>>> tune the test down to 10 threads with 3000 total iterations, I
> >>>>> notice a
> >>>> very
> >>>>> strange behavior. The test runs along at a pretty good clip for the
> >>>>> first
> >>>>> ~600 iterations (about 1 min, 25 seconds into the run), and then it
> >>>>> just stops making requests for about 35 seconds. Then, it picks
> >>>>> back up again
> >>>> and
> >>>>> runs for another 1 m 25s, and then stops again for 35 seconds...
> (NOTE:
> >>>> with
> >>>>> the 10 threads, 3000 total iterations - but with "Retrieve All
> >>>> Embedded..."
> >>>>> disabled - I don't see the 'stop' behavior either - so it isn't
> >>>>> caused by tuning it down...)
> >>>>>
> >>>>> I recently added the "Perfmon Metrics Collector" to the test
> >>>>> script, so I could see if one of the servers was maxed out - but it
> >>>>> looks like all the servers are idle during the 'stop' period.
> >>>>> Likewise, I added the Perfmon
> >>>> for
> >>>>> the "localhost" (running the JMeter test) to see if it was swamped
> >>>>> - but
> >>>> it
> >>>>> too is idle during the 'stop'. I swapped out our network switch
> >>>>> (the test environment is on an isolated network switch) with a
> >>>>> _much_ higher
> >>>> capacity
> >>>>> switch - in case there was a network issue, still no change.
> >>>>>
> >>>>> I'm running out of ideas for things to check - so I thought I'd ask
> >>>>> you
> >>>> guys
> >>>>> if you have any suggestions of things I should look at.
> >>>>>
> >>>>> My system consists of:
> >>>>>
> >>>>>       WinXP - running JMeter 2.4.1 - driving the test script in GUI
> >>>>> mode
> >>>>>       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as
> >>>>> the web server - using AJP Proxy to Server 2
> >>>>>       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the
> >>>>> App Server - connecting through Hibernate to Server 3
> >>>>>       Server 3 - running Red Hat Linux with MySQL 5.x as the DB
> >>>>> Server
> >>>>>
> >>>>> All 4 machines are running on a private switched network (32Gbs
> >>>> backplane).
> >>>>>
> >>>>> The requests are downloading about 3MB total (per thread per
> >>>>> iteration)
> >>>> over
> >>>>> 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
> >>>>>
> >>>>> At first I thought it was the network - but the new switch seemed
> >>>>> to deny that thought (the old switch had a much slower backplane).
> >>>>> Also, I'm
> >>>> having
> >>>>> no trouble collecting the PerfMon data during the 'stop' period -
> >>>>> so the network is still functioning just fine...
> >>>>> Then I thought it might be garbage collection on the tomcat - but I
> >>>> watched
> >>>>> the gc.log - and it doesn't do any GCs during the 'stop' period.
> >>>>> Then I thought it might be the garbage collection on the JMeter
> >>>>> side, so
> >>>> I
> >>>>> started the JMeter.bat from a 'cmd' prompt with gc logging enabled
> >>>>> - it doesn't do any GCs during the 'stop' period either.
> >>>>>
> >>>>> The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no
> >>>>> network I/O, no disk I/O, etc.) during the 'stop' period.
> >>>>>
> >>>>> The JMeter box (WinXP) is doing very little during that time too (I
> >>>>> attribute the little bit of activity to displaying the PerfMon
> >>>>> graphs,
> >>>> and
> >>>>> Remote Desktop display to my desktop computer)...
> >>>>>
> >>>>> --
> >>>>> Robin D. Wilson
> >>>>> Sr. Director of Web Development
> >>>>> KingsIsle Entertainment, Inc.
> >>>>> VOICE: 512-777-1861
> >>>>> www.KingsIsle.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> -------------------------------------------------------------------
> >>>>> -- To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >>>>> For additional commands, e-mail: user-help@jmeter.apache.org
> >>>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >>>> For additional commands, e-mail: user-help@jmeter.apache.org
> >>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >> For additional commands, e-mail: user-help@jmeter.apache.org
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >> For additional commands, e-mail: user-help@jmeter.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > For additional commands, e-mail: user-help@jmeter.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>

Re: Strange 'pause' activity during testing

Posted by "Robin D. Wilson" <rw...@gmail.com>.

Yes it is. I can turn it off if you think that matters. 

--
Robin D. Wilson



On Dec 14, 2011, at 6:42 PM, basim@baassiri.ca wrote:

> I'm curious to know if the system under test is https 
> 
> 
> -----Original Message-----
> From: sebb <se...@gmail.com>
> Date: Thu, 15 Dec 2011 00:35:39 
> To: JMeter Users List<us...@jmeter.apache.org>
> Reply-To: "JMeter Users List" <us...@jmeter.apache.org>
> Subject: Re: Strange 'pause' activity during testing
> 
> On 15 December 2011 00:22, Robin D. Wilson <rw...@gmail.com> wrote:
>> I can use non-GUI mode for testing out the problem, but with the exact same
>> test - just switching the 'Retrieve All Embedded...' disabled, the problem
>> goes away.
>> 
>> The only listeners I have enabled are the summary listener - the PerfMon
>> listeners (1 for each server) - and a 'tree' listener, but it only reports
>> 'errors'.
>> 
>> I can run the test with 'Retrieve All Embedded...' enabled, I see the
>> problem every time.
> 
> Maybe then there is a problem with the scanning of the HTML to extract
> the embedded resources, or maybe one of the embedded resources is a
> tar-pit.
> 
> Do all the expected embedded resources get downloaded?
> 
> Are there any unusually long elapsed times for embedded resources?
> 
> Or large gaps between the parent sample download completion and the
> start of the first embedded resource?
> 
> That would suggest the page took a while to parse.
> 
> You'll need to select the optiion to "save subresults" in order to see
> the embedded samples.
> 
>> If I just uncheck the box in the Request Defaults and re-run the test, it
>> never happens... So are you suggesting that I've hit some arbitrary limit of
>> the GUI mode?
> 
> No, just that GUI mode is inherently more resource intensive.
> 
> BTW, you wrote you were running JMeter 2.4.1 - that does not exist, so
> perhaps you meant the current version, 2.5.1?
> 
> 
>> What I'm wondering is why:
>> 
>> 1) It is so consistent - the 'stop' happens at nearly the exact same point
>> in every test run, and for nearly the exact same duration - and it happens
>> multiple times during the test run, in the same pattern.
>> 
>> 2) I can increase the load 10X (even more if I want), but without the
>> 'Retrieve All Embedded...' enabled, and it doesn’t happen.
>> 
>> If it were resource related on the JMeter box, I would guess that changing
>> the JVM memory config would show significant changes in behavior (like it
>> would take longer to encounter the first 'stop'). If it were resource
>> related on the servers, I would expect to see some sort of contention on the
>> servers.
>> 
>> Tomorrow I will try the thread dump stuff - to see if there is anything
>> obvious in that output.
>> 
>> From an "external" view - it looks like JMeter has some sort of 'pause'
>> built into it. Since all machines involved are nearly completely 'idle' when
>> the stop happens - it seems unlikely that I've run out of resources. The
>> other thought is that there is a deadlock somewhere - and something has to
>> timeout before it can free up the deadlock and start again. But if that were
>> the case, I would expect it to happen at slightly different intervals in
>> each test run - and not repeat in the same consistent pattern during the
>> test run.
>> 
>> Some thoughts on what I think I've ruled out:
>> 
>> 1) If it were the network 'backing off' I would expect the PerfMon listener
>> to stop updating during the 'stop' - but it trundles happily along.
>> 2) If it were JVM garbage collections - I would have expected to see them
>> running (or at least a 'stop-the-world' gc right before/after the 'stop'
>> period) during the 'stop' period - but there are no gc events at all during
>> the 'stop' period.
>> 3) If it were WinXP resources, I would have expected the PerfMon listener to
>> show some activity during the 'stop' period on the WinXP (JMeter client) box
>> - nothing like that appears.
>> 
>> I'll look at the thread dumps tomorrow and get back to you guys. Thanks for
>> helping me ponder this.
>> 
>> --
>> Robin D. Wilson
>> Sr. Director of Web Development
>> KingsIsle Entertainment, Inc.
>> VOICE: 512-777-1861
>> www.KingsIsle.com
>> 
>> 
>> -----Original Message-----
>> From: sebb [mailto:sebbaz@gmail.com]
>> Sent: Wednesday, December 14, 2011 5:27 PM
>> To: JMeter Users List
>> Subject: Re: Strange 'pause' activity during testing
>> 
>> On 14 December 2011 22:20, Deepak Shetty <sh...@gmail.com> wrote:
>>> I think the original post mentioned that GC was not running.
>>> 
>>> can you take a thread dump when there is a pause and see what the
>>> threads are waiting on?
>>> 
>> 
>> I would also try running JMeter in non-GUI mode; GUI mode is very expensive
>> in resources.
>> 
>> Ensure all non-essential Listeners are disabled, see:
>> 
>> http://jmeter.apache.org/usermanual/best-practices.html#lean_mean
>> 
>>> 
>>> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com> wrote:
>>> 
>>>> might be GC.. me be that JMeter's threads are all hung up in your
>>>> server doing stuff.
>>>> 
>>>> Regards,
>>>> Kirk
>>>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>>>> 
>>>>> I have a marginally complicated test case that performs a
>> 'registration'
>>>> on
>>>>> my site. It gets the home page, POSTS a form, gets the response,
>>>>> POSTS another form, gets that response, and then completes. The
>>>>> test runs fine with 100 threads, and 30000 iterations - IF I don't
>>>>> "Retrieve All
>>>> Embedded
>>>>> Resources from HTML Files". In this mode, I am really testing the
>>>> throughput
>>>>> of my 'tomcat' application, not the other elements of my system.
>>>>> (I'm assuming that the other elements are being retrieved from our
>>>>> Content
>>>> Data
>>>>> Network instead of our main system in this case.)
>>>>> 
>>>>> If I enable the "Retrieve All Embedded Resources from HTML Files"
>>>>> flag,
>>>> and
>>>>> tune the test down to 10 threads with 3000 total iterations, I
>>>>> notice a
>>>> very
>>>>> strange behavior. The test runs along at a pretty good clip for the
>>>>> first
>>>>> ~600 iterations (about 1 min, 25 seconds into the run), and then it
>>>>> just stops making requests for about 35 seconds. Then, it picks
>>>>> back up again
>>>> and
>>>>> runs for another 1 m 25s, and then stops again for 35 seconds... (NOTE:
>>>> with
>>>>> the 10 threads, 3000 total iterations - but with "Retrieve All
>>>> Embedded..."
>>>>> disabled - I don't see the 'stop' behavior either - so it isn't
>>>>> caused by tuning it down...)
>>>>> 
>>>>> I recently added the "Perfmon Metrics Collector" to the test
>>>>> script, so I could see if one of the servers was maxed out - but it
>>>>> looks like all the servers are idle during the 'stop' period.
>>>>> Likewise, I added the Perfmon
>>>> for
>>>>> the "localhost" (running the JMeter test) to see if it was swamped
>>>>> - but
>>>> it
>>>>> too is idle during the 'stop'. I swapped out our network switch
>>>>> (the test environment is on an isolated network switch) with a
>>>>> _much_ higher
>>>> capacity
>>>>> switch - in case there was a network issue, still no change.
>>>>> 
>>>>> I'm running out of ideas for things to check - so I thought I'd ask
>>>>> you
>>>> guys
>>>>> if you have any suggestions of things I should look at.
>>>>> 
>>>>> My system consists of:
>>>>> 
>>>>>       WinXP - running JMeter 2.4.1 - driving the test script in GUI
>>>>> mode
>>>>>       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as
>>>>> the web server - using AJP Proxy to Server 2
>>>>>       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the
>>>>> App Server - connecting through Hibernate to Server 3
>>>>>       Server 3 - running Red Hat Linux with MySQL 5.x as the DB
>>>>> Server
>>>>> 
>>>>> All 4 machines are running on a private switched network (32Gbs
>>>> backplane).
>>>>> 
>>>>> The requests are downloading about 3MB total (per thread per
>>>>> iteration)
>>>> over
>>>>> 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
>>>>> 
>>>>> At first I thought it was the network - but the new switch seemed
>>>>> to deny that thought (the old switch had a much slower backplane).
>>>>> Also, I'm
>>>> having
>>>>> no trouble collecting the PerfMon data during the 'stop' period -
>>>>> so the network is still functioning just fine...
>>>>> Then I thought it might be garbage collection on the tomcat - but I
>>>> watched
>>>>> the gc.log - and it doesn't do any GCs during the 'stop' period.
>>>>> Then I thought it might be the garbage collection on the JMeter
>>>>> side, so
>>>> I
>>>>> started the JMeter.bat from a 'cmd' prompt with gc logging enabled
>>>>> - it doesn't do any GCs during the 'stop' period either.
>>>>> 
>>>>> The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no
>>>>> network I/O, no disk I/O, etc.) during the 'stop' period.
>>>>> 
>>>>> The JMeter box (WinXP) is doing very little during that time too (I
>>>>> attribute the little bit of activity to displaying the PerfMon
>>>>> graphs,
>>>> and
>>>>> Remote Desktop display to my desktop computer)...
>>>>> 
>>>>> --
>>>>> Robin D. Wilson
>>>>> Sr. Director of Web Development
>>>>> KingsIsle Entertainment, Inc.
>>>>> VOICE: 512-777-1861
>>>>> www.KingsIsle.com
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -------------------------------------------------------------------
>>>>> -- To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>> 
>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>> 
>>>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by ba...@baassiri.ca.

I'm curious to know if the system under test is https 


-----Original Message-----
From: sebb <se...@gmail.com>
Date: Thu, 15 Dec 2011 00:35:39 
To: JMeter Users List<us...@jmeter.apache.org>
Reply-To: "JMeter Users List" <us...@jmeter.apache.org>
Subject: Re: Strange 'pause' activity during testing

On 15 December 2011 00:22, Robin D. Wilson <rw...@gmail.com> wrote:
> I can use non-GUI mode for testing out the problem, but with the exact same
> test - just switching the 'Retrieve All Embedded...' disabled, the problem
> goes away.
>
> The only listeners I have enabled are the summary listener - the PerfMon
> listeners (1 for each server) - and a 'tree' listener, but it only reports
> 'errors'.
>
> I can run the test with 'Retrieve All Embedded...' enabled, I see the
> problem every time.

Maybe then there is a problem with the scanning of the HTML to extract
the embedded resources, or maybe one of the embedded resources is a
tar-pit.

Do all the expected embedded resources get downloaded?

Are there any unusually long elapsed times for embedded resources?

Or large gaps between the parent sample download completion and the
start of the first embedded resource?

That would suggest the page took a while to parse.

You'll need to select the optiion to "save subresults" in order to see
the embedded samples.

> If I just uncheck the box in the Request Defaults and re-run the test, it
> never happens... So are you suggesting that I've hit some arbitrary limit of
> the GUI mode?

No, just that GUI mode is inherently more resource intensive.

BTW, you wrote you were running JMeter 2.4.1 - that does not exist, so
perhaps you meant the current version, 2.5.1?


> What I'm wondering is why:
>
> 1) It is so consistent - the 'stop' happens at nearly the exact same point
> in every test run, and for nearly the exact same duration - and it happens
> multiple times during the test run, in the same pattern.
>
> 2) I can increase the load 10X (even more if I want), but without the
> 'Retrieve All Embedded...' enabled, and it doesn’t happen.
>
> If it were resource related on the JMeter box, I would guess that changing
> the JVM memory config would show significant changes in behavior (like it
> would take longer to encounter the first 'stop'). If it were resource
> related on the servers, I would expect to see some sort of contention on the
> servers.
>
> Tomorrow I will try the thread dump stuff - to see if there is anything
> obvious in that output.
>
> From an "external" view - it looks like JMeter has some sort of 'pause'
> built into it. Since all machines involved are nearly completely 'idle' when
> the stop happens - it seems unlikely that I've run out of resources. The
> other thought is that there is a deadlock somewhere - and something has to
> timeout before it can free up the deadlock and start again. But if that were
> the case, I would expect it to happen at slightly different intervals in
> each test run - and not repeat in the same consistent pattern during the
> test run.
>
> Some thoughts on what I think I've ruled out:
>
> 1) If it were the network 'backing off' I would expect the PerfMon listener
> to stop updating during the 'stop' - but it trundles happily along.
> 2) If it were JVM garbage collections - I would have expected to see them
> running (or at least a 'stop-the-world' gc right before/after the 'stop'
> period) during the 'stop' period - but there are no gc events at all during
> the 'stop' period.
> 3) If it were WinXP resources, I would have expected the PerfMon listener to
> show some activity during the 'stop' period on the WinXP (JMeter client) box
> - nothing like that appears.
>
> I'll look at the thread dumps tomorrow and get back to you guys. Thanks for
> helping me ponder this.
>
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> VOICE: 512-777-1861
> www.KingsIsle.com
>
>
> -----Original Message-----
> From: sebb [mailto:sebbaz@gmail.com]
> Sent: Wednesday, December 14, 2011 5:27 PM
> To: JMeter Users List
> Subject: Re: Strange 'pause' activity during testing
>
> On 14 December 2011 22:20, Deepak Shetty <sh...@gmail.com> wrote:
>> I think the original post mentioned that GC was not running.
>>
>> can you take a thread dump when there is a pause and see what the
>> threads are waiting on?
>>
>
> I would also try running JMeter in non-GUI mode; GUI mode is very expensive
> in resources.
>
> Ensure all non-essential Listeners are disabled, see:
>
> http://jmeter.apache.org/usermanual/best-practices.html#lean_mean
>
>>
>> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com> wrote:
>>
>>> might be GC.. me be that JMeter's threads are all hung up in your
>>> server doing stuff.
>>>
>>> Regards,
>>> Kirk
>>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>>>
>>> > I have a marginally complicated test case that performs a
> 'registration'
>>> on
>>> > my site. It gets the home page, POSTS a form, gets the response,
>>> > POSTS another form, gets that response, and then completes. The
>>> > test runs fine with 100 threads, and 30000 iterations - IF I don't
>>> > "Retrieve All
>>> Embedded
>>> > Resources from HTML Files". In this mode, I am really testing the
>>> throughput
>>> > of my 'tomcat' application, not the other elements of my system.
>>> > (I'm assuming that the other elements are being retrieved from our
>>> > Content
>>> Data
>>> > Network instead of our main system in this case.)
>>> >
>>> > If I enable the "Retrieve All Embedded Resources from HTML Files"
>>> > flag,
>>> and
>>> > tune the test down to 10 threads with 3000 total iterations, I
>>> > notice a
>>> very
>>> > strange behavior. The test runs along at a pretty good clip for the
>>> > first
>>> > ~600 iterations (about 1 min, 25 seconds into the run), and then it
>>> > just stops making requests for about 35 seconds. Then, it picks
>>> > back up again
>>> and
>>> > runs for another 1 m 25s, and then stops again for 35 seconds... (NOTE:
>>> with
>>> > the 10 threads, 3000 total iterations - but with "Retrieve All
>>> Embedded..."
>>> > disabled - I don't see the 'stop' behavior either - so it isn't
>>> > caused by tuning it down...)
>>> >
>>> > I recently added the "Perfmon Metrics Collector" to the test
>>> > script, so I could see if one of the servers was maxed out - but it
>>> > looks like all the servers are idle during the 'stop' period.
>>> > Likewise, I added the Perfmon
>>> for
>>> > the "localhost" (running the JMeter test) to see if it was swamped
>>> > - but
>>> it
>>> > too is idle during the 'stop'. I swapped out our network switch
>>> > (the test environment is on an isolated network switch) with a
>>> > _much_ higher
>>> capacity
>>> > switch - in case there was a network issue, still no change.
>>> >
>>> > I'm running out of ideas for things to check - so I thought I'd ask
>>> > you
>>> guys
>>> > if you have any suggestions of things I should look at.
>>> >
>>> > My system consists of:
>>> >
>>> >       WinXP - running JMeter 2.4.1 - driving the test script in GUI
>>> > mode
>>> >       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as
>>> > the web server - using AJP Proxy to Server 2
>>> >       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the
>>> > App Server - connecting through Hibernate to Server 3
>>> >       Server 3 - running Red Hat Linux with MySQL 5.x as the DB
>>> > Server
>>> >
>>> > All 4 machines are running on a private switched network (32Gbs
>>> backplane).
>>> >
>>> > The requests are downloading about 3MB total (per thread per
>>> > iteration)
>>> over
>>> > 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
>>> >
>>> > At first I thought it was the network - but the new switch seemed
>>> > to deny that thought (the old switch had a much slower backplane).
>>> > Also, I'm
>>> having
>>> > no trouble collecting the PerfMon data during the 'stop' period -
>>> > so the network is still functioning just fine...
>>> > Then I thought it might be garbage collection on the tomcat - but I
>>> watched
>>> > the gc.log - and it doesn't do any GCs during the 'stop' period.
>>> > Then I thought it might be the garbage collection on the JMeter
>>> > side, so
>>> I
>>> > started the JMeter.bat from a 'cmd' prompt with gc logging enabled
>>> > - it doesn't do any GCs during the 'stop' period either.
>>> >
>>> > The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no
>>> > network I/O, no disk I/O, etc.) during the 'stop' period.
>>> >
>>> > The JMeter box (WinXP) is doing very little during that time too (I
>>> > attribute the little bit of activity to displaying the PerfMon
>>> > graphs,
>>> and
>>> > Remote Desktop display to my desktop computer)...
>>> >
>>> > --
>>> > Robin D. Wilson
>>> > Sr. Director of Web Development
>>> > KingsIsle Entertainment, Inc.
>>> > VOICE: 512-777-1861
>>> > www.KingsIsle.com
>>> >
>>> >
>>> >
>>> >
>>> > -------------------------------------------------------------------
>>> > -- To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> > For additional commands, e-mail: user-help@jmeter.apache.org
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

RE: Strange 'pause' activity during testing

Posted by Oliver Lloyd <ol...@hotmail.com>.

I think it was mentioned earlier in this thread but a simple check you can do
here to verify if the problem is with JMeter or not is to half the test
throughput and run it over two machines. So you'll be making the same volume
of requests (as far as the application is concerned) but each testplan will
be doing less. 

If you still see a pause at the same time then the problem is probably
outside of JMeter. If you don't see the issue or it happens twice as far
into the run then it suggests the test itself is the cause.


-----
http://www.http503.com/
--
View this message in context: http://jmeter.512774.n5.nabble.com/Strange-pause-activity-during-testing-tp5075754p5080471.html
Sent from the JMeter - User mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

RE: Strange 'pause' activity during testing

Posted by "Robin D. Wilson" <rw...@gmail.com>.

OK, I think I've narrowed the problem down - it appears to be something in
Apache (our web server) that is causing the pause. (I made this change based
on the note that basim@baassiri.ca sent asking if I was using HTTPS, I had
to change the configuration to get off of HTTPS, and that allowed me to try
some other test configurations.)

When I re-run the test but skip the Apache (AJP proxy) and go straight to
the Tomcat, I get no slowdowns or pauses... It is _not_ a JMeter problem
(near as I can tell).

Also, I re-ran it so that I get the main page from the tomcat, and the
'embedded' resources from the Apache - and I get no slow downs. So if I had
to guess, it is some sort of AJP configuration issue. I'll work on that now.

Thank you all for your assistance!

--
Robin D. Wilson
Sr. Director of Web Development
KingsIsle Entertainment, Inc.
VOICE: 512-777-1861
www.KingsIsle.com


-----Original Message-----
From: Philippe Mouawad [mailto:philippe.mouawad@gmail.com] 
Sent: Thursday, December 15, 2011 2:23 PM
To: JMeter Users List
Subject: Re: Strange 'pause' activity during testing

Do you use CacheManager ?
You should remove any plugin and activate GC logs to check it's not GC ?
How is CPU on JMeter stack ?

Regards
Philippe


On Thu, Dec 15, 2011 at 9:09 PM, Robin D. Wilson <rw...@gmail.com> wrote:

> >Maybe then there is a problem with the scanning of the HTML to 
> >extract the
> embedded resources, or maybe one of the embedded resources is a tar-pit.
>
> If this were the case, I would expect the first sample to show the
problem.
> The fact that it does 600+ iterations without a problem - and _then_ 
> stalls seems like it rules out any problem with the returned HTML 
> (especially since the only difference in the returned result is the 
> username supplied by JMeter).
>
> >Do all the expected embedded resources get downloaded?
>
> Zero errors (even with the pauses there are no errors at all).
>
> >Are there any unusually long elapsed times for embedded resources?
>
> I see the 'max request duration' jump up right after the pause - but 
> it is only 10-11 seconds (not ~35 seconds like the pause).
>
> >Or large gaps between the parent sample download completion and the 
> >start
> of the first embedded resource?
>
> Not that I can see... I'll see if I can get more detail on this.
>
> >That would suggest the page took a while to parse.
>
> I would assume that because this happens ~600+ iterations into the 
> test (the first time), that if it was related to parsing the page, I 
> would see it earlier in the test run cycle. And I wouldn't expect a 
> parsing problem to repeat on such a consistent basis - without 
> happening on every sample.
> Right
> now, if it is a parsing problem, it only happens the ~600th time it 
> sees the same page, which seems really surprising to me (then it 
> happens again after another ~600 iterations, etc.). Also, I would 
> expect the parser to take up some CPU and perhaps even some I/O 
> cycles, but the PerfMon shows idle during the pause period.
>
> >You'll need to select the optiion to "save subresults" in order to 
> >see the
> embedded samples.
>
> ...
>
>
> >BTW, you wrote you were running JMeter 2.4.1 - that does not exist, 
> >so
> perhaps you meant the current version, 2.5.1?
>
> Sorry, I meant 2.4.  I didn't upgrade to 2.5 (and beyond) because of a 
> previously reported problem where 2.5+ slows down my throughput to 
> about 60% of what I get on 2.4.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>


--
Cordialement.
Philippe Mouawad.


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by Adrian Speteanu <as...@gmail.com>.

Of course, no problems on test client points to the fact that you should
start investigating the application :).

Use jdk's jvisualvm & visualgc to monitor the application under test.

On Fri, Dec 16, 2011 at 7:34 AM, Kirk <ki...@gmail.com> wrote:

> Summary
>
> JMeter Machine
> CPU is idle, network is idle, no GC during "lockup". No matter, GC will
> drive CPU utilization high so the fact that CPU is idle suggest resource
> starvation.. i.e. all of the JMeter threads will be found to be blocked on
> a socket call.. i.e. the problem is in your application and not JMeter.
>
> I think you need outside help as the problems you're facing go beyond a
> simple Q&A here. Sorry to spam the list but I do offer tuning services and
> a performance tuning course. We can talk off-line about either if your
> interested.
>
> Kind regards,
> Kirk Pepperdine
>
> On Dec 15, 2011, at 9:40 PM, Robin D. Wilson wrote:
>
> > I don't use CacheManager (since that would defeat the purpose of the
> test -
> > which is to exercise the entire system for all assets from the pages
> being
> > requested).
> >
> > I have logged the GCs (without removing the plugins), and it doesn't do
> any
> > GCs at all during the 'pause' period. The GC right before the pause takes
> > less than a second (.26...) and the one right after is in the same
> ball-park
> > of duration. But during the pause, no GCs at all occur.
> >
> > CPU goes to idle on the JMeter box during the pause (as does network and
> > disk I/O, and swap usage). There is a _little_ activity, but I've been
> > attributing that to the Remote Desktop I'm using to get to the box, and
> the
> > PerfMon graphs that are still collecting data. But compared to before and
> > after the pause the activity level is less than 2% during the 'pause'
> > period, compared to 75-80% during the 'run' period.
> >
> > I will try to get the thread dump - I'm working on some production issues
> > today, so I haven't had time to setup for a thread dump today.
> >
> > --
> > Robin D. Wilson
> > Sr. Director of Web Development
> > KingsIsle Entertainment, Inc.
> > VOICE: 512-777-1861
> > www.KingsIsle.com
> >
> >
> > -----Original Message-----
> > From: Philippe Mouawad [mailto:philippe.mouawad@gmail.com]
> > Sent: Thursday, December 15, 2011 2:23 PM
> > To: JMeter Users List
> > Subject: Re: Strange 'pause' activity during testing
> >
> > Do you use CacheManager ?
> > You should remove any plugin and activate GC logs to check it's not GC ?
> > How is CPU on JMeter stack ?
> >
> > Regards
> > Philippe
> >
> >
> > On Thu, Dec 15, 2011 at 9:09 PM, Robin D. Wilson <rw...@gmail.com>
> wrote:
> >
> >>> Maybe then there is a problem with the scanning of the HTML to
> >>> extract the
> >> embedded resources, or maybe one of the embedded resources is a tar-pit.
> >>
> >> If this were the case, I would expect the first sample to show the
> > problem.
> >> The fact that it does 600+ iterations without a problem - and _then_
> >> stalls seems like it rules out any problem with the returned HTML
> >> (especially since the only difference in the returned result is the
> >> username supplied by JMeter).
> >>
> >>> Do all the expected embedded resources get downloaded?
> >>
> >> Zero errors (even with the pauses there are no errors at all).
> >>
> >>> Are there any unusually long elapsed times for embedded resources?
> >>
> >> I see the 'max request duration' jump up right after the pause - but
> >> it is only 10-11 seconds (not ~35 seconds like the pause).
> >>
> >>> Or large gaps between the parent sample download completion and the
> >>> start
> >> of the first embedded resource?
> >>
> >> Not that I can see... I'll see if I can get more detail on this.
> >>
> >>> That would suggest the page took a while to parse.
> >>
> >> I would assume that because this happens ~600+ iterations into the
> >> test (the first time), that if it was related to parsing the page, I
> >> would see it earlier in the test run cycle. And I wouldn't expect a
> >> parsing problem to repeat on such a consistent basis - without
> >> happening on every sample.
> >> Right
> >> now, if it is a parsing problem, it only happens the ~600th time it
> >> sees the same page, which seems really surprising to me (then it
> >> happens again after another ~600 iterations, etc.). Also, I would
> >> expect the parser to take up some CPU and perhaps even some I/O
> >> cycles, but the PerfMon shows idle during the pause period.
> >>
> >>> You'll need to select the optiion to "save subresults" in order to
> >>> see the
> >> embedded samples.
> >>
> >> ...
> >>
> >>
> >>> BTW, you wrote you were running JMeter 2.4.1 - that does not exist,
> >>> so
> >> perhaps you meant the current version, 2.5.1?
> >>
> >> Sorry, I meant 2.4.  I didn't upgrade to 2.5 (and beyond) because of a
> >> previously reported problem where 2.5+ slows down my throughput to
> >> about 60% of what I get on 2.4.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >> For additional commands, e-mail: user-help@jmeter.apache.org
> >>
> >>
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > For additional commands, e-mail: user-help@jmeter.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>

Re: Strange 'pause' activity during testing

Posted by Kirk <ki...@gmail.com>.

Summary

JMeter Machine
CPU is idle, network is idle, no GC during "lockup". No matter, GC will drive CPU utilization high so the fact that CPU is idle suggest resource starvation.. i.e. all of the JMeter threads will be found to be blocked on a socket call.. i.e. the problem is in your application and not JMeter.

I think you need outside help as the problems you're facing go beyond a simple Q&A here. Sorry to spam the list but I do offer tuning services and a performance tuning course. We can talk off-line about either if your interested.

Kind regards,
Kirk Pepperdine

On Dec 15, 2011, at 9:40 PM, Robin D. Wilson wrote:

> I don't use CacheManager (since that would defeat the purpose of the test -
> which is to exercise the entire system for all assets from the pages being
> requested).
> 
> I have logged the GCs (without removing the plugins), and it doesn't do any
> GCs at all during the 'pause' period. The GC right before the pause takes
> less than a second (.26...) and the one right after is in the same ball-park
> of duration. But during the pause, no GCs at all occur.
> 
> CPU goes to idle on the JMeter box during the pause (as does network and
> disk I/O, and swap usage). There is a _little_ activity, but I've been
> attributing that to the Remote Desktop I'm using to get to the box, and the
> PerfMon graphs that are still collecting data. But compared to before and
> after the pause the activity level is less than 2% during the 'pause'
> period, compared to 75-80% during the 'run' period.
> 
> I will try to get the thread dump - I'm working on some production issues
> today, so I haven't had time to setup for a thread dump today.
> 
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> VOICE: 512-777-1861
> www.KingsIsle.com
> 
> 
> -----Original Message-----
> From: Philippe Mouawad [mailto:philippe.mouawad@gmail.com] 
> Sent: Thursday, December 15, 2011 2:23 PM
> To: JMeter Users List
> Subject: Re: Strange 'pause' activity during testing
> 
> Do you use CacheManager ?
> You should remove any plugin and activate GC logs to check it's not GC ?
> How is CPU on JMeter stack ?
> 
> Regards
> Philippe
> 
> 
> On Thu, Dec 15, 2011 at 9:09 PM, Robin D. Wilson <rw...@gmail.com> wrote:
> 
>>> Maybe then there is a problem with the scanning of the HTML to 
>>> extract the
>> embedded resources, or maybe one of the embedded resources is a tar-pit.
>> 
>> If this were the case, I would expect the first sample to show the
> problem.
>> The fact that it does 600+ iterations without a problem - and _then_ 
>> stalls seems like it rules out any problem with the returned HTML 
>> (especially since the only difference in the returned result is the 
>> username supplied by JMeter).
>> 
>>> Do all the expected embedded resources get downloaded?
>> 
>> Zero errors (even with the pauses there are no errors at all).
>> 
>>> Are there any unusually long elapsed times for embedded resources?
>> 
>> I see the 'max request duration' jump up right after the pause - but 
>> it is only 10-11 seconds (not ~35 seconds like the pause).
>> 
>>> Or large gaps between the parent sample download completion and the 
>>> start
>> of the first embedded resource?
>> 
>> Not that I can see... I'll see if I can get more detail on this.
>> 
>>> That would suggest the page took a while to parse.
>> 
>> I would assume that because this happens ~600+ iterations into the 
>> test (the first time), that if it was related to parsing the page, I 
>> would see it earlier in the test run cycle. And I wouldn't expect a 
>> parsing problem to repeat on such a consistent basis - without 
>> happening on every sample.
>> Right
>> now, if it is a parsing problem, it only happens the ~600th time it 
>> sees the same page, which seems really surprising to me (then it 
>> happens again after another ~600 iterations, etc.). Also, I would 
>> expect the parser to take up some CPU and perhaps even some I/O 
>> cycles, but the PerfMon shows idle during the pause period.
>> 
>>> You'll need to select the optiion to "save subresults" in order to 
>>> see the
>> embedded samples.
>> 
>> ...
>> 
>> 
>>> BTW, you wrote you were running JMeter 2.4.1 - that does not exist, 
>>> so
>> perhaps you meant the current version, 2.5.1?
>> 
>> Sorry, I meant 2.4.  I didn't upgrade to 2.5 (and beyond) because of a 
>> previously reported problem where 2.5+ slows down my throughput to 
>> about 60% of what I get on 2.4.
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>> 
>> 
> 
> 
> --
> Cordialement.
> Philippe Mouawad.
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

RE: Strange 'pause' activity during testing

Posted by "Robin D. Wilson" <rw...@gmail.com>.

I don't use CacheManager (since that would defeat the purpose of the test -
which is to exercise the entire system for all assets from the pages being
requested).

I have logged the GCs (without removing the plugins), and it doesn't do any
GCs at all during the 'pause' period. The GC right before the pause takes
less than a second (.26...) and the one right after is in the same ball-park
of duration. But during the pause, no GCs at all occur.

CPU goes to idle on the JMeter box during the pause (as does network and
disk I/O, and swap usage). There is a _little_ activity, but I've been
attributing that to the Remote Desktop I'm using to get to the box, and the
PerfMon graphs that are still collecting data. But compared to before and
after the pause the activity level is less than 2% during the 'pause'
period, compared to 75-80% during the 'run' period.

I will try to get the thread dump - I'm working on some production issues
today, so I haven't had time to setup for a thread dump today.

--
Robin D. Wilson
Sr. Director of Web Development
KingsIsle Entertainment, Inc.
VOICE: 512-777-1861
www.KingsIsle.com


-----Original Message-----
From: Philippe Mouawad [mailto:philippe.mouawad@gmail.com] 
Sent: Thursday, December 15, 2011 2:23 PM
To: JMeter Users List
Subject: Re: Strange 'pause' activity during testing

Do you use CacheManager ?
You should remove any plugin and activate GC logs to check it's not GC ?
How is CPU on JMeter stack ?

Regards
Philippe


On Thu, Dec 15, 2011 at 9:09 PM, Robin D. Wilson <rw...@gmail.com> wrote:

> >Maybe then there is a problem with the scanning of the HTML to 
> >extract the
> embedded resources, or maybe one of the embedded resources is a tar-pit.
>
> If this were the case, I would expect the first sample to show the
problem.
> The fact that it does 600+ iterations without a problem - and _then_ 
> stalls seems like it rules out any problem with the returned HTML 
> (especially since the only difference in the returned result is the 
> username supplied by JMeter).
>
> >Do all the expected embedded resources get downloaded?
>
> Zero errors (even with the pauses there are no errors at all).
>
> >Are there any unusually long elapsed times for embedded resources?
>
> I see the 'max request duration' jump up right after the pause - but 
> it is only 10-11 seconds (not ~35 seconds like the pause).
>
> >Or large gaps between the parent sample download completion and the 
> >start
> of the first embedded resource?
>
> Not that I can see... I'll see if I can get more detail on this.
>
> >That would suggest the page took a while to parse.
>
> I would assume that because this happens ~600+ iterations into the 
> test (the first time), that if it was related to parsing the page, I 
> would see it earlier in the test run cycle. And I wouldn't expect a 
> parsing problem to repeat on such a consistent basis - without 
> happening on every sample.
> Right
> now, if it is a parsing problem, it only happens the ~600th time it 
> sees the same page, which seems really surprising to me (then it 
> happens again after another ~600 iterations, etc.). Also, I would 
> expect the parser to take up some CPU and perhaps even some I/O 
> cycles, but the PerfMon shows idle during the pause period.
>
> >You'll need to select the optiion to "save subresults" in order to 
> >see the
> embedded samples.
>
> ...
>
>
> >BTW, you wrote you were running JMeter 2.4.1 - that does not exist, 
> >so
> perhaps you meant the current version, 2.5.1?
>
> Sorry, I meant 2.4.  I didn't upgrade to 2.5 (and beyond) because of a 
> previously reported problem where 2.5+ slows down my throughput to 
> about 60% of what I get on 2.4.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>


--
Cordialement.
Philippe Mouawad.


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by Philippe Mouawad <ph...@gmail.com>.

Do you use CacheManager ?
You should remove any plugin and activate GC logs to check it's not GC ?
How is CPU on JMeter stack ?

Regards
Philippe


On Thu, Dec 15, 2011 at 9:09 PM, Robin D. Wilson <rw...@gmail.com> wrote:

> >Maybe then there is a problem with the scanning of the HTML to extract the
> embedded resources, or maybe one of the embedded resources is a tar-pit.
>
> If this were the case, I would expect the first sample to show the problem.
> The fact that it does 600+ iterations without a problem - and _then_ stalls
> seems like it rules out any problem with the returned HTML (especially
> since
> the only difference in the returned result is the username supplied by
> JMeter).
>
> >Do all the expected embedded resources get downloaded?
>
> Zero errors (even with the pauses there are no errors at all).
>
> >Are there any unusually long elapsed times for embedded resources?
>
> I see the 'max request duration' jump up right after the pause - but it is
> only 10-11 seconds (not ~35 seconds like the pause).
>
> >Or large gaps between the parent sample download completion and the start
> of the first embedded resource?
>
> Not that I can see... I'll see if I can get more detail on this.
>
> >That would suggest the page took a while to parse.
>
> I would assume that because this happens ~600+ iterations into the test
> (the
> first time), that if it was related to parsing the page, I would see it
> earlier in the test run cycle. And I wouldn't expect a parsing problem to
> repeat on such a consistent basis - without happening on every sample.
> Right
> now, if it is a parsing problem, it only happens the ~600th time it sees
> the
> same page, which seems really surprising to me (then it happens again after
> another ~600 iterations, etc.). Also, I would expect the parser to take up
> some CPU and perhaps even some I/O cycles, but the PerfMon shows idle
> during
> the pause period.
>
> >You'll need to select the optiion to "save subresults" in order to see the
> embedded samples.
>
> ...
>
>
> >BTW, you wrote you were running JMeter 2.4.1 - that does not exist, so
> perhaps you meant the current version, 2.5.1?
>
> Sorry, I meant 2.4.  I didn't upgrade to 2.5 (and beyond) because of a
> previously reported problem where 2.5+ slows down my throughput to about
> 60%
> of what I get on 2.4.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>


-- 
Cordialement.
Philippe Mouawad.

RE: Strange 'pause' activity during testing

Posted by "Robin D. Wilson" <rw...@gmail.com>.

>Maybe then there is a problem with the scanning of the HTML to extract the
embedded resources, or maybe one of the embedded resources is a tar-pit.

If this were the case, I would expect the first sample to show the problem.
The fact that it does 600+ iterations without a problem - and _then_ stalls
seems like it rules out any problem with the returned HTML (especially since
the only difference in the returned result is the username supplied by
JMeter).

>Do all the expected embedded resources get downloaded?

Zero errors (even with the pauses there are no errors at all).

>Are there any unusually long elapsed times for embedded resources?

I see the 'max request duration' jump up right after the pause - but it is
only 10-11 seconds (not ~35 seconds like the pause).

>Or large gaps between the parent sample download completion and the start
of the first embedded resource?

Not that I can see... I'll see if I can get more detail on this.

>That would suggest the page took a while to parse.

I would assume that because this happens ~600+ iterations into the test (the
first time), that if it was related to parsing the page, I would see it
earlier in the test run cycle. And I wouldn't expect a parsing problem to
repeat on such a consistent basis - without happening on every sample. Right
now, if it is a parsing problem, it only happens the ~600th time it sees the
same page, which seems really surprising to me (then it happens again after
another ~600 iterations, etc.). Also, I would expect the parser to take up
some CPU and perhaps even some I/O cycles, but the PerfMon shows idle during
the pause period.

>You'll need to select the optiion to "save subresults" in order to see the
embedded samples.

...


>BTW, you wrote you were running JMeter 2.4.1 - that does not exist, so
perhaps you meant the current version, 2.5.1?

Sorry, I meant 2.4.  I didn't upgrade to 2.5 (and beyond) because of a
previously reported problem where 2.5+ slows down my throughput to about 60%
of what I get on 2.4.


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by sebb <se...@gmail.com>.

On 15 December 2011 00:22, Robin D. Wilson <rw...@gmail.com> wrote:
> I can use non-GUI mode for testing out the problem, but with the exact same
> test - just switching the 'Retrieve All Embedded...' disabled, the problem
> goes away.
>
> The only listeners I have enabled are the summary listener - the PerfMon
> listeners (1 for each server) - and a 'tree' listener, but it only reports
> 'errors'.
>
> I can run the test with 'Retrieve All Embedded...' enabled, I see the
> problem every time.

Maybe then there is a problem with the scanning of the HTML to extract
the embedded resources, or maybe one of the embedded resources is a
tar-pit.

Do all the expected embedded resources get downloaded?

Are there any unusually long elapsed times for embedded resources?

Or large gaps between the parent sample download completion and the
start of the first embedded resource?

That would suggest the page took a while to parse.

You'll need to select the optiion to "save subresults" in order to see
the embedded samples.

> If I just uncheck the box in the Request Defaults and re-run the test, it
> never happens... So are you suggesting that I've hit some arbitrary limit of
> the GUI mode?

No, just that GUI mode is inherently more resource intensive.

BTW, you wrote you were running JMeter 2.4.1 - that does not exist, so
perhaps you meant the current version, 2.5.1?


> What I'm wondering is why:
>
> 1) It is so consistent - the 'stop' happens at nearly the exact same point
> in every test run, and for nearly the exact same duration - and it happens
> multiple times during the test run, in the same pattern.
>
> 2) I can increase the load 10X (even more if I want), but without the
> 'Retrieve All Embedded...' enabled, and it doesn’t happen.
>
> If it were resource related on the JMeter box, I would guess that changing
> the JVM memory config would show significant changes in behavior (like it
> would take longer to encounter the first 'stop'). If it were resource
> related on the servers, I would expect to see some sort of contention on the
> servers.
>
> Tomorrow I will try the thread dump stuff - to see if there is anything
> obvious in that output.
>
> From an "external" view - it looks like JMeter has some sort of 'pause'
> built into it. Since all machines involved are nearly completely 'idle' when
> the stop happens - it seems unlikely that I've run out of resources. The
> other thought is that there is a deadlock somewhere - and something has to
> timeout before it can free up the deadlock and start again. But if that were
> the case, I would expect it to happen at slightly different intervals in
> each test run - and not repeat in the same consistent pattern during the
> test run.
>
> Some thoughts on what I think I've ruled out:
>
> 1) If it were the network 'backing off' I would expect the PerfMon listener
> to stop updating during the 'stop' - but it trundles happily along.
> 2) If it were JVM garbage collections - I would have expected to see them
> running (or at least a 'stop-the-world' gc right before/after the 'stop'
> period) during the 'stop' period - but there are no gc events at all during
> the 'stop' period.
> 3) If it were WinXP resources, I would have expected the PerfMon listener to
> show some activity during the 'stop' period on the WinXP (JMeter client) box
> - nothing like that appears.
>
> I'll look at the thread dumps tomorrow and get back to you guys. Thanks for
> helping me ponder this.
>
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> VOICE: 512-777-1861
> www.KingsIsle.com
>
>
> -----Original Message-----
> From: sebb [mailto:sebbaz@gmail.com]
> Sent: Wednesday, December 14, 2011 5:27 PM
> To: JMeter Users List
> Subject: Re: Strange 'pause' activity during testing
>
> On 14 December 2011 22:20, Deepak Shetty <sh...@gmail.com> wrote:
>> I think the original post mentioned that GC was not running.
>>
>> can you take a thread dump when there is a pause and see what the
>> threads are waiting on?
>>
>
> I would also try running JMeter in non-GUI mode; GUI mode is very expensive
> in resources.
>
> Ensure all non-essential Listeners are disabled, see:
>
> http://jmeter.apache.org/usermanual/best-practices.html#lean_mean
>
>>
>> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com> wrote:
>>
>>> might be GC.. me be that JMeter's threads are all hung up in your
>>> server doing stuff.
>>>
>>> Regards,
>>> Kirk
>>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>>>
>>> > I have a marginally complicated test case that performs a
> 'registration'
>>> on
>>> > my site. It gets the home page, POSTS a form, gets the response,
>>> > POSTS another form, gets that response, and then completes. The
>>> > test runs fine with 100 threads, and 30000 iterations - IF I don't
>>> > "Retrieve All
>>> Embedded
>>> > Resources from HTML Files". In this mode, I am really testing the
>>> throughput
>>> > of my 'tomcat' application, not the other elements of my system.
>>> > (I'm assuming that the other elements are being retrieved from our
>>> > Content
>>> Data
>>> > Network instead of our main system in this case.)
>>> >
>>> > If I enable the "Retrieve All Embedded Resources from HTML Files"
>>> > flag,
>>> and
>>> > tune the test down to 10 threads with 3000 total iterations, I
>>> > notice a
>>> very
>>> > strange behavior. The test runs along at a pretty good clip for the
>>> > first
>>> > ~600 iterations (about 1 min, 25 seconds into the run), and then it
>>> > just stops making requests for about 35 seconds. Then, it picks
>>> > back up again
>>> and
>>> > runs for another 1 m 25s, and then stops again for 35 seconds... (NOTE:
>>> with
>>> > the 10 threads, 3000 total iterations - but with "Retrieve All
>>> Embedded..."
>>> > disabled - I don't see the 'stop' behavior either - so it isn't
>>> > caused by tuning it down...)
>>> >
>>> > I recently added the "Perfmon Metrics Collector" to the test
>>> > script, so I could see if one of the servers was maxed out - but it
>>> > looks like all the servers are idle during the 'stop' period.
>>> > Likewise, I added the Perfmon
>>> for
>>> > the "localhost" (running the JMeter test) to see if it was swamped
>>> > - but
>>> it
>>> > too is idle during the 'stop'. I swapped out our network switch
>>> > (the test environment is on an isolated network switch) with a
>>> > _much_ higher
>>> capacity
>>> > switch - in case there was a network issue, still no change.
>>> >
>>> > I'm running out of ideas for things to check - so I thought I'd ask
>>> > you
>>> guys
>>> > if you have any suggestions of things I should look at.
>>> >
>>> > My system consists of:
>>> >
>>> >       WinXP - running JMeter 2.4.1 - driving the test script in GUI
>>> > mode
>>> >       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as
>>> > the web server - using AJP Proxy to Server 2
>>> >       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the
>>> > App Server - connecting through Hibernate to Server 3
>>> >       Server 3 - running Red Hat Linux with MySQL 5.x as the DB
>>> > Server
>>> >
>>> > All 4 machines are running on a private switched network (32Gbs
>>> backplane).
>>> >
>>> > The requests are downloading about 3MB total (per thread per
>>> > iteration)
>>> over
>>> > 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
>>> >
>>> > At first I thought it was the network - but the new switch seemed
>>> > to deny that thought (the old switch had a much slower backplane).
>>> > Also, I'm
>>> having
>>> > no trouble collecting the PerfMon data during the 'stop' period -
>>> > so the network is still functioning just fine...
>>> > Then I thought it might be garbage collection on the tomcat - but I
>>> watched
>>> > the gc.log - and it doesn't do any GCs during the 'stop' period.
>>> > Then I thought it might be the garbage collection on the JMeter
>>> > side, so
>>> I
>>> > started the JMeter.bat from a 'cmd' prompt with gc logging enabled
>>> > - it doesn't do any GCs during the 'stop' period either.
>>> >
>>> > The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no
>>> > network I/O, no disk I/O, etc.) during the 'stop' period.
>>> >
>>> > The JMeter box (WinXP) is doing very little during that time too (I
>>> > attribute the little bit of activity to displaying the PerfMon
>>> > graphs,
>>> and
>>> > Remote Desktop display to my desktop computer)...
>>> >
>>> > --
>>> > Robin D. Wilson
>>> > Sr. Director of Web Development
>>> > KingsIsle Entertainment, Inc.
>>> > VOICE: 512-777-1861
>>> > www.KingsIsle.com
>>> >
>>> >
>>> >
>>> >
>>> > -------------------------------------------------------------------
>>> > -- To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> > For additional commands, e-mail: user-help@jmeter.apache.org
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

RE: Strange 'pause' activity during testing

Posted by "Robin D. Wilson" <rw...@gmail.com>.

I can use non-GUI mode for testing out the problem, but with the exact same
test - just switching the 'Retrieve All Embedded...' disabled, the problem
goes away. 

The only listeners I have enabled are the summary listener - the PerfMon
listeners (1 for each server) - and a 'tree' listener, but it only reports
'errors'.

I can run the test with 'Retrieve All Embedded...' enabled, I see the
problem every time.

If I just uncheck the box in the Request Defaults and re-run the test, it
never happens... So are you suggesting that I've hit some arbitrary limit of
the GUI mode?

What I'm wondering is why:

1) It is so consistent - the 'stop' happens at nearly the exact same point
in every test run, and for nearly the exact same duration - and it happens
multiple times during the test run, in the same pattern.

2) I can increase the load 10X (even more if I want), but without the
'Retrieve All Embedded...' enabled, and it doesnt happen.

If it were resource related on the JMeter box, I would guess that changing
the JVM memory config would show significant changes in behavior (like it
would take longer to encounter the first 'stop'). If it were resource
related on the servers, I would expect to see some sort of contention on the
servers.

Tomorrow I will try the thread dump stuff - to see if there is anything
obvious in that output.

>From an "external" view - it looks like JMeter has some sort of 'pause'
built into it. Since all machines involved are nearly completely 'idle' when
the stop happens - it seems unlikely that I've run out of resources. The
other thought is that there is a deadlock somewhere - and something has to
timeout before it can free up the deadlock and start again. But if that were
the case, I would expect it to happen at slightly different intervals in
each test run - and not repeat in the same consistent pattern during the
test run.

Some thoughts on what I think I've ruled out:

1) If it were the network 'backing off' I would expect the PerfMon listener
to stop updating during the 'stop' - but it trundles happily along.
2) If it were JVM garbage collections - I would have expected to see them
running (or at least a 'stop-the-world' gc right before/after the 'stop'
period) during the 'stop' period - but there are no gc events at all during
the 'stop' period.
3) If it were WinXP resources, I would have expected the PerfMon listener to
show some activity during the 'stop' period on the WinXP (JMeter client) box
- nothing like that appears.

I'll look at the thread dumps tomorrow and get back to you guys. Thanks for
helping me ponder this.

--
Robin D. Wilson
Sr. Director of Web Development
KingsIsle Entertainment, Inc.
VOICE: 512-777-1861
www.KingsIsle.com


-----Original Message-----
From: sebb [mailto:sebbaz@gmail.com] 
Sent: Wednesday, December 14, 2011 5:27 PM
To: JMeter Users List
Subject: Re: Strange 'pause' activity during testing

On 14 December 2011 22:20, Deepak Shetty <sh...@gmail.com> wrote:
> I think the original post mentioned that GC was not running.
>
> can you take a thread dump when there is a pause and see what the 
> threads are waiting on?
>

I would also try running JMeter in non-GUI mode; GUI mode is very expensive
in resources.

Ensure all non-essential Listeners are disabled, see:

http://jmeter.apache.org/usermanual/best-practices.html#lean_mean

>
> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com> wrote:
>
>> might be GC.. me be that JMeter's threads are all hung up in your 
>> server doing stuff.
>>
>> Regards,
>> Kirk
>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>>
>> > I have a marginally complicated test case that performs a
'registration'
>> on
>> > my site. It gets the home page, POSTS a form, gets the response, 
>> > POSTS another form, gets that response, and then completes. The 
>> > test runs fine with 100 threads, and 30000 iterations - IF I don't 
>> > "Retrieve All
>> Embedded
>> > Resources from HTML Files". In this mode, I am really testing the
>> throughput
>> > of my 'tomcat' application, not the other elements of my system. 
>> > (I'm assuming that the other elements are being retrieved from our 
>> > Content
>> Data
>> > Network instead of our main system in this case.)
>> >
>> > If I enable the "Retrieve All Embedded Resources from HTML Files" 
>> > flag,
>> and
>> > tune the test down to 10 threads with 3000 total iterations, I 
>> > notice a
>> very
>> > strange behavior. The test runs along at a pretty good clip for the 
>> > first
>> > ~600 iterations (about 1 min, 25 seconds into the run), and then it 
>> > just stops making requests for about 35 seconds. Then, it picks 
>> > back up again
>> and
>> > runs for another 1 m 25s, and then stops again for 35 seconds... (NOTE:
>> with
>> > the 10 threads, 3000 total iterations - but with "Retrieve All
>> Embedded..."
>> > disabled - I don't see the 'stop' behavior either - so it isn't 
>> > caused by tuning it down...)
>> >
>> > I recently added the "Perfmon Metrics Collector" to the test 
>> > script, so I could see if one of the servers was maxed out - but it 
>> > looks like all the servers are idle during the 'stop' period. 
>> > Likewise, I added the Perfmon
>> for
>> > the "localhost" (running the JMeter test) to see if it was swamped 
>> > - but
>> it
>> > too is idle during the 'stop'. I swapped out our network switch 
>> > (the test environment is on an isolated network switch) with a 
>> > _much_ higher
>> capacity
>> > switch - in case there was a network issue, still no change.
>> >
>> > I'm running out of ideas for things to check - so I thought I'd ask 
>> > you
>> guys
>> > if you have any suggestions of things I should look at.
>> >
>> > My system consists of:
>> >
>> >       WinXP - running JMeter 2.4.1 - driving the test script in GUI 
>> > mode
>> >       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as 
>> > the web server - using AJP Proxy to Server 2
>> >       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the 
>> > App Server - connecting through Hibernate to Server 3
>> >       Server 3 - running Red Hat Linux with MySQL 5.x as the DB 
>> > Server
>> >
>> > All 4 machines are running on a private switched network (32Gbs
>> backplane).
>> >
>> > The requests are downloading about 3MB total (per thread per 
>> > iteration)
>> over
>> > 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
>> >
>> > At first I thought it was the network - but the new switch seemed 
>> > to deny that thought (the old switch had a much slower backplane). 
>> > Also, I'm
>> having
>> > no trouble collecting the PerfMon data during the 'stop' period - 
>> > so the network is still functioning just fine...
>> > Then I thought it might be garbage collection on the tomcat - but I
>> watched
>> > the gc.log - and it doesn't do any GCs during the 'stop' period.
>> > Then I thought it might be the garbage collection on the JMeter 
>> > side, so
>> I
>> > started the JMeter.bat from a 'cmd' prompt with gc logging enabled 
>> > - it doesn't do any GCs during the 'stop' period either.
>> >
>> > The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no 
>> > network I/O, no disk I/O, etc.) during the 'stop' period.
>> >
>> > The JMeter box (WinXP) is doing very little during that time too (I 
>> > attribute the little bit of activity to displaying the PerfMon 
>> > graphs,
>> and
>> > Remote Desktop display to my desktop computer)...
>> >
>> > --
>> > Robin D. Wilson
>> > Sr. Director of Web Development
>> > KingsIsle Entertainment, Inc.
>> > VOICE: 512-777-1861
>> > www.KingsIsle.com
>> >
>> >
>> >
>> >
>> > -------------------------------------------------------------------
>> > -- To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> > For additional commands, e-mail: user-help@jmeter.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by sebb <se...@gmail.com>.

On 14 December 2011 22:20, Deepak Shetty <sh...@gmail.com> wrote:
> I think the original post mentioned that GC was not running.
>
> can you take a thread dump when there is a pause and see what the threads
> are waiting on?
>

I would also try running JMeter in non-GUI mode; GUI mode is very
expensive in resources.

Ensure all non-essential Listeners are disabled, see:

http://jmeter.apache.org/usermanual/best-practices.html#lean_mean

>
> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com> wrote:
>
>> might be GC.. me be that JMeter's threads are all hung up in your server
>> doing stuff.
>>
>> Regards,
>> Kirk
>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>>
>> > I have a marginally complicated test case that performs a 'registration'
>> on
>> > my site. It gets the home page, POSTS a form, gets the response, POSTS
>> > another form, gets that response, and then completes. The test runs fine
>> > with 100 threads, and 30000 iterations - IF I don't "Retrieve All
>> Embedded
>> > Resources from HTML Files". In this mode, I am really testing the
>> throughput
>> > of my 'tomcat' application, not the other elements of my system. (I'm
>> > assuming that the other elements are being retrieved from our Content
>> Data
>> > Network instead of our main system in this case.)
>> >
>> > If I enable the "Retrieve All Embedded Resources from HTML Files" flag,
>> and
>> > tune the test down to 10 threads with 3000 total iterations, I notice a
>> very
>> > strange behavior. The test runs along at a pretty good clip for the first
>> > ~600 iterations (about 1 min, 25 seconds into the run), and then it just
>> > stops making requests for about 35 seconds. Then, it picks back up again
>> and
>> > runs for another 1 m 25s, and then stops again for 35 seconds... (NOTE:
>> with
>> > the 10 threads, 3000 total iterations - but with "Retrieve All
>> Embedded..."
>> > disabled - I don't see the 'stop' behavior either - so it isn't caused by
>> > tuning it down...)
>> >
>> > I recently added the "Perfmon Metrics Collector" to the test script, so I
>> > could see if one of the servers was maxed out - but it looks like all the
>> > servers are idle during the 'stop' period. Likewise, I added the Perfmon
>> for
>> > the "localhost" (running the JMeter test) to see if it was swamped - but
>> it
>> > too is idle during the 'stop'. I swapped out our network switch (the test
>> > environment is on an isolated network switch) with a _much_ higher
>> capacity
>> > switch - in case there was a network issue, still no change.
>> >
>> > I'm running out of ideas for things to check - so I thought I'd ask you
>> guys
>> > if you have any suggestions of things I should look at.
>> >
>> > My system consists of:
>> >
>> >       WinXP - running JMeter 2.4.1 - driving the test script in GUI mode
>> >       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as the web
>> > server - using AJP Proxy to Server 2
>> >       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the App
>> > Server - connecting through Hibernate to Server 3
>> >       Server 3 - running Red Hat Linux with MySQL 5.x as the DB Server
>> >
>> > All 4 machines are running on a private switched network (32Gbs
>> backplane).
>> >
>> > The requests are downloading about 3MB total (per thread per iteration)
>> over
>> > 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
>> >
>> > At first I thought it was the network - but the new switch seemed to deny
>> > that thought (the old switch had a much slower backplane). Also, I'm
>> having
>> > no trouble collecting the PerfMon data during the 'stop' period - so the
>> > network is still functioning just fine...
>> > Then I thought it might be garbage collection on the tomcat - but I
>> watched
>> > the gc.log - and it doesn't do any GCs during the 'stop' period.
>> > Then I thought it might be the garbage collection on the JMeter side, so
>> I
>> > started the JMeter.bat from a 'cmd' prompt with gc logging enabled - it
>> > doesn't do any GCs during the 'stop' period either.
>> >
>> > The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no network
>> > I/O, no disk I/O, etc.) during the 'stop' period.
>> >
>> > The JMeter box (WinXP) is doing very little during that time too (I
>> > attribute the little bit of activity to displaying the PerfMon graphs,
>> and
>> > Remote Desktop display to my desktop computer)...
>> >
>> > --
>> > Robin D. Wilson
>> > Sr. Director of Web Development
>> > KingsIsle Entertainment, Inc.
>> > VOICE: 512-777-1861
>> > www.KingsIsle.com
>> >
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> > For additional commands, e-mail: user-help@jmeter.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: Strange 'pause' activity during testing

Posted by Deepak Shetty <sh...@gmail.com>.

I think the original post mentioned that GC was not running.
can you take a thread dump when there is a pause and see what the threads
are waiting on?



On Wed, Dec 14, 2011 at 1:56 PM, Kirk <ki...@gmail.com> wrote:

> might be GC.. me be that JMeter's threads are all hung up in your server
> doing stuff.
>
> Regards,
> Kirk
> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:
>
> > I have a marginally complicated test case that performs a 'registration'
> on
> > my site. It gets the home page, POSTS a form, gets the response, POSTS
> > another form, gets that response, and then completes. The test runs fine
> > with 100 threads, and 30000 iterations - IF I don't "Retrieve All
> Embedded
> > Resources from HTML Files". In this mode, I am really testing the
> throughput
> > of my 'tomcat' application, not the other elements of my system. (I'm
> > assuming that the other elements are being retrieved from our Content
> Data
> > Network instead of our main system in this case.)
> >
> > If I enable the "Retrieve All Embedded Resources from HTML Files" flag,
> and
> > tune the test down to 10 threads with 3000 total iterations, I notice a
> very
> > strange behavior. The test runs along at a pretty good clip for the first
> > ~600 iterations (about 1 min, 25 seconds into the run), and then it just
> > stops making requests for about 35 seconds. Then, it picks back up again
> and
> > runs for another 1 m 25s, and then stops again for 35 seconds... (NOTE:
> with
> > the 10 threads, 3000 total iterations - but with "Retrieve All
> Embedded..."
> > disabled - I don't see the 'stop' behavior either - so it isn't caused by
> > tuning it down...)
> >
> > I recently added the "Perfmon Metrics Collector" to the test script, so I
> > could see if one of the servers was maxed out - but it looks like all the
> > servers are idle during the 'stop' period. Likewise, I added the Perfmon
> for
> > the "localhost" (running the JMeter test) to see if it was swamped - but
> it
> > too is idle during the 'stop'. I swapped out our network switch (the test
> > environment is on an isolated network switch) with a _much_ higher
> capacity
> > switch - in case there was a network issue, still no change.
> >
> > I'm running out of ideas for things to check - so I thought I'd ask you
> guys
> > if you have any suggestions of things I should look at.
> >
> > My system consists of:
> >
> >       WinXP - running JMeter 2.4.1 - driving the test script in GUI mode
> >       Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as the web
> > server - using AJP Proxy to Server 2
> >       Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the App
> > Server - connecting through Hibernate to Server 3
> >       Server 3 - running Red Hat Linux with MySQL 5.x as the DB Server
> >
> > All 4 machines are running on a private switched network (32Gbs
> backplane).
> >
> > The requests are downloading about 3MB total (per thread per iteration)
> over
> > 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
> >
> > At first I thought it was the network - but the new switch seemed to deny
> > that thought (the old switch had a much slower backplane). Also, I'm
> having
> > no trouble collecting the PerfMon data during the 'stop' period - so the
> > network is still functioning just fine...
> > Then I thought it might be garbage collection on the tomcat - but I
> watched
> > the gc.log - and it doesn't do any GCs during the 'stop' period.
> > Then I thought it might be the garbage collection on the JMeter side, so
> I
> > started the JMeter.bat from a 'cmd' prompt with gc logging enabled - it
> > doesn't do any GCs during the 'stop' period either.
> >
> > The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no network
> > I/O, no disk I/O, etc.) during the 'stop' period.
> >
> > The JMeter box (WinXP) is doing very little during that time too (I
> > attribute the little bit of activity to displaying the PerfMon graphs,
> and
> > Remote Desktop display to my desktop computer)...
> >
> > --
> > Robin D. Wilson
> > Sr. Director of Web Development
> > KingsIsle Entertainment, Inc.
> > VOICE: 512-777-1861
> > www.KingsIsle.com
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > For additional commands, e-mail: user-help@jmeter.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>

Re: Strange 'pause' activity during testing

Posted by Kirk <ki...@gmail.com>.

might be GC.. me be that JMeter's threads are all hung up in your server doing stuff.

Regards,
Kirk
On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote:

> I have a marginally complicated test case that performs a 'registration' on
> my site. It gets the home page, POSTS a form, gets the response, POSTS
> another form, gets that response, and then completes. The test runs fine
> with 100 threads, and 30000 iterations - IF I don't "Retrieve All Embedded
> Resources from HTML Files". In this mode, I am really testing the throughput
> of my 'tomcat' application, not the other elements of my system. (I'm
> assuming that the other elements are being retrieved from our Content Data
> Network instead of our main system in this case.)
> 
> If I enable the "Retrieve All Embedded Resources from HTML Files" flag, and
> tune the test down to 10 threads with 3000 total iterations, I notice a very
> strange behavior. The test runs along at a pretty good clip for the first
> ~600 iterations (about 1 min, 25 seconds into the run), and then it just
> stops making requests for about 35 seconds. Then, it picks back up again and
> runs for another 1 m 25s, and then stops again for 35 seconds... (NOTE: with
> the 10 threads, 3000 total iterations - but with "Retrieve All Embedded..."
> disabled - I don't see the 'stop' behavior either - so it isn't caused by
> tuning it down...)
> 
> I recently added the "Perfmon Metrics Collector" to the test script, so I
> could see if one of the servers was maxed out - but it looks like all the
> servers are idle during the 'stop' period. Likewise, I added the Perfmon for
> the "localhost" (running the JMeter test) to see if it was swamped - but it
> too is idle during the 'stop'. I swapped out our network switch (the test
> environment is on an isolated network switch) with a _much_ higher capacity
> switch - in case there was a network issue, still no change.
> 
> I'm running out of ideas for things to check - so I thought I'd ask you guys
> if you have any suggestions of things I should look at.
> 
> My system consists of:
> 
> 	WinXP - running JMeter 2.4.1 - driving the test script in GUI mode
> 	Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as the web
> server - using AJP Proxy to Server 2
> 	Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the App
> Server - connecting through Hibernate to Server 3
> 	Server 3 - running Red Hat Linux with MySQL 5.x as the DB Server
> 
> All 4 machines are running on a private switched network (32Gbs backplane).
> 
> The requests are downloading about 3MB total (per thread per iteration) over
> 4 main URL requests, and 30+ 'Retrieve All Embedded' requests.
> 
> At first I thought it was the network - but the new switch seemed to deny
> that thought (the old switch had a much slower backplane). Also, I'm having
> no trouble collecting the PerfMon data during the 'stop' period - so the
> network is still functioning just fine...
> Then I thought it might be garbage collection on the tomcat - but I watched
> the gc.log - and it doesn't do any GCs during the 'stop' period.
> Then I thought it might be the garbage collection on the JMeter side, so I
> started the JMeter.bat from a 'cmd' prompt with gc logging enabled - it
> doesn't do any GCs during the 'stop' period either.
> 
> The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no network
> I/O, no disk I/O, etc.) during the 'stop' period.
> 
> The JMeter box (WinXP) is doing very little during that time too (I
> attribute the little bit of activity to displaying the PerfMon graphs, and
> Remote Desktop display to my desktop computer)...
> 
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> VOICE: 512-777-1861
> www.KingsIsle.com
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org