You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Noel O'Brien <no...@newbay.com> on 2009/03/16 10:47:45 UTC
Stress Testing in a Virtual Server
Hi All,
I've been running some load testing in a virtual server (Xen) as that's where
our product will be deployed. I've noticed some unusual behavior and I can't
seem to be able to pinpoint what's going wrong.
Basically, I start 1 JMeter instance running 10 threads against the product (a
http REST server in Tomcat). It connects to several auxillary programs (DB,
other HTTP servers, SwiftMQ, etc), all of which are on different virtual
machines within the same subnet. I'm using top and lsof to monitor the
resources of each running application. Virtualised OS is RHEL 5, running
Tomcat 6.0.16 and Java 1.6.0_07
The problem is that every few mins (about 3) the CPU usage of all the
processes I'm monitoring (product, JMeter, DB, etc.) all drop to 0% for about
30 seconds. None of the logs suggest that anything is wrong, so it's
incredibly hard to track down what's causing this. I'm thinking that this
issue is caused by the virtual servers upon which the programs are running,
perhaps mis- / mal-configuration of the virtual network cards.
Has anyone else experienced anything like this while stress testing in a
virtualised environment?
Regards,
Noel
Re: Stress Testing in a Virtual Server
Posted by Noel O'Brien <no...@newbay.com>.
Hi All,
After much investigation and help from our ops department, it was found that
the ip_conntrack buffer on the host OS was filling up under load tests.
Disabling iptables on the host (/etc/init.d/iptables stop) resolved the issue
and did not seem to impact the hosted VMs.
Regards,
Noel
On Monday 16 March 2009 12:32:54 sebb wrote:
> On 16/03/2009, Noel O'Brien <no...@newbay.com> wrote:
> > On Monday 16 March 2009 12:01:53 sebb wrote:
> > > On 16/03/2009, Noel O'Brien <no...@newbay.com> wrote:
> > > > Hi All,
> > > >
> > > >
> > > >
> > > > I've been running some load testing in a virtual server (Xen) as
> > > > that's
> > > >
> > > > where our product will be deployed. I've noticed some unusual
> > > > behavior
> > > >
> > > > and I can't seem to be able to pinpoint what's going wrong.
> > > >
> > > >
> > > >
> > > > Basically, I start 1 JMeter instance running 10 threads against the
> > > >
> > > > product (a http REST server in Tomcat). It connects to several
> > > > auxillary
> > > >
> > > > programs (DB, other HTTP servers, SwiftMQ, etc), all of which are on
> > > >
> > > > different virtual machines within the same subnet. I'm using top and
> >
> > lsof
> >
> > > > to monitor the resources of each running application. Virtualised OS
> > > > is
> > > >
> > > > RHEL 5, running Tomcat 6.0.16 and Java 1.6.0_07
> > > >
> > > >
> > > >
> > > > The problem is that every few mins (about 3) the CPU usage of all the
> > > >
> > > > processes I'm monitoring (product, JMeter, DB, etc.) all drop to 0%
> > > > for
> > > >
> > > > about 30 seconds. None of the logs suggest that anything is wrong, so
> > > >
> > > > it's incredibly hard to track down what's causing this. I'm thinking
> >
> > that
> >
> > > > this issue is caused by the virtual servers upon which the programs
> > > > are
> > > >
> > > > running, perhaps mis- / mal-configuration of the virtual network
> > > > cards.
> > > >
> > > >
> > > >
> > > > Has anyone else experienced anything like this while stress testing
> > > > in a
> > > >
> > > > virtualised environment?
> > >
> > > What happens to I/O during this time?
> >
> > I'm not sure. lsof blocks, so running it during this time is fruitless.
> > Is there another command I can run to check I/O?
>
> Doesn't top show I/O?
>
> Also, try running ping to each of the systems (ideally from a non-Xen
> system).
>
> This should help show if it is a single system locking up and causing
> the others to wait for it.
>
> Are all the Xen servers on the same host?
>
> > > Do top and lsof continue responding OK?
> >
> > Top continues to respond (unless it blocks and shows 0% CPU for the
> > duration), but lsof blocks until the CPU appears to be given back to the
> > processes.
> >
> > > > Regards,
> > > >
> > > >
> > > >
> > > > Noel
> >
> > ---------------------------------------------------------------------
> >
> > > To unsubscribe, e-mail:
> >
> > jmeter-user-unsubscribe@jakarta.apache.org
> >
> > > For additional commands, e-mail:
> >
> > jmeter-user-help@jakarta.apache.org
Re: Stress Testing in a Virtual Server
Posted by sebb <se...@gmail.com>.
On 16/03/2009, Noel O'Brien <no...@newbay.com> wrote:
>
>
>
> On Monday 16 March 2009 12:01:53 sebb wrote:
>
> > On 16/03/2009, Noel O'Brien <no...@newbay.com> wrote:
>
> > > Hi All,
>
> > >
>
> > > I've been running some load testing in a virtual server (Xen) as that's
>
> > > where our product will be deployed. I've noticed some unusual behavior
>
> > > and I can't seem to be able to pinpoint what's going wrong.
>
> > >
>
> > > Basically, I start 1 JMeter instance running 10 threads against the
>
> > > product (a http REST server in Tomcat). It connects to several auxillary
>
> > > programs (DB, other HTTP servers, SwiftMQ, etc), all of which are on
>
> > > different virtual machines within the same subnet. I'm using top and
> lsof
>
> > > to monitor the resources of each running application. Virtualised OS is
>
> > > RHEL 5, running Tomcat 6.0.16 and Java 1.6.0_07
>
> > >
>
> > > The problem is that every few mins (about 3) the CPU usage of all the
>
> > > processes I'm monitoring (product, JMeter, DB, etc.) all drop to 0% for
>
> > > about 30 seconds. None of the logs suggest that anything is wrong, so
>
> > > it's incredibly hard to track down what's causing this. I'm thinking
> that
>
> > > this issue is caused by the virtual servers upon which the programs are
>
> > > running, perhaps mis- / mal-configuration of the virtual network cards.
>
> > >
>
> > > Has anyone else experienced anything like this while stress testing in a
>
> > > virtualised environment?
>
> >
>
> > What happens to I/O during this time?
>
>
>
> I'm not sure. lsof blocks, so running it during this time is fruitless. Is
> there another command I can run to check I/O?
>
Doesn't top show I/O?
Also, try running ping to each of the systems (ideally from a non-Xen system).
This should help show if it is a single system locking up and causing
the others to wait for it.
Are all the Xen servers on the same host?
>
> > Do top and lsof continue responding OK?
>
>
>
> Top continues to respond (unless it blocks and shows 0% CPU for the
> duration), but lsof blocks until the CPU appears to be given back to the
> processes.
>
>
>
> > > Regards,
>
> > >
>
> > > Noel
>
> >
>
> >
> ---------------------------------------------------------------------
>
> > To unsubscribe, e-mail:
> jmeter-user-unsubscribe@jakarta.apache.org
>
> > For additional commands, e-mail:
> jmeter-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Stress Testing in a Virtual Server
Posted by Noel O'Brien <no...@newbay.com>.
On Monday 16 March 2009 12:01:53 sebb wrote:
> On 16/03/2009, Noel O'Brien <no...@newbay.com> wrote:
> > Hi All,
> >
> > I've been running some load testing in a virtual server (Xen) as that's
> > where our product will be deployed. I've noticed some unusual behavior
> > and I can't seem to be able to pinpoint what's going wrong.
> >
> > Basically, I start 1 JMeter instance running 10 threads against the
> > product (a http REST server in Tomcat). It connects to several auxillary
> > programs (DB, other HTTP servers, SwiftMQ, etc), all of which are on
> > different virtual machines within the same subnet. I'm using top and lsof
> > to monitor the resources of each running application. Virtualised OS is
> > RHEL 5, running Tomcat 6.0.16 and Java 1.6.0_07
> >
> > The problem is that every few mins (about 3) the CPU usage of all the
> > processes I'm monitoring (product, JMeter, DB, etc.) all drop to 0% for
> > about 30 seconds. None of the logs suggest that anything is wrong, so
> > it's incredibly hard to track down what's causing this. I'm thinking that
> > this issue is caused by the virtual servers upon which the programs are
> > running, perhaps mis- / mal-configuration of the virtual network cards.
> >
> > Has anyone else experienced anything like this while stress testing in a
> > virtualised environment?
>
> What happens to I/O during this time?
I'm not sure. lsof blocks, so running it during this time is fruitless. Is
there another command I can run to check I/O?
> Do top and lsof continue responding OK?
Top continues to respond (unless it blocks and shows 0% CPU for the duration),
but lsof blocks until the CPU appears to be given back to the processes.
> > Regards,
> >
> > Noel
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
Re: Stress Testing in a Virtual Server
Posted by sebb <se...@gmail.com>.
On 16/03/2009, Noel O'Brien <no...@newbay.com> wrote:
> Hi All,
>
> I've been running some load testing in a virtual server (Xen) as that's where
> our product will be deployed. I've noticed some unusual behavior and I can't
> seem to be able to pinpoint what's going wrong.
>
> Basically, I start 1 JMeter instance running 10 threads against the product (a
> http REST server in Tomcat). It connects to several auxillary programs (DB,
> other HTTP servers, SwiftMQ, etc), all of which are on different virtual
> machines within the same subnet. I'm using top and lsof to monitor the
> resources of each running application. Virtualised OS is RHEL 5, running
> Tomcat 6.0.16 and Java 1.6.0_07
>
> The problem is that every few mins (about 3) the CPU usage of all the
> processes I'm monitoring (product, JMeter, DB, etc.) all drop to 0% for about
> 30 seconds. None of the logs suggest that anything is wrong, so it's
> incredibly hard to track down what's causing this. I'm thinking that this
> issue is caused by the virtual servers upon which the programs are running,
> perhaps mis- / mal-configuration of the virtual network cards.
>
> Has anyone else experienced anything like this while stress testing in a
> virtualised environment?
What happens to I/O during this time?
Do top and lsof continue responding OK?
> Regards,
>
> Noel
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org