You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tapestry.apache.org by P....@albourne.com on 2011/07/20 11:16:24 UTC
100% CPU load after hot deploy of Tapestry 5 Application on Tomcat
6
Hi All,
This is a tricky question one since I do realize there are several possibilities to explain the problem, not all of which are Tapestry related. We have battled to understand what is going on in production for the past year or so, and have tried to pinpoint where the problem lies and come up short. If anyone has experienced anything similar *please share.
Our environment is:
Ubuntu Server
Tapestry 5.1
JDK 6
Tomcat 6
Apache in front (reverse proxy)
We are deploying our Tapestry apps to Tomcat 6, and they do run great even with load, and it remains very stable as the session count increases; this is when a server restart is done after a deployment.
The problem appears to be specific to 'Tapestry applications' and hot deployment. We run several apps, both small and large and they all react the same way... If we stop then undeploy any of these Tapestry apps, and afterwards do a hot deploy with a newer version (without the restart), the CPU usage shoots to 100% and stays there. JSP based apps pure Servlets don't appear to react in the same way.
Has anybody experienced something similar?
Kind regards,
Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: 100% CPU load after hot deploy of Tapestry 5 Application on
Tomcat 6
Posted by Peter Stavrinides <P....@albourne.com>.
Martin thanks, I will try this, will let you know.
Cheers,
Peter
----- Original Message -----
From: "Martin Strand" <do...@gmail.com>
To: "Tapestry users" <us...@tapestry.apache.org>
Sent: Wednesday, 20 July, 2011 12:52:52 GMT +02:00 Athens, Beirut, Bucharest, Istanbul
Subject: Re: 100% CPU load after hot deploy of Tapestry 5 Application on Tomcat 6
We've only got 5.2 apps runing on Jetty, but hot redeploy works great
there and is even without the PermGen leak that was present in Tapestry 4.
Anyway, here's how I would try to diagnose the problem:
1. Trigger the problem so that you have something consuming 100% CPU
2. Find the native ID of the thread that is eating all your CPU power:
$ top -H -u tomcat (or whatever user is running Tomcat)
-H will show individual *threads* and you should be able to find the
offending thread. Remember its ID.
(Hit 'P' to sort by CPU usage)
3. Find Tomcat's PID:
$ sudo jps
4. Get a thread dump from Tomcat:
$ sudo kill -QUIT <TOMCAT_PID>
The thread dump will be written to Tomcat's stderr, wherever that is
redirected
5. Inspect the thread dump
Convert the thread ID from step 2 to hexadecimal and look for it in the
thread dump from step 4. The thread you're looking for will say "nid=xxx",
where xxx is the offending thread's native ID written in hexadecimal. The
thread's stacktrace should give you a clue as to what's going on,
especially if you get more samples to see where the thread is spending
most of its time.
You can also hook up VisualVM to Tomcat and inspect threads from there.
There's a thread inspector plugin in the plugin repository.
Hope that helps!
Martin
On Wed, 20 Jul 2011 11:16:24 +0200, <P....@albourne.com> wrote:
> Hi All,
>
> This is a tricky question one since I do realize there are several
> possibilities to explain the problem, not all of which are Tapestry
> related. We have battled to understand what is going on in production
> for the past year or so, and have tried to pinpoint where the problem
> lies and come up short. If anyone has experienced anything similar
> *please share.
>
> Our environment is:
>
> Ubuntu Server
> Tapestry 5.1
> JDK 6
> Tomcat 6
> Apache in front (reverse proxy)
>
> We are deploying our Tapestry apps to Tomcat 6, and they do run great
> even with load, and it remains very stable as the session count
> increases; this is when a server restart is done after a deployment.
>
> The problem appears to be specific to 'Tapestry applications' and hot
> deployment. We run several apps, both small and large and they all react
> the same way... If we stop then undeploy any of these Tapestry apps, and
> afterwards do a hot deploy with a newer version (without the restart),
> the CPU usage shoots to 100% and stays there. JSP based apps pure
> Servlets don't appear to react in the same way.
>
> Has anybody experienced something similar?
>
> Kind regards,
> Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: 100% CPU load after hot deploy of Tapestry 5 Application on
Tomcat 6
Posted by Martin Strand <do...@gmail.com>.
We've only got 5.2 apps runing on Jetty, but hot redeploy works great
there and is even without the PermGen leak that was present in Tapestry 4.
Anyway, here's how I would try to diagnose the problem:
1. Trigger the problem so that you have something consuming 100% CPU
2. Find the native ID of the thread that is eating all your CPU power:
$ top -H -u tomcat (or whatever user is running Tomcat)
-H will show individual *threads* and you should be able to find the
offending thread. Remember its ID.
(Hit 'P' to sort by CPU usage)
3. Find Tomcat's PID:
$ sudo jps
4. Get a thread dump from Tomcat:
$ sudo kill -QUIT <TOMCAT_PID>
The thread dump will be written to Tomcat's stderr, wherever that is
redirected
5. Inspect the thread dump
Convert the thread ID from step 2 to hexadecimal and look for it in the
thread dump from step 4. The thread you're looking for will say "nid=xxx",
where xxx is the offending thread's native ID written in hexadecimal. The
thread's stacktrace should give you a clue as to what's going on,
especially if you get more samples to see where the thread is spending
most of its time.
You can also hook up VisualVM to Tomcat and inspect threads from there.
There's a thread inspector plugin in the plugin repository.
Hope that helps!
Martin
On Wed, 20 Jul 2011 11:16:24 +0200, <P....@albourne.com> wrote:
> Hi All,
>
> This is a tricky question one since I do realize there are several
> possibilities to explain the problem, not all of which are Tapestry
> related. We have battled to understand what is going on in production
> for the past year or so, and have tried to pinpoint where the problem
> lies and come up short. If anyone has experienced anything similar
> *please share.
>
> Our environment is:
>
> Ubuntu Server
> Tapestry 5.1
> JDK 6
> Tomcat 6
> Apache in front (reverse proxy)
>
> We are deploying our Tapestry apps to Tomcat 6, and they do run great
> even with load, and it remains very stable as the session count
> increases; this is when a server restart is done after a deployment.
>
> The problem appears to be specific to 'Tapestry applications' and hot
> deployment. We run several apps, both small and large and they all react
> the same way... If we stop then undeploy any of these Tapestry apps, and
> afterwards do a hot deploy with a newer version (without the restart),
> the CPU usage shoots to 100% and stays there. JSP based apps pure
> Servlets don't appear to react in the same way.
>
> Has anybody experienced something similar?
>
> Kind regards,
> Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: 100% CPU load after hot deploy of Tapestry 5 Application on
Tomcat 6
Posted by Peter Stavrinides <P....@albourne.com>.
Hi Guys,
Thanks for the attempts...
these machines are monsters, 8 cores and about 32 gigs... Tomcat has more resources allocated than it should ever need, the CPU never drops just grows after the hot deploy (and only then). The machine doesn't fall over either because of the extensive resources. As I have said this only happens with a hot deploy, so not sure what role GC could play in that.
Thanks,
Peter
----- Original Message -----
From: "Kalle Korhonen" <ka...@gmail.com>
To: "Tapestry users" <us...@tapestry.apache.org>
Sent: Thursday, 21 July, 2011 07:53:56 GMT +02:00 Athens, Beirut, Bucharest, Istanbul
Subject: Re: 100% CPU load after hot deploy of Tapestry 5 Application on Tomcat 6
As Mark said, GC exhausting the system is a possibility if it
eventually recovers from the 100% CPU. If at all possible, try with
higher max memory allocation and Tomcat 7.
Kalle
On Wed, Jul 20, 2011 at 2:16 AM, <P....@albourne.com> wrote:
> Hi All,
>
> This is a tricky question one since I do realize there are several possibilities to explain the problem, not all of which are Tapestry related. We have battled to understand what is going on in production for the past year or so, and have tried to pinpoint where the problem lies and come up short. If anyone has experienced anything similar *please share.
>
> Our environment is:
>
> Ubuntu Server
> Tapestry 5.1
> JDK 6
> Tomcat 6
> Apache in front (reverse proxy)
>
> We are deploying our Tapestry apps to Tomcat 6, and they do run great even with load, and it remains very stable as the session count increases; this is when a server restart is done after a deployment.
>
> The problem appears to be specific to 'Tapestry applications' and hot deployment. We run several apps, both small and large and they all react the same way... If we stop then undeploy any of these Tapestry apps, and afterwards do a hot deploy with a newer version (without the restart), the CPU usage shoots to 100% and stays there. JSP based apps pure Servlets don't appear to react in the same way.
>
> Has anybody experienced something similar?
>
> Kind regards,
> Peter
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> For additional commands, e-mail: users-help@tapestry.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: 100% CPU load after hot deploy of Tapestry 5 Application on
Tomcat 6
Posted by Kalle Korhonen <ka...@gmail.com>.
As Mark said, GC exhausting the system is a possibility if it
eventually recovers from the 100% CPU. If at all possible, try with
higher max memory allocation and Tomcat 7.
Kalle
On Wed, Jul 20, 2011 at 2:16 AM, <P....@albourne.com> wrote:
> Hi All,
>
> This is a tricky question one since I do realize there are several possibilities to explain the problem, not all of which are Tapestry related. We have battled to understand what is going on in production for the past year or so, and have tried to pinpoint where the problem lies and come up short. If anyone has experienced anything similar *please share.
>
> Our environment is:
>
> Ubuntu Server
> Tapestry 5.1
> JDK 6
> Tomcat 6
> Apache in front (reverse proxy)
>
> We are deploying our Tapestry apps to Tomcat 6, and they do run great even with load, and it remains very stable as the session count increases; this is when a server restart is done after a deployment.
>
> The problem appears to be specific to 'Tapestry applications' and hot deployment. We run several apps, both small and large and they all react the same way... If we stop then undeploy any of these Tapestry apps, and afterwards do a hot deploy with a newer version (without the restart), the CPU usage shoots to 100% and stays there. JSP based apps pure Servlets don't appear to react in the same way.
>
> Has anybody experienced something similar?
>
> Kind regards,
> Peter
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> For additional commands, e-mail: users-help@tapestry.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: 100% CPU load after hot deploy of Tapestry 5 Application on
Tomcat 6
Posted by Mark <ma...@xeric.net>.
I've run into a few problems when using maven/cargo to deploy, but
they appear to be issues with simply running out of memory as the
deployment process seems to use a lot of memory relative to my app. In
a few cases this ended up pegging the processor, but normally it would
just lock up Tomcat.
Mark
On Wed, Jul 20, 2011 at 4:16 AM, <P....@albourne.com> wrote:
> Hi All,
>
> This is a tricky question one since I do realize there are several possibilities to explain the problem, not all of which are Tapestry related. We have battled to understand what is going on in production for the past year or so, and have tried to pinpoint where the problem lies and come up short. If anyone has experienced anything similar *please share.
>
> Our environment is:
>
> Ubuntu Server
> Tapestry 5.1
> JDK 6
> Tomcat 6
> Apache in front (reverse proxy)
>
> We are deploying our Tapestry apps to Tomcat 6, and they do run great even with load, and it remains very stable as the session count increases; this is when a server restart is done after a deployment.
>
> The problem appears to be specific to 'Tapestry applications' and hot deployment. We run several apps, both small and large and they all react the same way... If we stop then undeploy any of these Tapestry apps, and afterwards do a hot deploy with a newer version (without the restart), the CPU usage shoots to 100% and stays there. JSP based apps pure Servlets don't appear to react in the same way.
>
> Has anybody experienced something similar?
>
> Kind regards,
> Peter
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> For additional commands, e-mail: users-help@tapestry.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org