You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Michael Südkamp <Mi...@docware.de> on 2011/07/20 10:15:22 UTC
server status reports long request times
Hello,
>From time to time we have customers experiencing Tomcat going to 100% CPU. One customer sent us the output from the server status page from the Tomcat 6.0.32 Manager web-app. This page reports a lot of HTTP connector requests belonging to a certain client with stage value "S" and "time" values between 1000000 and 3000000 ms, i.e. almost one hour. We don't know what causes the requests to never finish. Believe me our web-app is not in a loop. Maybe it's a problem with the database but we have already often spend a lot of time to analyze but couldn't find a problem on our side. Anyway, isn't Tomcat somehow capable to detect such hanging requests (say requests that take longer than a minute) and kill them? I studied the HTTP connector reference but haven't found a parameter related to this. The system runs on a Windows Server 2008 R2 and Java 5 (OK we could ask the customer to upgrade Java).
Best regards
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: server status reports long request times
Posted by Michael Südkamp <Mi...@docware.de>.
> That sounds like you looked at ps or similar output, not a thread dump. Take an actual JVM thread dump and see where the worker threads are during the busy periods.
Wrong, to identify the thread in question I looked with Sysinternals Process Explorer which thread ID caused the 100% CPU. Then I looked up this thread ID in the Java thread dump. And this was the thread named "VM Thread".
> Turn on GC logging to see if you are experiencing heap churn and heavy GC activity.
Thank you for the hint.
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: server status reports long request times
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Michael Südkamp [mailto:Michael.Suedkamp@docware.de]
> Subject: AW: server status reports long request times
> We looked at thread dumps several times. The CPU charge was always
> reported for the "VM Thread" and we didn't know how this information
> could help us finding a problem on our side.
That sounds like you looked at ps or similar output, not a thread dump. Take an actual JVM thread dump and see where the worker threads are during the busy periods.
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
Turn on GC logging to see if you are experiencing heap churn and heavy GC activity.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: server status reports long request times
Posted by Michael Südkamp <Mi...@docware.de>.
-----Ursprüngliche Nachricht-----
Von: André Warnier [mailto:aw@ice-sa.com]
Gesendet: Mittwoch, 20. Juli 2011 11:06
An: Tomcat Users List
Betreff: Re: server status reports long request times
Hello Andre
"a certain client" related to the entries in the "client" column of the mentioned server status page having the same IP address
>> Believe me our web-app is not in a loop.
> How do you know that ? We do not believe anything or anyone here without some proof.
Sure but how could I proove this ?
> I believe that taking some thread dumps would tell you exactly what Tomcat is doing at such times.
We looked at thread dumps several times. The CPU charge was always reported for the "VM Thread" and we didn't know how this information could help us finding a problem on our side. If the problem occurs on an internal system we would probably connect with a debugger to this Tomcat to find evidence for extensive GC activity relating to our classes or the like. But on production customer system it's not easy to further analyze the situation.
> Well, if from the Tomcat point of view the request is not finished, it should not kill it, should it ? some valid requests/responses may take a long time to finish. How is Tomcat supposed to know ?
Sure, Tomcat can't know but we would like to tell Tomcat to do so .
> A very long KeepAlive setting may explain connections lingering on. What is your setting ?
The default or the value from the default server.xml
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: server status reports long request times
Posted by André Warnier <aw...@ice-sa.com>.
Michael Südkamp wrote:
> Hello,
>
>>From time to time we have customers experiencing Tomcat going to 100% CPU.
One customer sent us the output from the server status page from the Tomcat 6.0.32 Manager
web-app.
This page reports a lot of HTTP connector requests belonging to a certain client
can you explain "a certain client" ? is it a browser, or some other app ? is it one single
client ? from the point of view of that client, is the request finished, or is the client
still waiting for something ?
with stage value "S" and "time" values between 1000000 and 3000000 ms, i.e. almost one
hour.
We don't know what causes the requests to never finish.
The question is : is the request/response really finished ?
Believe me our web-app is not in a loop.
How do you know that ? We do not believe anything or anyone here without some proof.
Maybe it's a problem with the database but we have already often spend a lot of time to
analyze
but couldn't find a problem on our side.
I believe that taking some thread dumps would tell you exactly what Tomcat is doing at
such times.
Anyway, isn't Tomcat somehow capable to detect such hanging requests (say requests that take
longer than a minute) and kill them?
Well, if from the Tomcat point of view the request is not finished, it should not kill it,
should it ? some valid requests/responses may take a long time to finish. How is Tomcat
supposed to know ?
I studied the HTTP connector reference but haven't found a parameter related to this.
A very long KeepAlive setting may explain connections lingering on. What is your setting ?
The system runs on a Windows Server 2008 R2 and Java 5 (OK we could ask the customer to
upgrade Java).
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: server status reports long request times
Posted by Mark Thomas <ma...@apache.org>.
On 20/07/2011 09:15, Michael Südkamp wrote:
> Hello,
>
> From time to time we have customers experiencing Tomcat going to
> 100% CPU. One customer sent us the output from the server status
> page from the Tomcat 6.0.32 Manager web-app. This page reports a lot
> of HTTP connector requests belonging to a certain client with stage
> value "S" and "time" values between 1000000 and 3000000 ms, i.e.
> almost one hour. We don't know what causes the requests to never
> finish. Believe me our web-app is not in a loop. Maybe it's a problem
> with the database but we have already often spend a lot of time to
> analyze but couldn't find a problem on our side.
Thread dumps.
> Anyway, isn't
> Tomcat somehow capable to detect such hanging requests (say requests
> that take longer than a minute) and kill them?
Detecting them, yes. Killing them, no. This is a Java limitation.
> I studied the HTTP
> connector reference but haven't found a parameter related to this.
> The system runs on a Windows Server 2008 R2 and Java 5 (OK we could
> ask the customer to upgrade Java).
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org