You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by sasidhar prabhakar <sa...@gmail.com> on 2010/08/17 11:48:22 UTC

Tomcat hangs every few hours

Tomcat hangs ever few hours.
In server.xml maxThreads are 300.

In thread dump I observed every thread doing the same thing.
In my code I sends sms to users by using HttpURLConnection.
I am using this code for months I didn't get the problem earlier.
For few days I am getting this problem consistently. Whenever tomcat hangs I
takes the thread dump,
every time it shows the same results.

Here is my one of threads dump. this is common for all remaining threads.

Somebody guide me to solve this problem. I am trying but unable to do this

"http-80-300" daemon prio=1 tid=0x76890ef0 nid=0x6bb6 runnable
[0x6def0000..0x6def0db0]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    - locked <0xb1332658> (a java.io.BufferedInputStream)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:681)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:626)
    at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:983)
    - locked <0xb1332690> (a sun.net.www.protocol.http.HttpURLConnection)
    at SendSms.sendToSingleRecipientNoNpri(SendSms.java:89)
    at CreateAccountAction.execute(CreateAccountAction.java:86)
    at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
    at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
    at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
    at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.amulyam.servlet.RedirectFilter.doFilter(RedirectFilter.java:56)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
    at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
    at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
    at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
    at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
    at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
    at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
    at java.lang.Thread.run(Thread.java:595)

RE: [OT] Re: Tomcat hangs every few hours

Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: sasidhar prabhakar [mailto:sasidhar1229@gmail.com]
> Subject: Re: [OT] Re: Tomcat hangs every few hours
> 
> I don't know what wireshark is

www.wireshark.org

Equivalents are available for whatever platform you're running on.

> If my webapp is not handling the responses what can I do.

Fix your webapp.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: [OT] Re: Tomcat hangs every few hours

Posted by sasidhar prabhakar <sa...@gmail.com>.
I don't know what wireshark is can we install it in production servers. If
my webapp is not handling the responses what can I do.
Could you please suggest that.

On Tue, Aug 17, 2010 at 9:22 PM, Caldarale, Charles R <
Chuck.Caldarale@unisys.com> wrote:

> > From: sasidhar prabhakar [mailto:sasidhar1229@gmail.com]
> > Subject: Re: [OT] Re: Tomcat hangs every few hours
> >
> > For months it is working properly.
>
> Then something must have changed, either on the Tomcat system, or the
> server your webapp is trying to communicate with.  Try to find out what was
> altered.
>
> > Is there any other reasons.
>
> Possible that the code in your webapp is not handling the responses it's
> getting from the mail server.  Get Wireshark traces of the traffic between
> your webapp and the mail server and compare that with what the webapp is
> expecting.
>
>  - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all computers.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

RE: [OT] Re: Tomcat hangs every few hours

Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: sasidhar prabhakar [mailto:sasidhar1229@gmail.com]
> Subject: Re: [OT] Re: Tomcat hangs every few hours
> 
> For months it is working properly.

Then something must have changed, either on the Tomcat system, or the server your webapp is trying to communicate with.  Try to find out what was altered.

> Is there any other reasons.

Possible that the code in your webapp is not handling the responses it's getting from the mail server.  Get Wireshark traces of the traffic between your webapp and the mail server and compare that with what the webapp is expecting.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: [OT] Re: Tomcat hangs every few hours

Posted by sasidhar prabhakar <sa...@gmail.com>.
For months it is working properly. I tried that one also. Immediately after
hangs I accessed the url directly several times it is working.
Is that the only reason, I thought that but when accessed it is working. Is
there any other reasons.

On Tue, Aug 17, 2010 at 5:33 PM, Peter Crowther <peter.crowther@melandra.com
> wrote:

> On 17 August 2010 10:48, sasidhar prabhakar <sa...@gmail.com>
> wrote:
>
> > In thread dump I observed every thread doing the same thing.
> > In my code I sends sms to users by using HttpURLConnection.
> > I am using this code for months I didn't get the problem earlier.
> > For few days I am getting this problem consistently. Whenever tomcat
> hangs
> > I
> > takes the thread dump,
> > every time it shows the same results.
> >
> > Here is my one of threads dump. this is common for all remaining threads.
> >
> > Somebody guide me to solve this problem. I am trying but unable to do
> this
> >
> > "http-80-300" daemon prio=1 tid=0x76890ef0 nid=0x6bb6 runnable
> > [0x6def0000..0x6def0db0]
> >    at java.net.SocketInputStream.socketRead0(Native Method)
> >    at java.net.SocketInputStream.read(SocketInputStream.java:129)
> > [...]
>
>
>
> >   at SendSms.sendToSingleRecipientNoNpri(SendSms.java:89)
> >
> [...]
>
> The socket is hanging trying to read the response.  Your SMS provider
> probably isn't responding to you on occasion.  I assume you've written test
> code to do a SMS send; if not, write it and test the SMS send outside of
> Tomcat.
>
> This is not a Tomcat issue, so I've marked it [OT].
>
> - Peter
>

[OT] Re: Tomcat hangs every few hours

Posted by Peter Crowther <pe...@melandra.com>.
On 17 August 2010 10:48, sasidhar prabhakar <sa...@gmail.com> wrote:

> In thread dump I observed every thread doing the same thing.
> In my code I sends sms to users by using HttpURLConnection.
> I am using this code for months I didn't get the problem earlier.
> For few days I am getting this problem consistently. Whenever tomcat hangs
> I
> takes the thread dump,
> every time it shows the same results.
>
> Here is my one of threads dump. this is common for all remaining threads.
>
> Somebody guide me to solve this problem. I am trying but unable to do this
>
> "http-80-300" daemon prio=1 tid=0x76890ef0 nid=0x6bb6 runnable
> [0x6def0000..0x6def0db0]
>    at java.net.SocketInputStream.socketRead0(Native Method)
>    at java.net.SocketInputStream.read(SocketInputStream.java:129)
> [...]



>   at SendSms.sendToSingleRecipientNoNpri(SendSms.java:89)
>
[...]

The socket is hanging trying to read the response.  Your SMS provider
probably isn't responding to you on occasion.  I assume you've written test
code to do a SMS send; if not, write it and test the SMS send outside of
Tomcat.

This is not a Tomcat issue, so I've marked it [OT].

- Peter

Re: Tomcat hangs every few hours

Posted by Brett Delle Grazie <br...@intact-is.com>.
Hi,

One obvious possibility is that your SMS gateway (or your connection to
it) is down - thus all threads are being 'blocked' waiting for a
response from the gateway. This is supported by the fact that the
threads are all blcoked in the socketRead0 call.

Can I suggest, if you haven't already, to ensure you set a connect
timeout and a 'wait' timeout (i.e. only wait x ms or secs before
terminating the request and signalling a failure).

I'm not sure what the default socket connect and read/write timeouts in
Java are (they may be forever) but if they default to the OS level
timeouts then they can be as much as 15 minutes (5 minutes + 3x retry).

Hope this helps

On Tue, 2010-08-17 at 15:18 +0530, sasidhar prabhakar wrote:
> Tomcat hangs ever few hours.
> In server.xml maxThreads are 300.
> 
> In thread dump I observed every thread doing the same thing.
> In my code I sends sms to users by using HttpURLConnection.
> I am using this code for months I didn't get the problem earlier.
> For few days I am getting this problem consistently. Whenever tomcat hangs I
> takes the thread dump,
> every time it shows the same results.
> 
> Here is my one of threads dump. this is common for all remaining threads.
> 
> Somebody guide me to solve this problem. I am trying but unable to do this
> 
> "http-80-300" daemon prio=1 tid=0x76890ef0 nid=0x6bb6 runnable
> [0x6def0000..0x6def0db0]
>     at java.net.SocketInputStream.socketRead0(Native Method)
>     at java.net.SocketInputStream.read(SocketInputStream.java:129)
>     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>     at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>     at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
>     - locked <0xb1332658> (a java.io.BufferedInputStream)
>     at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:681)
>     at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:626)
>     at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:983)
>     - locked <0xb1332690> (a sun.net.www.protocol.http.HttpURLConnection)
>     at SendSms.sendToSingleRecipientNoNpri(SendSms.java:89)
>     at CreateAccountAction.execute(CreateAccountAction.java:86)
>     at
> org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
>     at
> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
>     at
> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
>     at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>     at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>     at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>     at com.amulyam.servlet.RedirectFilter.doFilter(RedirectFilter.java:56)
>     at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>     at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>     at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>     at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>     at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
>     at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>     at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>     at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>     at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>     at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
>     at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>     at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
>     at java.lang.Thread.run(Thread.java:595)

-- 
Best Regards,

Brett Delle Grazie

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat hangs every few hours

Posted by sasidhar prabhakar <sa...@gmail.com>.
Thanks I will follow that

On Tue, Aug 17, 2010 at 6:12 PM, Pid <pi...@pidster.com> wrote:

> On 17/08/2010 10:48, sasidhar prabhakar wrote:
> > Tomcat hangs ever few hours.
>
> *Exact* Tomcat, JVM and OS versions?
>
> > In server.xml maxThreads are 300.
> >
> > In thread dump I observed every thread doing the same thing.
> > In my code I sends sms to users by using HttpURLConnection.
> > I am using this code for months I didn't get the problem earlier.
> > For few days I am getting this problem consistently. Whenever tomcat
> hangs I
> > takes the thread dump,
> > every time it shows the same results.
> >
> > Here is my one of threads dump. this is common for all remaining threads.
> >
> > Somebody guide me to solve this problem. I am trying but unable to do
> this
>
> If each request processor thread is dependent on an inline request to an
> external resource, and that resource is unavailable or otherwise slow to
> respond then your app will inevitably run out of threads.
>
> You should consider offloading the request to a thread pool (see the
> Java Concurrency API in recent JVMs) with appropriate timeouts.
>
>
> p
>
> > "http-80-300" daemon prio=1 tid=0x76890ef0 nid=0x6bb6 runnable
> > [0x6def0000..0x6def0db0]
> >     at java.net.SocketInputStream.socketRead0(Native Method)
> >     at java.net.SocketInputStream.read(SocketInputStream.java:129)
> >     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> >     at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> >     at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> >     - locked <0xb1332658> (a java.io.BufferedInputStream)
> >     at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:681)
> >     at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:626)
> >     at
> >
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:983)
> >     - locked <0xb1332690> (a sun.net.www.protocol.http.HttpURLConnection)
> >     at SendSms.sendToSingleRecipientNoNpri(SendSms.java:89)
> >     at CreateAccountAction.execute(CreateAccountAction.java:86)
> >     at
> >
> org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
> >     at
> >
> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
> >     at
> > org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
> >     at
> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
> >     at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
> >     at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> >     at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> >     at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> >     at
> com.amulyam.servlet.RedirectFilter.doFilter(RedirectFilter.java:56)
> >     at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> >     at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> >     at
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> >     at
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> >     at
> >
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
> >     at
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
> >     at
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> >     at
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> >     at
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
> >     at
> >
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
> >     at
> >
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
> >     at
> > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
> >     at java.lang.Thread.run(Thread.java:595)
> >
>
>

Re: Tomcat hangs every few hours

Posted by Pid <pi...@pidster.com>.
On 17/08/2010 10:48, sasidhar prabhakar wrote:
> Tomcat hangs ever few hours.

*Exact* Tomcat, JVM and OS versions?

> In server.xml maxThreads are 300.
> 
> In thread dump I observed every thread doing the same thing.
> In my code I sends sms to users by using HttpURLConnection.
> I am using this code for months I didn't get the problem earlier.
> For few days I am getting this problem consistently. Whenever tomcat hangs I
> takes the thread dump,
> every time it shows the same results.
> 
> Here is my one of threads dump. this is common for all remaining threads.
> 
> Somebody guide me to solve this problem. I am trying but unable to do this

If each request processor thread is dependent on an inline request to an
external resource, and that resource is unavailable or otherwise slow to
respond then your app will inevitably run out of threads.

You should consider offloading the request to a thread pool (see the
Java Concurrency API in recent JVMs) with appropriate timeouts.


p

> "http-80-300" daemon prio=1 tid=0x76890ef0 nid=0x6bb6 runnable
> [0x6def0000..0x6def0db0]
>     at java.net.SocketInputStream.socketRead0(Native Method)
>     at java.net.SocketInputStream.read(SocketInputStream.java:129)
>     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>     at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>     at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
>     - locked <0xb1332658> (a java.io.BufferedInputStream)
>     at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:681)
>     at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:626)
>     at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:983)
>     - locked <0xb1332690> (a sun.net.www.protocol.http.HttpURLConnection)
>     at SendSms.sendToSingleRecipientNoNpri(SendSms.java:89)
>     at CreateAccountAction.execute(CreateAccountAction.java:86)
>     at
> org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
>     at
> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
>     at
> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
>     at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>     at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>     at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>     at com.amulyam.servlet.RedirectFilter.doFilter(RedirectFilter.java:56)
>     at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>     at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>     at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>     at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>     at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
>     at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>     at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>     at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>     at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>     at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
>     at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>     at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
>     at java.lang.Thread.run(Thread.java:595)
>