You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-dev@axis.apache.org by jay25 <js...@yahoo.com> on 2009/05/01 21:22:23 UTC
Re: [jira] Commented: (AXIS2-3670) Problem using REUSE_HTTP_CLIENT
- Application stalls
Thanks a lot for taking time during your vacation for this issue. Really
under lot of pressure to get this going. Yes, I could see that the generated
stub has
_messageContext.getTransportOut().getSender().cleanup(_messageContext); just
before send response.
I made little more progress on this issue.
My settings: Java 1.5, Axis2 1.3, default values for axis settings - number
of connections to host, timeout.
This is what is happening:
default timeout for connection is getting set to 60000 and socket timeout to
30000. When the server takes more time to respond (I simulated this by
adding wait(70000) in service code ), axis fault exception is thrown: Also,
thread dump shows thread in stuck state; netstat shows some connections
lingering in close_wait for about 10 minutes. Looks like after the timeout
the connections or threads are not getting released to the connection pool.
So subsequent requests keep waiting.
"xml version='1.0' encoding='UTF-8'?><soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><soapenv:Fault><faultcode>soapenv:Client</faultcode><faultstring>current
thread not owner</faultstring><detail><Exception>org.apache.axis2.AxisFault:
current thread not owner
[\n]">
"[0x9]at
org.apache.axis2.AxisFault.makeFault(AxisFault.java:381)
[\n]">
"[0x9]at
com.rsa.pso.aa.services.AccountServiceMessageReceiverInOut.invokeBusinessLogic(AccountServiceMessageReceiverInOut.java:101)
[\n]">
"[0x9]at
org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.receive(AbstractInOutSyncMessageReceiver.java:39)
[\n]">
"[0x9]at
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:144)
[\n]">
"[0x9]at
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:279)
[\n]">
"[0x9]at
org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:116)
[\n]">
"[0x9]at
javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
[\n]">
"[0x9]at
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
[\n]">
"[0x9]at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
[\n]">
"[0x9]at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
[\n]">
"[0x9]at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
[\n]">
"[0x9]at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
[\n]">
"[0x9]at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
[\n]">
"[0x9]at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
[\n]">
"[0x9]at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:548)
[\n]">
"[0x9]at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
[\n]">
"[0x9]at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
[\n]">
"[0x9]at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
[\n]">
"[0x9]at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
[\n]">
"[0x9]at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
[\n]">
"[0x9]at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
[\n]">
"[0x9]at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
[\n]">
"[0x9]at java.lang.Thread.run(Unknown Source)
[\n]">
"Caused by: java.lang.IllegalMonitorStateException: current thread not
owner
[\n]">
"[0x9]at java.lang.Object.wait(Native Method)
[\n]">
"[0x9]at com.xx.HelloWorld.hello(HelloWorld.java:49)
[\n]">
"[0x9]at
com.xx.HelloWorld.HelloWorldMessageReceiverInOut.invokeBusinessLogic(HelloWorldMessageReceiverInOut.java:48)
[\n]">
"[0x9]... 21 more
[\n]">
"</Exception></detail></soapenv:Fault></soapenv:Body></soapenv:Envelope>">
Hi Jayashree,
you can check if:
_messageContext.getTransportOut().getSender().cleanup(_messageContext);
is really invoked. AFAI can remember there is an issue when an exception is
thrown into the stub. If this is the case the
_messageContext.getTransportOut().getSender().cleanup(_messageContext); is
not invoked.
Sorry for not being really responsive, but I am on a vacatio and will be
back on 7th of May.
hope this helps
Regards,
Dobri
On Wed, Apr 29, 2009 at 11:49 PM, jay25 <js...@yahoo.com> wrote:
>
> I am facing the same issue in production right now. The axis stub already
> has cleanTransport --
> _messageContext.getTransportOut().getSender().cleanup(_messageContext);
>
> We have set reuse_option to true. Rest of values are set to default..
> Greatly appreciate help...
>
> The thread dump shows threads waiting like this:
> at java.lang.Object.wait(Native Method)
> at
>
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:509)
> at
>
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:394)
> at
>
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:152)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
> at
>
> org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
> at
> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
> at
>
> org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:305)
> at
>
> org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:201)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:452)
> at
>
> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
> at
>
> org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
> at com.xxx.HelloStub.hello1(HelloStub.java:173)
>
--
View this message in context: http://www.nabble.com/-jira--Created%3A-%28AXIS2-3670%29-Probelm-using-REUSE_HTTP_CLIENT-tp16331279p23338385.html
Sent from the Axis - Dev mailing list archive at Nabble.com.
Re: [jira] Commented: (AXIS2-3670) Problem using REUSE_HTTP_CLIENT -
Application stalls
Posted by Dobri Kitipov <kd...@googlemail.com>.
Hi,
just for the test you can modify the stub and move the
_messageContext.
>
> getTransportOut().getSender().cleanup(_messageContext);
into a finally block. So with or without exceptions the connection close
should be triggered.
Regards,
Dobri
On Fri, May 1, 2009 at 10:22 PM, jay25 <js...@yahoo.com> wrote:
>
> Thanks a lot for taking time during your vacation for this issue. Really
> under lot of pressure to get this going. Yes, I could see that the
> generated
> stub has
> _messageContext.getTransportOut().getSender().cleanup(_messageContext);
> just
> before send response.
>
> I made little more progress on this issue.
> My settings: Java 1.5, Axis2 1.3, default values for axis settings - number
> of connections to host, timeout.
> This is what is happening:
> default timeout for connection is getting set to 60000 and socket timeout
> to
> 30000. When the server takes more time to respond (I simulated this by
> adding wait(70000) in service code ), axis fault exception is thrown: Also,
> thread dump shows thread in stuck state; netstat shows some connections
> lingering in close_wait for about 10 minutes. Looks like after the timeout
> the connections or threads are not getting released to the connection pool.
> So subsequent requests keep waiting.
>
> "xml version='1.0' encoding='UTF-8'?><soapenv:Envelope
> xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/
> "><soapenv:Body><soapenv:Fault><faultcode>soapenv:Client</faultcode><faultstring>current
> thread not
> owner</faultstring><detail><Exception>org.apache.axis2.AxisFault:
> current thread not owner
> [\n]">
> "[0x9]at
> org.apache.axis2.AxisFault.makeFault(AxisFault.java:381)
> [\n]">
> "[0x9]at
>
> com.rsa.pso.aa.services.AccountServiceMessageReceiverInOut.invokeBusinessLogic(AccountServiceMessageReceiverInOut.java:101)
> [\n]">
> "[0x9]at
>
> org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.receive(AbstractInOutSyncMessageReceiver.java:39)
> [\n]">
> "[0x9]at
> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:144)
> [\n]">
> "[0x9]at
>
> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:279)
> [\n]">
> "[0x9]at
> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:116)
> [\n]">
> "[0x9]at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> [\n]">
> "[0x9]at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> [\n]">
> "[0x9]at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
> [\n]">
> "[0x9]at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> [\n]">
> "[0x9]at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> [\n]">
> "[0x9]at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
> [\n]">
> "[0x9]at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> [\n]">
> "[0x9]at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> [\n]">
> "[0x9]at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:548)
> [\n]">
> "[0x9]at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
> [\n]">
> "[0x9]at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
> [\n]">
> "[0x9]at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
> [\n]">
> "[0x9]at
>
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
> [\n]">
> "[0x9]at
>
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
> [\n]">
> "[0x9]at
>
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
> [\n]">
> "[0x9]at
>
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> [\n]">
> "[0x9]at java.lang.Thread.run(Unknown Source)
> [\n]">
> "Caused by: java.lang.IllegalMonitorStateException: current thread not
> owner
> [\n]">
> "[0x9]at java.lang.Object.wait(Native Method)
> [\n]">
> "[0x9]at com.xx.HelloWorld.hello(HelloWorld.java:49)
> [\n]">
> "[0x9]at
>
> com.xx.HelloWorld.HelloWorldMessageReceiverInOut.invokeBusinessLogic(HelloWorldMessageReceiverInOut.java:48)
> [\n]">
> "[0x9]... 21 more
> [\n]">
> "</Exception></detail></soapenv:Fault></soapenv:Body></soapenv:Envelope>">
>
>
>
> Hi Jayashree,
> you can check if:
>
> _messageContext.getTransportOut().getSender().cleanup(_messageContext);
>
> is really invoked. AFAI can remember there is an issue when an exception is
> thrown into the stub. If this is the case the
> _messageContext.getTransportOut().getSender().cleanup(_messageContext); is
> not invoked.
>
> Sorry for not being really responsive, but I am on a vacatio and will be
> back on 7th of May.
>
> hope this helps
>
> Regards,
> Dobri
>
>
> On Wed, Apr 29, 2009 at 11:49 PM, jay25 <js...@yahoo.com> wrote:
>
> >
> > I am facing the same issue in production right now. The axis stub
> already
> > has cleanTransport --
> > _messageContext.getTransportOut().getSender().cleanup(_messageContext);
> >
> > We have set reuse_option to true. Rest of values are set to default..
> > Greatly appreciate help...
> >
> > The thread dump shows threads waiting like this:
> > at java.lang.Object.wait(Native Method)
> > at
> >
> >
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:509)
> > at
> >
> >
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:394)
> > at
> >
> >
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:152)
> > at
> >
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
> > at
> >
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
> > at
> >
> >
> org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
> > at
> >
> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
> > at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
> > at
> >
> >
> org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:305)
> > at
> >
> >
> org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:201)
> > at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:452)
> > at
> >
> >
> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
> > at
> >
> >
> org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
> > at com.xxx.HelloStub.hello1(HelloStub.java:173)
> >
>
> --
> View this message in context:
> http://www.nabble.com/-jira--Created%3A-%28AXIS2-3670%29-Probelm-using-REUSE_HTTP_CLIENT-tp16331279p23338385.html
> Sent from the Axis - Dev mailing list archive at Nabble.com.
>
>