You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by bradtwurst <na...@xoxy.net> on 2007/04/23 17:19:02 UTC
Re: http lock up
gnodet wrote:
>
> Have you tried a thread dump to see if all the threads were waiting on
> a particular resource ?
>
>
I have been encountering another lock up this morning. It appears that alot
of the threads are waiting for a response from a sendSync message that is
going out to a http endpoint (external webservice)
Here is a example stack
"pool-flow.seda.Esb-thread-1020" prio=6 tid=0x2b4a7430 nid=0x3a0 in
Object.wait() [0x2fb9f000..0x2fb9fb1c]
at java.lang.Object.wait(Native Method)
- waiting on <0x261b2440> (a
org.apache.servicemix.jbi.messaging.InOptionalOutImpl)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.waitForExchange(DeliveryChannelImpl.java:679)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.sendSync(DeliveryChannelImpl.java:458)
- locked <0x261b2440> (a
org.apache.servicemix.jbi.messaging.InOptionalOutImpl)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.sendSync(DeliveryChannelImpl.java:428)
at
org.apache.servicemix.components.util.PojoSupport.sendSync(PojoSupport.java:201)
at esb.SheetMgmtServiceESB.sendStringMessage(SheetMgmtServiceESB.java:1824)
at esb.SheetMgmtServiceESB.outOfUv(SheetMgmtServiceESB.java:1012)
at
sheetmgmt.esb.SheetMgmtServiceESB.onMessageExchange(SheetMgmtServiceESB.java:445)
at
org.apache.servicemix.components.util.ComponentAdaptorMEListener.onMessageExchange(ComponentAdaptorMEListener.java:47)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:593)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:174)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:176)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
at java.lang.Thread.run(Thread.java:595)
Any suggestions of where to start looking for problems?
Would I make the situation better by moving from a seda queue to a jms type
flow? (I don't think that it would but I want to verify)
Is there any sort of timeout for a message (http bc) that would fail the
message?
I've not done any special configuration for the http bc endpoint. As such,
what are the default number of threads that I can expect for outgoing calls.
Would a blocking of threads caused by outgoing calls have an effect on
incoming calls?
Thanks,
James
--
View this message in context: http://www.nabble.com/http-lock-up-tf3488493s12049.html#a10142439
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: http lock up
Posted by bradtwurst <na...@xoxy.net>.
As a follow-up, a stack trace for a blocked http resource is
"pool-flow.seda.servicemix-http-thread-1342" prio=6 tid=0x2a73ac10
nid=0x1898 in Object.wait() [0x2fe8f000..0x2fe8fc1c]
at java.lang.Object.wait(Native Method)
- waiting on <0x07c7ac18> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:509)
- locked <0x07c7ac18> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:394)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:152)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at
org.apache.servicemix.http.processors.ProviderProcessor.process(ProviderProcessor.java:168)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:489)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:441)
at
org.apache.servicemix.common.BaseLifeCycle.onMessageExchange(BaseLifeCycle.java:46)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:593)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:174)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:176)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
at java.lang.Thread.run(Thread.java:595)
Any suggestions?
Thanks,
James
--
View this message in context: http://www.nabble.com/http-lock-up-tf3488493s12049.html#a10144482
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: http lock up
Posted by Guillaume Nodet <gn...@gmail.com>.
Yeah, you need to send back the exchange with a DONE status.
In a lw component, just call
done(exchange)
On 4/24/07, bradtwurst <na...@xoxy.net> wrote:
>
>
> Guillaume (and others),
>
> Thanks for the reply.
>
> I've stepped through the http provider and I'm not sure exactly what the
> best practice is.
>
> I am sending the message to the http provider from a lw pojo SE. The
> message type is InOptionalOut, and it is being transferred via seda
> (sendsync).
>
> In the http provider, the code calls the provider web service and it
> returns
> with a response. Since there is an actual response, the system sends the
> responseBodyAsStream back to the sender. Then, the code explicitly states
> to not close the connection (as I am not in a transaction)
>
> Then, the control returns back to the original SE and the process
> continues.
> However, the connection is not closed.
>
> Currently, I am not doing anything after the message returns - just
> returning out of the lw se.
>
>
> My questions:
>
> What causes the connection to close in this scenario? Do I need to do
> something with the returned message that will result in the connection
> closing?
>
> Any other suggestions?
>
> Thanks,
> James
> --
> View this message in context:
> http://www.nabble.com/http-lock-up-tf3488493s12049.html#a10160639
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
------------------------
Principal Engineer, IONA
Blog: http://gnodet.blogspot.com/
Re: http lock up
Posted by bradtwurst <na...@xoxy.net>.
Guillaume (and others),
Thanks for the reply.
I've stepped through the http provider and I'm not sure exactly what the
best practice is.
I am sending the message to the http provider from a lw pojo SE. The
message type is InOptionalOut, and it is being transferred via seda
(sendsync).
In the http provider, the code calls the provider web service and it returns
with a response. Since there is an actual response, the system sends the
responseBodyAsStream back to the sender. Then, the code explicitly states
to not close the connection (as I am not in a transaction)
Then, the control returns back to the original SE and the process continues.
However, the connection is not closed.
Currently, I am not doing anything after the message returns - just
returning out of the lw se.
My questions:
What causes the connection to close in this scenario? Do I need to do
something with the returned message that will result in the connection
closing?
Any other suggestions?
Thanks,
James
--
View this message in context: http://www.nabble.com/http-lock-up-tf3488493s12049.html#a10160639
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: http lock up
Posted by Guillaume Nodet <gn...@gmail.com>.
Yeah, there may be a resource problem between the number of outgoing calls
that need
to be services and the number of connections available (connections are
pooled).
You should configure the servicemix-http component (using JMX at runtime
after installation
or by editing the conf/components.properties file before installation) and
change the number of
connections available to be much higher than the number of threads.
The number of threads is configured in the conf/servicemix.xml config file
(default values are accessible from conf/servicemix.properties)
See http://incubator.apache.org/servicemix/thread-pools.html
Just ensure that
maxConnectionsPerHost >> servicemix.maximumPoolSize
and it should help.
On 4/23/07, bradtwurst <na...@xoxy.net> wrote:
>
>
>
> gnodet wrote:
> >
> > Have you tried a thread dump to see if all the threads were waiting on
> > a particular resource ?
> >
> >
>
>
> I have been encountering another lock up this morning. It appears that
> alot
> of the threads are waiting for a response from a sendSync message that is
> going out to a http endpoint (external webservice)
>
> Here is a example stack
>
> "pool-flow.seda.Esb-thread-1020" prio=6 tid=0x2b4a7430 nid=0x3a0 in
> Object.wait() [0x2fb9f000..0x2fb9fb1c]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x261b2440> (a
> org.apache.servicemix.jbi.messaging.InOptionalOutImpl)
> at
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.waitForExchange(
> DeliveryChannelImpl.java:679)
> at
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.sendSync(
> DeliveryChannelImpl.java:458)
> - locked <0x261b2440> (a
> org.apache.servicemix.jbi.messaging.InOptionalOutImpl)
> at
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.sendSync(
> DeliveryChannelImpl.java:428)
> at
> org.apache.servicemix.components.util.PojoSupport.sendSync(
> PojoSupport.java:201)
> at esb.SheetMgmtServiceESB.sendStringMessage(
> SheetMgmtServiceESB.java:1824)
> at esb.SheetMgmtServiceESB.outOfUv(SheetMgmtServiceESB.java:1012)
> at
> sheetmgmt.esb.SheetMgmtServiceESB.onMessageExchange(
> SheetMgmtServiceESB.java:445)
> at
>
> org.apache.servicemix.components.util.ComponentAdaptorMEListener.onMessageExchange
> (ComponentAdaptorMEListener.java:47)
> at
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(
> DeliveryChannelImpl.java:593)
> at
> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(
> AbstractFlow.java:174)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java
> :176)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java
> :134)
> at
>
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> (ThreadPoolExecutor.java:665)
> at
>
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:690)
> at java.lang.Thread.run(Thread.java:595)
>
>
>
> Any suggestions of where to start looking for problems?
>
> Would I make the situation better by moving from a seda queue to a jms
> type
> flow? (I don't think that it would but I want to verify)
>
> Is there any sort of timeout for a message (http bc) that would fail the
> message?
>
> I've not done any special configuration for the http bc endpoint. As
> such,
> what are the default number of threads that I can expect for outgoing
> calls.
> Would a blocking of threads caused by outgoing calls have an effect on
> incoming calls?
>
> Thanks,
> James
> --
> View this message in context:
> http://www.nabble.com/http-lock-up-tf3488493s12049.html#a10142439
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
------------------------
Principal Engineer, IONA
Blog: http://gnodet.blogspot.com/