You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by subh <su...@aol.com> on 2010/02/25 05:59:30 UTC
ServicemixBean thread issue..
I have a simple flow as below
External message producer --> SMX JMS queue --> SMX Bean --> Camel router
--> SMX JMS Topic
The servicemix bean has the business logic to generate the outgoing message.
To forward the message onto the next service, I am using ServiceMixClient.
I am using the ClientFactory to create a instance of the client.
ClientFactory factory = new
InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
ServiceMixClient client = factory.createClient();
But after processing 1000 - 1500 messages, the Servicemix bean component
becomes very slow to respond. On checking the log, I see that the bean
thread count increases considerably and the expired threads are not garbage
collected.
Due to this, all incoming messages are getting blocked which forces us to
restart the servicemix server. Once it is restarted, the processing becomes
very fast and we come to the same situation again in a day or so.
Am I doing something wrong by using the ServicemixClient inside the bean? If
no, then what are the best practices for using the Servicemix client class.
Are there alternative ways to specify the "target service" from the SMX bean
endpoint?
Any hint or suggestion would be helpful.
I am using activemq5.3 and servicemix 3.3
Regards
Subh
--
View this message in context: http://old.nabble.com/ServicemixBean-thread-issue..-tp27714350p27714350.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: ServicemixBean thread issue..
Posted by subh <su...@aol.com>.
And the all the messages shows this on thread dump..
"pool-flow.seda.servicemix-jms-thread-1433" - Thread t@6141
java.lang.Thread.State: WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1540b71
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:676)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@19353a3
.............
............
subh wrote:
>
> Continuing the discussion. I am seeing jms messages getting stuck in the
> queue and not getting published to the topic..On taking thread dump, I saw
> one instance where SMX client I am using was in waiting state..causing all
> the messages subsequent to that to be queued.
>
> Also, I think it happened while both jmsconsumers (i have another consumer
> and provider combination) was receiving messages simultaneously although
> the other jms provider has a higher priority than this one.
>
> Any idea/suggestions on what might be causing it will be great..
>
> Below is my jmsconsumer
>
> <jms:consumer service="orderservice:extOrderConsumerQueue"
> endpoint="jmsInQueueConsumer"
> targetService="orderservice:orderPublisherBean"
> targetEndpoint="endpoint"
> destinationName="queue/ORS/OrderIn"
> pubSubDomain="false"
> cacheLevel="1"
> connectionFactory="#connectionFactory" />
>
> and jmsprovider
>
> <jms:provider service="orderservice:orsDispatchedOrder"
> endpoint="orsProvider"
> destinationName="topic/ORS/OrderIn"
> connectionFactory="#connectionFactory"
> pubSubDomain="true"
> />
>
> and this the SMX bean code using SMX client to publish message to the
> topic
>
> private static final String ORS_LOE_JMS_PROVIDER_SERVICE =
> "service:http://services.fuelquest.com/orderservice/orsDispatchedOrder";
> ......
> .....
> if (exchange.getStatus() == ExchangeStatus.ACTIVE) {
> smxclient = SMXClient.getInstance();
> Destination destination =
> smxclient.createDestination(ORS_LOE_JMS_PROVIDER_SERVICE);
> Message message = destination.createInOnlyMessage();
> message.setProperty(REPLY_HEADER, Boolean.TRUE);
> message.setBody(messageBody);
> smxclient.send(message);
> smxclient.done(exchange);
> }
>
> and the snippet from the thread dump....
>
> "pool-flow.seda.servicemix-bean-thread-1975" - Thread t@6142
> java.lang.Thread.State: WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1540b71
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
> at
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:676)
> at
> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
> at
> java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at
> org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:53)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:183)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:163)
> at
> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
> at
> org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
> at
> org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
> at
> org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:903)
> at
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:396)
> at
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:432)
> at
> org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:89)
> at
> org.apache.servicemix.common.endpoints.SimpleEndpoint.send(SimpleEndpoint.java:70)
> at
> org.apache.servicemix.common.endpoints.SimpleEndpoint.done(SimpleEndpoint.java:85)
> at
> org.apache.servicemix.jms.endpoints.JmsProviderEndpoint.process(JmsProviderEndpoint.java:500)
> at
> org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:627)
> at
> org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:581)
> at
> org.apache.servicemix.common.AsyncBaseLifeCycle.onMessageExchange(AsyncBaseLifeCycle.java:535)
> at
> org.apache.servicemix.common.SyncLifeCycleWrapper.onMessageExchange(SyncLifeCycleWrapper.java:60)
> at
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:623)
> at
> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
> at
> java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at
> org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:53)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:183)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:163)
> at
> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
> at
> org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
> at
> org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
> at
> org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:903)
> at
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:396)
> at
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:432)
> at
> org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:148)
> at
> org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:152)
> at
> com.fuelquest.service.fqservices.orsesb.beansu.OrderServicePublishCallerBean.writeOrderPublishedMessageToSendOut(OrderServicePublishCallerBean.java:235)
> at
> com.fuelquest.service.fqservices.orsesb.beansu.OrderServicePublishCallerBean.onMessageExchange(OrderServicePublishCallerBean.java:141)
> at
> org.apache.servicemix.bean.BeanEndpoint.onProviderExchange(BeanEndpoint.java:230)
> - locked org.apache.servicemix.bean.support.Request@17474ef
> at org.apache.servicemix.bean.BeanEndpoint.process(BeanEndpoint.java:217)
> at
> org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:627)
> at
> org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:581)
> at
> org.apache.servicemix.common.AsyncBaseLifeCycle.onMessageExchange(AsyncBaseLifeCycle.java:535)
> at
> org.apache.servicemix.common.SyncLifeCycleWrapper.onMessageExchange(SyncLifeCycleWrapper.java:60)
> at
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:623)
> at
> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
> at
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>
> Locked ownable synchronizers:
> - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@a93013
>
>
>
> Eighty8 wrote:
>>
>> No probelem. I saw blocks i the Saxon API's too...which is used by
>> things other than eip...good luck.
>> -A
>>
>> subh wrote:
>>>
>>> Thanks for the input and suggestions.
>>>
>>> As of now, I have removed the EIP component from the flow as I had
>>> observed the exchange waiting for threads to finish.
>>>
>>> Also, I have defined separate connection pool for the jms components as
>>> suggested in a different thread. I am seeing much better performance,
>>> but will need to monitor for a few days to.
>>>
>>>
>>>
>>> Eighty8 wrote:
>>>>
>>>> I found I had to throttle my messages by sleeping ~250-500ms apart. I
>>>> recommend buying a YourKit license and attaching it to your server.
>>>> You'll see a lot more of what's going on under the hood. Look for
>>>> 'lots of read' in the threads indicating blocks that are waiting to
>>>> clear.
>>>>
>>>> I have not had success, yet, using component.properties to controls the
>>>> eip and saxon LifeCycles, but am continuing to try to get that working.
>>>>
>>>> You can inspect the configurable properties by enabling jetty on your
>>>> ESB and access jmx-console.
>>>> Good luck.
>>>>
>>>>
>>>> subh wrote:
>>>>>
>>>>> I have a simple flow as below
>>>>>
>>>>> External message producer --> SMX JMS queue --> SMX Bean --> Camel
>>>>> router --> SMX JMS Topic
>>>>>
>>>>> The servicemix bean has the business logic to generate the outgoing
>>>>> message.
>>>>> To forward the message onto the next service, I am using
>>>>> ServiceMixClient.
>>>>>
>>>>> I am using the ClientFactory to create a instance of the client.
>>>>>
>>>>> ClientFactory factory = new
>>>>> InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
>>>>> ServiceMixClient client = factory.createClient();
>>>>>
>>>>> But after processing 1000 - 1500 messages, the Servicemix bean
>>>>> component becomes very slow to respond. On checking the log, I see
>>>>> that the bean thread count increases considerably and the expired
>>>>> threads are not garbage collected.
>>>>>
>>>>> Due to this, all incoming messages are getting blocked which forces us
>>>>> to restart the servicemix server. Once it is restarted, the processing
>>>>> becomes very fast and we come to the same situation again in a day or
>>>>> so.
>>>>>
>>>>> Am I doing something wrong by using the ServicemixClient inside the
>>>>> bean? If no, then what are the best practices for using the Servicemix
>>>>> client class.
>>>>>
>>>>> Are there alternative ways to specify the "target service" from the
>>>>> SMX bean endpoint?
>>>>>
>>>>> Any hint or suggestion would be helpful.
>>>>>
>>>>> I am using activemq5.3 and servicemix 3.3
>>>>>
>>>>> Regards
>>>>> Subh
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
--
View this message in context: http://old.nabble.com/ServicemixBean-thread-issue..-tp27714350p27823959.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: ServicemixBean thread issue..
Posted by subh <su...@aol.com>.
Continuing the discussion. I am seeing jms messages getting stuck in the
queue and not getting published to the topic..On taking thread dump, I saw
one instance where SMX client I am using was in waiting state..causing all
the messages subsequent to that to be queued.
Also, I think it happened while both jmsconsumers (i have another consumer
and provider combination) was receiving messages simultaneously although the
other jms provider has a higher priority than this one.
Any idea/suggestions on what might be causing it will be great..
Below is my jmsconsumer
<jms:consumer service="orderservice:extOrderConsumerQueue"
endpoint="jmsInQueueConsumer"
targetService="orderservice:orderPublisherBean"
targetEndpoint="endpoint"
destinationName="queue/ORS/OrderIn"
pubSubDomain="false"
cacheLevel="1"
connectionFactory="#connectionFactory" />
and jmsprovider
<jms:provider service="orderservice:orsDispatchedOrder"
endpoint="orsProvider"
destinationName="topic/ORS/OrderIn"
connectionFactory="#connectionFactory"
pubSubDomain="true"
/>
and this the SMX bean code using SMX client to publish message to the topic
private static final String ORS_LOE_JMS_PROVIDER_SERVICE =
"service:http://services.fuelquest.com/orderservice/orsDispatchedOrder";
......
.....
if (exchange.getStatus() == ExchangeStatus.ACTIVE) {
smxclient = SMXClient.getInstance();
Destination destination =
smxclient.createDestination(ORS_LOE_JMS_PROVIDER_SERVICE);
Message message = destination.createInOnlyMessage();
message.setProperty(REPLY_HEADER, Boolean.TRUE);
message.setBody(messageBody);
smxclient.send(message);
smxclient.done(exchange);
}
and the snippet from the thread dump....
"pool-flow.seda.servicemix-bean-thread-1975" - Thread t@6142
java.lang.Thread.State: WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1540b71
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:676)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:53)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:183)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:163)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
at
org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
at
org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
at
org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:903)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:396)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:432)
at
org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:89)
at
org.apache.servicemix.common.endpoints.SimpleEndpoint.send(SimpleEndpoint.java:70)
at
org.apache.servicemix.common.endpoints.SimpleEndpoint.done(SimpleEndpoint.java:85)
at
org.apache.servicemix.jms.endpoints.JmsProviderEndpoint.process(JmsProviderEndpoint.java:500)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:627)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:581)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.onMessageExchange(AsyncBaseLifeCycle.java:535)
at
org.apache.servicemix.common.SyncLifeCycleWrapper.onMessageExchange(SyncLifeCycleWrapper.java:60)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:623)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:53)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:183)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:163)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
at
org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
at
org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
at
org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:903)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:396)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:432)
at
org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:148)
at
org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:152)
at
com.fuelquest.service.fqservices.orsesb.beansu.OrderServicePublishCallerBean.writeOrderPublishedMessageToSendOut(OrderServicePublishCallerBean.java:235)
at
com.fuelquest.service.fqservices.orsesb.beansu.OrderServicePublishCallerBean.onMessageExchange(OrderServicePublishCallerBean.java:141)
at
org.apache.servicemix.bean.BeanEndpoint.onProviderExchange(BeanEndpoint.java:230)
- locked org.apache.servicemix.bean.support.Request@17474ef
at org.apache.servicemix.bean.BeanEndpoint.process(BeanEndpoint.java:217)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:627)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:581)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.onMessageExchange(AsyncBaseLifeCycle.java:535)
at
org.apache.servicemix.common.SyncLifeCycleWrapper.onMessageExchange(SyncLifeCycleWrapper.java:60)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:623)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@a93013
Eighty8 wrote:
>
> No probelem. I saw blocks i the Saxon API's too...which is used by things
> other than eip...good luck.
> -A
>
> subh wrote:
>>
>> Thanks for the input and suggestions.
>>
>> As of now, I have removed the EIP component from the flow as I had
>> observed the exchange waiting for threads to finish.
>>
>> Also, I have defined separate connection pool for the jms components as
>> suggested in a different thread. I am seeing much better performance, but
>> will need to monitor for a few days to.
>>
>>
>>
>> Eighty8 wrote:
>>>
>>> I found I had to throttle my messages by sleeping ~250-500ms apart. I
>>> recommend buying a YourKit license and attaching it to your server.
>>> You'll see a lot more of what's going on under the hood. Look for 'lots
>>> of read' in the threads indicating blocks that are waiting to clear.
>>>
>>> I have not had success, yet, using component.properties to controls the
>>> eip and saxon LifeCycles, but am continuing to try to get that working.
>>>
>>> You can inspect the configurable properties by enabling jetty on your
>>> ESB and access jmx-console.
>>> Good luck.
>>>
>>>
>>> subh wrote:
>>>>
>>>> I have a simple flow as below
>>>>
>>>> External message producer --> SMX JMS queue --> SMX Bean --> Camel
>>>> router --> SMX JMS Topic
>>>>
>>>> The servicemix bean has the business logic to generate the outgoing
>>>> message.
>>>> To forward the message onto the next service, I am using
>>>> ServiceMixClient.
>>>>
>>>> I am using the ClientFactory to create a instance of the client.
>>>>
>>>> ClientFactory factory = new
>>>> InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
>>>> ServiceMixClient client = factory.createClient();
>>>>
>>>> But after processing 1000 - 1500 messages, the Servicemix bean
>>>> component becomes very slow to respond. On checking the log, I see that
>>>> the bean thread count increases considerably and the expired threads
>>>> are not garbage collected.
>>>>
>>>> Due to this, all incoming messages are getting blocked which forces us
>>>> to restart the servicemix server. Once it is restarted, the processing
>>>> becomes very fast and we come to the same situation again in a day or
>>>> so.
>>>>
>>>> Am I doing something wrong by using the ServicemixClient inside the
>>>> bean? If no, then what are the best practices for using the Servicemix
>>>> client class.
>>>>
>>>> Are there alternative ways to specify the "target service" from the SMX
>>>> bean endpoint?
>>>>
>>>> Any hint or suggestion would be helpful.
>>>>
>>>> I am using activemq5.3 and servicemix 3.3
>>>>
>>>> Regards
>>>> Subh
>>>>
>>>
>>>
>>
>>
>
>
--
View this message in context: http://old.nabble.com/ServicemixBean-thread-issue..-tp27714350p27823844.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: ServicemixBean thread issue..
Posted by Eighty8 <ah...@adobe.com>.
No probelem. I saw blocks i the Saxon API's too...which is used by things
other than eip...good luck.
-A
subh wrote:
>
> Thanks for the input and suggestions.
>
> As of now, I have removed the EIP component from the flow as I had
> observed the exchange waiting for threads to finish.
>
> Also, I have defined separate connection pool for the jms components as
> suggested in a different thread. I am seeing much better performance, but
> will need to monitor for a few days to.
>
>
>
> Eighty8 wrote:
>>
>> I found I had to throttle my messages by sleeping ~250-500ms apart. I
>> recommend buying a YourKit license and attaching it to your server.
>> You'll see a lot more of what's going on under the hood. Look for 'lots
>> of read' in the threads indicating blocks that are waiting to clear.
>>
>> I have not had success, yet, using component.properties to controls the
>> eip and saxon LifeCycles, but am continuing to try to get that working.
>>
>> You can inspect the configurable properties by enabling jetty on your ESB
>> and access jmx-console.
>> Good luck.
>>
>>
>> subh wrote:
>>>
>>> I have a simple flow as below
>>>
>>> External message producer --> SMX JMS queue --> SMX Bean --> Camel
>>> router --> SMX JMS Topic
>>>
>>> The servicemix bean has the business logic to generate the outgoing
>>> message.
>>> To forward the message onto the next service, I am using
>>> ServiceMixClient.
>>>
>>> I am using the ClientFactory to create a instance of the client.
>>>
>>> ClientFactory factory = new
>>> InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
>>> ServiceMixClient client = factory.createClient();
>>>
>>> But after processing 1000 - 1500 messages, the Servicemix bean component
>>> becomes very slow to respond. On checking the log, I see that the bean
>>> thread count increases considerably and the expired threads are not
>>> garbage collected.
>>>
>>> Due to this, all incoming messages are getting blocked which forces us
>>> to restart the servicemix server. Once it is restarted, the processing
>>> becomes very fast and we come to the same situation again in a day or
>>> so.
>>>
>>> Am I doing something wrong by using the ServicemixClient inside the
>>> bean? If no, then what are the best practices for using the Servicemix
>>> client class.
>>>
>>> Are there alternative ways to specify the "target service" from the SMX
>>> bean endpoint?
>>>
>>> Any hint or suggestion would be helpful.
>>>
>>> I am using activemq5.3 and servicemix 3.3
>>>
>>> Regards
>>> Subh
>>>
>>
>>
>
>
--
View this message in context: http://old.nabble.com/ServicemixBean-thread-issue..-tp27714350p27772710.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: ServicemixBean thread issue..
Posted by subh <su...@aol.com>.
Thanks for the input and suggestions.
As of now, I have removed the EIP component from the flow as I had observed
the exchange waiting for threads to finish.
Also, I have defined separate connection pool for the jms components as
suggested in a different thread. I am seeing much better performance, but
will need to monitor for a few days to.
Eighty8 wrote:
>
> I found I had to throttle my messages by sleeping ~250-500ms apart. I
> recommend buying a YourKit license and attaching it to your server.
> You'll see a lot more of what's going on under the hood. Look for 'lots
> of read' in the threads indicating blocks that are waiting to clear.
>
> I have not had success, yet, using component.properties to controls the
> eip and saxon LifeCycles, but am continuing to try to get that working.
>
> You can inspect the configurable properties by enabling jetty on your ESB
> and access jmx-console.
> Good luck.
>
>
> subh wrote:
>>
>> I have a simple flow as below
>>
>> External message producer --> SMX JMS queue --> SMX Bean --> Camel router
>> --> SMX JMS Topic
>>
>> The servicemix bean has the business logic to generate the outgoing
>> message.
>> To forward the message onto the next service, I am using
>> ServiceMixClient.
>>
>> I am using the ClientFactory to create a instance of the client.
>>
>> ClientFactory factory = new
>> InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
>> ServiceMixClient client = factory.createClient();
>>
>> But after processing 1000 - 1500 messages, the Servicemix bean component
>> becomes very slow to respond. On checking the log, I see that the bean
>> thread count increases considerably and the expired threads are not
>> garbage collected.
>>
>> Due to this, all incoming messages are getting blocked which forces us to
>> restart the servicemix server. Once it is restarted, the processing
>> becomes very fast and we come to the same situation again in a day or so.
>>
>> Am I doing something wrong by using the ServicemixClient inside the bean?
>> If no, then what are the best practices for using the Servicemix client
>> class.
>>
>> Are there alternative ways to specify the "target service" from the SMX
>> bean endpoint?
>>
>> Any hint or suggestion would be helpful.
>>
>> I am using activemq5.3 and servicemix 3.3
>>
>> Regards
>> Subh
>>
>
>
--
View this message in context: http://old.nabble.com/ServicemixBean-thread-issue..-tp27714350p27772707.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: ServicemixBean thread issue..
Posted by Eighty8 <ah...@adobe.com>.
I found I had to throttle my messages by sleeping ~250-500ms apart. I
recommend buying a YourKit license and attaching it to your server. You'll
see a lot more of what's going on under the hood. Look for 'lots of read'
in the threads indicating blocks that are waiting to clear.
I have not had success, yet, using component.properties to controls the eip
and saxon LifeCycles, but am continuing to try to get that working.
You can inspect the configurable properties by enabling jetty on your ESB
and access jmx-console.
Good luck.
subh wrote:
>
> I have a simple flow as below
>
> External message producer --> SMX JMS queue --> SMX Bean --> Camel router
> --> SMX JMS Topic
>
> The servicemix bean has the business logic to generate the outgoing
> message.
> To forward the message onto the next service, I am using ServiceMixClient.
>
> I am using the ClientFactory to create a instance of the client.
>
> ClientFactory factory = new
> InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
> ServiceMixClient client = factory.createClient();
>
> But after processing 1000 - 1500 messages, the Servicemix bean component
> becomes very slow to respond. On checking the log, I see that the bean
> thread count increases considerably and the expired threads are not
> garbage collected.
>
> Due to this, all incoming messages are getting blocked which forces us to
> restart the servicemix server. Once it is restarted, the processing
> becomes very fast and we come to the same situation again in a day or so.
>
> Am I doing something wrong by using the ServicemixClient inside the bean?
> If no, then what are the best practices for using the Servicemix client
> class.
>
> Are there alternative ways to specify the "target service" from the SMX
> bean endpoint?
>
> Any hint or suggestion would be helpful.
>
> I am using activemq5.3 and servicemix 3.3
>
> Regards
> Subh
>
--
View this message in context: http://old.nabble.com/ServicemixBean-thread-issue..-tp27714350p27772153.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: ServicemixBean thread issue..
Posted by Freeman Fang <fr...@gmail.com>.
Hi,
I guess you are using client.sendSync but not client.send, right?
If so, use client.send as much as possible to avoid the thread locked
waiting for response(which in consequence no thread left to handle the
later incoming request message).
Freeman
On 2010-2-25, at 下午12:59, subh wrote:
>
> I have a simple flow as below
>
> External message producer --> SMX JMS queue --> SMX Bean --> Camel
> router
> --> SMX JMS Topic
>
> The servicemix bean has the business logic to generate the outgoing
> message.
> To forward the message onto the next service, I am using
> ServiceMixClient.
>
> I am using the ClientFactory to create a instance of the client.
>
> ClientFactory factory = new
> InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
> ServiceMixClient client = factory.createClient();
>
> But after processing 1000 - 1500 messages, the Servicemix bean
> component
> becomes very slow to respond. On checking the log, I see that the bean
> thread count increases considerably and the expired threads are not
> garbage
> collected.
>
> Due to this, all incoming messages are getting blocked which forces
> us to
> restart the servicemix server. Once it is restarted, the processing
> becomes
> very fast and we come to the same situation again in a day or so.
>
> Am I doing something wrong by using the ServicemixClient inside the
> bean? If
> no, then what are the best practices for using the Servicemix client
> class.
>
> Are there alternative ways to specify the "target service" from the
> SMX bean
> endpoint?
>
> Any hint or suggestion would be helpful.
>
> I am using activemq5.3 and servicemix 3.3
>
> Regards
> Subh
> --
> View this message in context: http://old.nabble.com/ServicemixBean-thread-issue..-tp27714350p27714350.html
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
--
Freeman Fang
------------------------
Open Source SOA: http://fusesource.com
Re: ServicemixBean thread issue..
Posted by Ryan Moquin <fr...@gmail.com>.
A couple comments. What you are describing sounds suspiciously like some of
your requests aren't being handled or returned properly. Since you are
sending the requests asynchronously, then this would explain why it starts
hanging between 1000-1500 messages. I actually have experienced the same
problem and while I thought it was a servicemix bug of some kind, it was a
problem in my message flow. I would guarentee you that the service you are
sending messages to has a bug in it. You might want to trace it and make
sure it's returning EVERY exchange. If it received a request it can't
handle, then it should throw an exception. It sounds like there is a can
where you aren't doing that. If you mishandle an exchange, it will end up
sitting on a queue, hogging a thread and eventually you'll have the exact
situation you are describing.
As to your other question, the best way to send a message from a bean
component to another component is to follow the info on this page:
http://servicemix.apache.org/servicemix-bean.html
It tells you how to have a deliverychannel injected into your bean and you
can use that to send your exchanges. Regardless, there is a problem with
exchanges hanging in your scenario. I think if you look at jconsole and you
look at the stacktrace for some of those threads you might be able to see
why they are hanging or at the very least you'll see they are stuck in
transit.
Ryan
On Thu, Feb 25, 2010 at 10:02 AM, subh <su...@aol.com> wrote:
>
> Hi Freeman,
>
> I am using client.send. This is how I use it..
>
> smxclient = SMXClient.getInstance();
> Destination destination =
> smxclient.createDestination(ORS_EIP_STATIC_RECIPIENT_SERVICE);
> Message message = destination.createInOnlyMessage();
> message.setProperty(REPLY_HEADER, Boolean.TRUE);
> message.setContent(new StringSource(messageBody));
> smxclient.send(message);
> smxclient.done(exchange);
>
>
> Eighty8 wrote:
> >
> > Perhaps related to my post about generateId( ) synchronized call is
> > blocking. Only saw this with a profiler tool, but maybe you'd see it if
> > you took a thread dump and analyzed it.
> >
> >
> > subh wrote:
> >>
> >> I have a simple flow as below
> >>
> >> External message producer --> SMX JMS queue --> SMX Bean --> Camel
> router
> >> --> SMX JMS Topic
> >>
> >> The servicemix bean has the business logic to generate the outgoing
> >> message.
> >> To forward the message onto the next service, I am using
> >> ServiceMixClient.
> >>
> >> I am using the ClientFactory to create a instance of the client.
> >>
> >> ClientFactory factory = new
> >> InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
> >> ServiceMixClient client = factory.createClient();
> >>
> >> But after processing 1000 - 1500 messages, the Servicemix bean component
> >> becomes very slow to respond. On checking the log, I see that the bean
> >> thread count increases considerably and the expired threads are not
> >> garbage collected.
> >>
> >> Due to this, all incoming messages are getting blocked which forces us
> to
> >> restart the servicemix server. Once it is restarted, the processing
> >> becomes very fast and we come to the same situation again in a day or
> so.
> >>
> >> Am I doing something wrong by using the ServicemixClient inside the
> bean?
> >> If no, then what are the best practices for using the Servicemix client
> >> class.
> >>
> >> Are there alternative ways to specify the "target service" from the SMX
> >> bean endpoint?
> >>
> >> Any hint or suggestion would be helpful.
> >>
> >> I am using activemq5.3 and servicemix 3.3
> >>
> >> Regards
> >> Subh
> >>
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/ServicemixBean-thread-issue..-tp27714350p27714450.html
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
Re: ServicemixBean thread issue..
Posted by subh <su...@aol.com>.
Hi Freeman,
I am using client.send. This is how I use it..
smxclient = SMXClient.getInstance();
Destination destination =
smxclient.createDestination(ORS_EIP_STATIC_RECIPIENT_SERVICE);
Message message = destination.createInOnlyMessage();
message.setProperty(REPLY_HEADER, Boolean.TRUE);
message.setContent(new StringSource(messageBody));
smxclient.send(message);
smxclient.done(exchange);
Eighty8 wrote:
>
> Perhaps related to my post about generateId( ) synchronized call is
> blocking. Only saw this with a profiler tool, but maybe you'd see it if
> you took a thread dump and analyzed it.
>
>
> subh wrote:
>>
>> I have a simple flow as below
>>
>> External message producer --> SMX JMS queue --> SMX Bean --> Camel router
>> --> SMX JMS Topic
>>
>> The servicemix bean has the business logic to generate the outgoing
>> message.
>> To forward the message onto the next service, I am using
>> ServiceMixClient.
>>
>> I am using the ClientFactory to create a instance of the client.
>>
>> ClientFactory factory = new
>> InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
>> ServiceMixClient client = factory.createClient();
>>
>> But after processing 1000 - 1500 messages, the Servicemix bean component
>> becomes very slow to respond. On checking the log, I see that the bean
>> thread count increases considerably and the expired threads are not
>> garbage collected.
>>
>> Due to this, all incoming messages are getting blocked which forces us to
>> restart the servicemix server. Once it is restarted, the processing
>> becomes very fast and we come to the same situation again in a day or so.
>>
>> Am I doing something wrong by using the ServicemixClient inside the bean?
>> If no, then what are the best practices for using the Servicemix client
>> class.
>>
>> Are there alternative ways to specify the "target service" from the SMX
>> bean endpoint?
>>
>> Any hint or suggestion would be helpful.
>>
>> I am using activemq5.3 and servicemix 3.3
>>
>> Regards
>> Subh
>>
>
>
--
View this message in context: http://old.nabble.com/ServicemixBean-thread-issue..-tp27714350p27714450.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: ServicemixBean thread issue..
Posted by Eighty8 <ah...@adobe.com>.
Perhaps related to my post about generateId( ) synchronized call is blocking.
Only saw this with a profiler tool, but maybe you'd see it if you took a
thread dump and analyzed it.
subh wrote:
>
> I have a simple flow as below
>
> External message producer --> SMX JMS queue --> SMX Bean --> Camel router
> --> SMX JMS Topic
>
> The servicemix bean has the business logic to generate the outgoing
> message.
> To forward the message onto the next service, I am using ServiceMixClient.
>
> I am using the ClientFactory to create a instance of the client.
>
> ClientFactory factory = new
> InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
> ServiceMixClient client = factory.createClient();
>
> But after processing 1000 - 1500 messages, the Servicemix bean component
> becomes very slow to respond. On checking the log, I see that the bean
> thread count increases considerably and the expired threads are not
> garbage collected.
>
> Due to this, all incoming messages are getting blocked which forces us to
> restart the servicemix server. Once it is restarted, the processing
> becomes very fast and we come to the same situation again in a day or so.
>
> Am I doing something wrong by using the ServicemixClient inside the bean?
> If no, then what are the best practices for using the Servicemix client
> class.
>
> Are there alternative ways to specify the "target service" from the SMX
> bean endpoint?
>
> Any hint or suggestion would be helpful.
>
> I am using activemq5.3 and servicemix 3.3
>
> Regards
> Subh
>
--
View this message in context: http://old.nabble.com/ServicemixBean-thread-issue..-tp27714350p27714366.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.