Re: Servicemix deadlocking on JMS threads (Giving up)

[Ryan Moquin <fr...@gmail.com>]

Ok, so I'm concluding that the deliveryMode setting on the jms:provider
endpoint doesn't work.  No matter what I try, I cannot get my messages to be
nonpersistent, even though I confirmed the deliveryMode is being set to the
correct property.  I'll just have to find another option to prevent my whole
service from going out of commission as a result.

On Tue, Aug 19, 2008 at 9:17 PM, Ryan Moquin <fr...@gmail.com> wrote:

> Just a little bit more.  So everything I find on this issue, seems to
> indicate I guess that since the JMS delivery mode is persistent by default,
> you should turn off persistence if you don't want a producer to block.  So
> since I don't want my service to block, nor do I want the messages
> persistent, I configured my jms provider like this:
>
> <jms:provider service="not:notification-jms-service"
>                 endpoint="notification"
>                 pubSubDomain="true"
>                 destinationName="topic.notication"
>                 connectionFactory="#connectionFactory"
>                 deliveryMode="1"
>                 timeToLive="1000"/>
>
> As far as I can tell (since it's not documented what the values of
> deliveryMode are), this should keep my producer from blocking, but it still
> does after it hits a little over a 1000 messages.  Is there a way to turn
> off persistence in my producers that work?  I have not found a way to make
> this happen.
>
>
> On Tue, Aug 19, 2008 at 7:54 PM, Ryan Moquin <fr...@gmail.com>wrote:
>
>> Maybe this is the better way to ask this as I try to rationalize this
>> behavior in my head:
>>
>> If you have a service unit that sends messages to a topic created by a
>> servicemix-jms provider endpoint that has no consumers, shouldn't every
>> message sent to it, be discarded?  I'm not really sure I actually know what
>> the problem is, since all my configuration attempts to fix it have failed.
>> All I want at this point is any way possible to prevent the system from
>> deadlocking.  I'd be happy going back to exceptions being thrown all the
>> time instead.  At one point with Servicemix 3.2.2, we've processed lots and
>> lots of messages (eventually Servicemix ran out of memory and crashed, no
>> clue as to why that happened consistently every 2 hours, but that's another
>> issue, I'm guessing because ActiveMQ never discards any messages).  We
>> haven't changed any of the servicemix configurations since then, so I'm not
>> sure exactly why we can only process a small amount before Servicemix will
>> hang from ActiveMQ blocking.
>>
>> I'm very puzzled at this point.
>>
>>
>> On Tue, Aug 19, 2008 at 10:33 AM, Ryan Moquin <fr...@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> So, due to my service units hanging under certain conditions when I try
>>> to send a JMS message using an injected DeliveryChannel, I switched to using
>>> a ServicemixClient I look up out of the servicemix JNDI context.  This is
>>> done by several of my deployed services that are doing various pieces of
>>> work.  I'm now noticing that while switching to a separate ServicemixClient
>>> has fixed some of my issues, I'm now seeing certain services that send
>>> messages to a JMS service unit to send out of the syste, are also now
>>> deadlocking around the same point and therefore my service halts.  I did a
>>> thread dump to verify this and I see A LOT of threads that are waiting on a
>>> specific lock with id: 0x1abeba80.  You can see one of these below:
>>>
>>> "pool-flow.seda.servicemix-jms-thread-1157" prio=6 tid=0x3cee2400
>>> nid=0x1a00 waiting on condition [0
>>> x3869f000..0x3869fa98]
>>>    java.lang.Thread.State: WAITING (parking)
>>>         at sun.misc.Unsafe.park(Native Method)
>>>         - parking to wait for  <0x1abeba80> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer
>>> $ConditionObject)
>>>         at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>         at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueue
>>> dSynchronizer.java:1925)
>>>         at
>>> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
>>>         at
>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
>>> l.java:663)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>         at java.lang.Thread.run(Thread.java:619)
>>>
>>> My service itself is also locking on the same id, while trying to send a
>>> message to the JMS consumer (I think is the right term):
>>>
>>> "QuartzScheduler_Worker-7" prio=6 tid=0x3a2f1400 nid=0x1050 waiting on
>>> condition [0x37e2e000..0x37e2
>>> fc18]
>>>    java.lang.Thread.State: WAITING (parking)
>>>         at sun.misc.Unsafe.park(Native Method)
>>>         - parking to wait for  <0x1abeba80> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer
>>> $ConditionObject)
>>>         at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>         at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueue
>>> dSynchronizer.java:1925)
>>>         at
>>> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
>>>         at
>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
>>> l.java:663)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExec
>>> utor.java:1737)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>>>         at
>>> org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:182)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
>>>         at
>>> org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
>>>         at
>>> org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88
>>> )
>>>         at
>>> org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830)
>>>         at
>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:3
>>> 95)
>>>         at
>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431
>>> )
>>>         at
>>> org.apache.servicemix.jms.multiplexing.MultiplexingProviderProcessor.process(Multiplexing
>>> ProviderProcessor.java:143)
>>>         at
>>> org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:538)
>>>         at
>>> org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:4
>>> 90)
>>>         at
>>> org.apache.servicemix.common.BaseLifeCycle.onMessageExchange(BaseLifeCycle.java:46)
>>>         at
>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImp
>>> l.java:610)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:167)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExec
>>> utor.java:1737)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>>>         at
>>> org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:182)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162)
>>>         at
>>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
>>>         at
>>> org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
>>>         at
>>> org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88
>>> )
>>>         at
>>> org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830)
>>>         at
>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:3
>>> 95)
>>>         at
>>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431
>>> )
>>>         at
>>> org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:14
>>> 7)
>>>         at
>>> org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:21
>>> 1)
>>>
>>> So my question is, is activemq freezing up for some reason?  All my
>>> problems I have, seem to somehow stem to activemq in various ways, usually
>>> all my freeze ups seem to be attributed to ActiveMQ freezing up for one
>>> reason or another.  If the above case is that ActiveMQ is freezing up (it
>>> happens over a period of time, I thought I read that ActiveMQ will block
>>> when a topic or Queue fills up, which makes no sense to me in regards to a
>>> topic, since I thought that a topic doesn't hold messages unless they are
>>> marked durable), are there settings I can use to keep this from happening?
>>> There is no way I can ever trust this sort of system in production unless I
>>> can be sure that Servicemix/ActiveMQ isn't going to lock up on me after a
>>> couple hours.
>>>
>>> In case you want to see how I'm using the ServicemixClient, I'm not doing
>>> anything overly special, but here it is:
>>>
>>> if (client == null) {
>>>         ClientFactory factory = (ClientFactory) new
>>> InitialContext().lookup(ClientFactory.DEFAULT_JNDI_NAME);
>>>         client = factory.createClient();
>>>       }
>>>
>>> if (destinationQname == null) {
>>>         destinationQname = new QName(getDestinationNamespace(),
>>> getDestinationService());
>>>       }
>>>       EndpointResolver resolver =
>>> client.createResolverForService(destinationQname);
>>>       client.send(resolver, null, null, new
>>> StringSource(writer.toString()));
>>>
>>> I'm not missing anything in the above code am I?
>>> Thanks,
>>> Ryan
>>>
>>
>>
>