Camel hangs during shutdown

[zikas <eu...@gmail.com>]

Hi!

Playing a bit with DefaultShutdownStrategy and possibility of redelivering,
I've turned out that from time to time camel can hang.. I've stuck trying to
understand the problem and how I can avoid it.

To check shutdown process. I've just created handler 

public class LazyHandler {
    private static final Object monitor = new Object();

    @Handler
    public void handle(String s) throws InterruptedException {
        System.out.println("Received, let's wait a bit: " + s);
        synchronized (monitor) {
            try {
                monitor.wait();
            } catch (InterruptedException e) {
                System.out.println("Interrupted");
                throw e;
            }
        }
        PublicCamel.producer().sendBody("activemq:nfd.processed", s);
        System.out.println("Processed: " + s);
    }
}

So, idea is to emulate situation when consumer just awaits for some long
process is accomplished (like blocked IO)

I have the following configuration of dead letter error handler
   <bean id="deadLetterEH"
class="org.apache.camel.builder.DeadLetterChannelBuilder">
        <property name="deadLetterUri" value="activemq:nfd.dead"/>
        <property name="redeliveryPolicy" ref="redeliveryPolicyConfig"/>
    </bean>

    <bean id="redeliveryPolicyConfig"
class="org.apache.camel.processor.RedeliveryPolicy">
        <property name="maximumRedeliveries" value="3"/>
        <property name="redeliveryDelay" value="5000"/>
        <property name="maximumRedeliveryDelay" value="5000"/>
        <property name="allowRedeliveryWhileStopping" value="false"/>
    </bean>


My investigation showed two possible ways of execution

=== DefaultShutdownStrategy:171 ===

   Future<?> future = getExecutorService().submit(new ShutdownTask(context,
routesOrdered, timeout, timeUnit, suspendOnly, abortAfterTimeout));
        try {
            if (timeout > 0) {
                future.get(timeout, timeUnit);
            } else {
                future.get();
            }
        } catch (TimeoutException e) {
            // timeout then cancel the task
            future.cancel(true);

            // signal we are forcing shutdown now, since timeout occurred
            this.forceShutdown = forceShutdown;
			
			[SHUTDOWN ROUTES]
		}

Looks like once TimeoutException is throwed we have thread racing (one is in
catch block, another is in ShutdownTask logic) and both threads try to
shutdown consumers
Observation showed that when Thread-1 wins we have hanged Camel (finally it
hangs somewhere at DefaultMessageListenerContainer:doShutdown trying to
attemt notification for this.lifecycleMonitor.wait()), otherwise shutdown
completes successfully


Thread-1@3424, prio=5, in group 'main', status: 'RUNNING'
	  at
org.springframework.jms.listener.AbstractJmsListeningContainer.shutdown(AbstractJmsListeningContainer.java:213)
	  at
org.springframework.jms.listener.AbstractJmsListeningContainer.destroy(AbstractJmsListeningContainer.java:173)
	  at
org.apache.camel.component.jms.JmsConsumer.stopAndDestroyListenerContainer(JmsConsumer.java:177)
	  at
org.apache.camel.component.jms.JmsConsumer.doStop(JmsConsumer.java:212)
	  at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:92)
	  at
org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:114)
	  at
org.apache.camel.impl.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:306)
	  at
org.apache.camel.impl.DefaultShutdownStrategy.shutdownRoutesNow(DefaultShutdownStrategy.java:280)
	  at
org.apache.camel.impl.DefaultShutdownStrategy.doShutdown(DefaultShutdownStrategy.java:193)
	  at
org.apache.camel.impl.DefaultShutdownStrategy.shutdownForced(DefaultShutdownStrategy.java:122)
	  at
org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:1630)
	  at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:92)
	  at
org.apache.camel.core.xml.AbstractCamelContextFactoryBean.destroy(AbstractCamelContextFactoryBean.java:432)
	  at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:211)
	  at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:500)
	  at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:476)
	  at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:445)
	  at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1078)
	  at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1052)
	  at
org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1000)
	  at org.apache.camel.spring.Main.doStop(Main.java:152)
	  at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:92)
	  at
org.apache.camel.main.MainSupport$HangupInterceptor.run(MainSupport.java:82)''
	  
	  
Camel (nfdData) thread #2 - ShutdownTask@3438 daemon, prio=5, in group
'main', status: 'RUNNING'
	  at
org.springframework.jms.listener.AbstractJmsListeningContainer.shutdown(AbstractJmsListeningContainer.java:211)
	  at
org.springframework.jms.listener.AbstractJmsListeningContainer.destroy(AbstractJmsListeningContainer.java:173)
	  at
org.apache.camel.component.jms.JmsConsumer.stopAndDestroyListenerContainer(JmsConsumer.java:177)
	  at
org.apache.camel.component.jms.JmsConsumer.doStop(JmsConsumer.java:212)
	  at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:92)
	  at
org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:114)
	  at
org.apache.camel.impl.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:306)
	  at
org.apache.camel.impl.DefaultShutdownStrategy$ShutdownTask.run(DefaultShutdownStrategy.java:571)
	  at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	  at java.lang.Thread.run(Thread.java:722)
	  
	  
Full logs for the both cases are attached to the post.
success.log <http://camel.465427.n5.nabble.com/file/n5734050/success.log>  
failed.log <http://camel.465427.n5.nabble.com/file/n5734050/failed.log>  
(process was terminated)

Could you please advice, what's going on, maybe Camel isn't properly
configured?.. Or this may be caused by activemq (when camel hangs, any other
attempt to receive message from the queue just finishes without any result
despite queue is not empty)



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-hangs-during-shutdown-tp5734050.html
Sent from the Camel - Users mailing list archive at Nabble.com.