You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@uima.apache.org by Satya Nand Kanodia <sa...@orkash.com> on 2015/04/01 13:12:47 UTC
Re: Job gets stuck for infinite time
Hi,
yeah It can be done in the way you are telling. But there is a parameter
in ducc.properties for specifying broker on any system. Why isn't there
any way for Job Driver? Because in any network of systems there is no
guaranty that all will be available at certain time. Either we should
have a option to specify job driver on the system which has highest
availability or If a system running a job driver goes down , DUCC Head
should allocate a new job driver.
On 03/30/2015 06:02 PM, Lou DeGenaro wrote:
> I don't think forcing the JD node and the broker on the same host will help
> much if your network has issues since all parts of DUCC communicate thru
> the network, some via the broker.
>
> Nevertheless, if you have only two machines then you could first start DUCC
> only on the machine that you wish to have the JD+broker. After DUCC is up
> on the single machine, you could start the DUCC agent on the other machine.
>
> Lou.
>
> On Mon, Mar 30, 2015 at 12:34 AM, Satya Nand Kanodia <
> satya.kanodia@orkash.com> wrote:
>
>> Hi,
>>
>> Actually I have two systems. One is running broker and other one is
>> running job driver( I could not find any way to force job driver to be on
>> any specific system, It automatically goes on 2nd system). I think when the
>> connectivity goes, the problem comes because broker and driver are on two
>> different systems. Is there any way I can force job driver to be on the
>> same system, which is running broker.
>>
>>
>> On 03/24/2015 06:00 PM, Lou DeGenaro wrote:
>>
>>> By "serious" I simply meant that if your network is having problems then
>>> applications that are dependent on the network will be having problems,
>>> DUCC
>>> being one such application. Granted, DUCC's Job Driver should not take
>>> NPE,
>>> but fixing the NPE may not make your application run much better.
>>>
>>> I recommend fixing your network, then it won't matter if you run DUCC 1.x
>>> or DUCC 2.x.
>>>
>>> I can make no promises, but my guess is that DUCC 2.0 will be released in
>>> April or May.
>>>
>>> If this error is really a problem for you and you need a fix for DUCC 1.x,
>>> I can try to do that - but I'd prefer to leave it as Won't Fix in light of
>>> the re-write that was done for DUCC 2.0.
>>>
>>> Lou.
>>>
>>> On Tue, Mar 24, 2015 at 12:50 AM, Satya Nand Kanodia <
>>> satya.kanodia@orkash.com> wrote:
>>>
>>> Dear Lou,
>>>> It some time happens in weekends. when It happens we also get some
>>>> exceptions of networking like exception in connection with database etc
>>>> etc. So upgrading to DUCC 2.0 is a solution or we can do some thing else
>>>> ?
>>>> what did you mean by more serious problems?
>>>>
>>>> On 03/23/2015 06:26 PM, Lou DeGenaro wrote:
>>>>
>>>> A Jira already exists for this issue https://issues.apache.org/
>>>>> jira/browse/
>>>>> UIMA-4225. It has been marked "Won't Fix" because the Job Driver and
>>>>> corresponding Job Process have been re-written for 2.0 version of DUCC
>>>>> (not
>>>>> yet released, but available from trunk).
>>>>>
>>>>> Are you able to reproduce this error with regularity? I've only seen it
>>>>> once before, which resulted in Jira 4225 and its subsequent closure. It
>>>>> also seems that you may have more serious problems than just one job
>>>>> failing. Based on the message:
>>>>>
>>>>> 22 Mar 2015 08:49:49,723 14 WARN jms.DefaultJmsMessageListenerC
>>>>> ontainer Setup of JMS message listener invoker failed for destination
>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover.
>>>>>
>>>>> it seems you may have network or broker problems?
>>>>>
>>>>> Lou.
>>>>>
>>>>>
>>>>> On Mon, Mar 23, 2015 at 2:16 AM, Satya Nand Kanodia <
>>>>> satya.kanodia@orkash.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>> In DUCC when I submit a job , sometimes it gets hung for infinite time.
>>>>>> Today it happened again. A job started with 2 processes on 22 Mar 2015
>>>>>> 04:34:38 had been continuously running till today.
>>>>>>
>>>>>> I used this configuration in job file
>>>>>>
>>>>>> process_thread_count=3
>>>>>> process_failures_limit=50
>>>>>> process_per_item_time_max=5
>>>>>> process_deployments_max=2
>>>>>> attach_console=true
>>>>>> process_jvm_args=-Xms4g -XX\:+UseConcMarkSweepGC
>>>>>> process_memory_size=7
>>>>>> classpath_order=ducc-before-user
>>>>>>
>>>>>> Finally I had to terminate it from DUCC-MON. Job usually takes 15
>>>>>> minutes
>>>>>> max to process documents. I submitted a batch of 433 documents and 342
>>>>>> documents had been processed. There were two errors of cas timed out.
>>>>>> There
>>>>>> were connection exceptions in both processes , which lasted for 10
>>>>>> seconds.
>>>>>> exception happened at 22 Mar 2015 08:50:42. After that no response from
>>>>>> both processes.
>>>>>>
>>>>>> I got these exceptions from jd.out file
>>>>>>
>>>>>> These exception in jd.out file started at 05:09:12.
>>>>>>
>>>>>> 22 Mar 2015 05:32:12,352 154 ERROR jd.JobDriver waitForProcessStatus
>>>>>> N/A process status error? java.lang.NullPointerException
>>>>>>
>>>>>> At:
>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>> JobDriver.java:1081)
>>>>>> org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(
>>>>>> JobDriver.java:1123)
>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>> java:1585)
>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>> WorkItem.java:196)
>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>> java:160)
>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>> call(Executors.java:471)
>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1145)
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>> ThreadPoolExecutor.java:615)
>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> 22 Mar 2015 05:32:12,353 154 ERROR jd.JobDriver exception 19524
>>>>>> processing error? java.lang.NullPointerException
>>>>>>
>>>>>> At:
>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>> JobDriver.java:1081)
>>>>>> org.apache.uima.ducc.jd.JobDriver.isNodeFailure(
>>>>>> JobDriver.java:1092)
>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>> java:1586)
>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>> WorkItem.java:196)
>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>> java:160)
>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>> call(Executors.java:471)
>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1145)
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>> ThreadPoolExecutor.java:615)
>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> 22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver waitForProcessStatus
>>>>>> N/A process status error? java.lang.NullPointerException
>>>>>>
>>>>>> At:
>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>> JobDriver.java:1081)
>>>>>> org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(
>>>>>> JobDriver.java:1123)
>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>> java:1585)
>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>> WorkItem.java:196)
>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>> java:160)
>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>> call(Executors.java:471)
>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1145)
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>> ThreadPoolExecutor.java:615)
>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> 22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver exception 19524
>>>>>> processing error? java.lang.NullPointerException
>>>>>>
>>>>>> At:
>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>> JobDriver.java:1081)
>>>>>> org.apache.uima.ducc.jd.JobDriver.isNodeFailure(
>>>>>> JobDriver.java:1092)
>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>> java:1586)
>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>> WorkItem.java:196)
>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>> java:160)
>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>> call(Executors.java:471)
>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1145)
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>> ThreadPoolExecutor.java:615)
>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> 22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver waitForProcessStatus
>>>>>> N/A process status error? java.lang.NullPointerException
>>>>>>
>>>>>> At:
>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>> JobDriver.java:1081)
>>>>>> org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(
>>>>>> JobDriver.java:1123)
>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>> java:1585)
>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>> WorkItem.java:196)
>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>> java:160)
>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>> call(Executors.java:471)
>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1145)
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>> ThreadPoolExecutor.java:615)
>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> 22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver exception 19524
>>>>>> processing error? java.lang.NullPointerException
>>>>>>
>>>>>> At:
>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>> JobDriver.java:1081)
>>>>>> org.apache.uima.ducc.jd.JobDriver.isNodeFailure(
>>>>>> JobDriver.java:1092)
>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>> java:1586)
>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>> WorkItem.java:196)
>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>> java:160)
>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>> call(Executors.java:471)
>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1145)
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>> ThreadPoolExecutor.java:615)
>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> 22 Mar 2015 07:10:59,845 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 07:13:09,855 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 07:16:50,007 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 07:24:35,994 14 WARN jms.EndpointMessageListener Execution
>>>>>> of
>>>>>> JMS message listener failed. Caused by: [javax.jms.
>>>>>> IllegalStateException
>>>>>> - The Session is closed]
>>>>>> javax.jms.IllegalStateException: The Session is closed
>>>>>> at org.apache.activemq.ActiveMQSession.checkClosed(
>>>>>> ActiveMQSession.java:731)
>>>>>> at org.apache.activemq.ActiveMQSession.getTransacted(
>>>>>> ActiveMQSession.java:521)
>>>>>> at org.apache.activemq.pool.PooledSession.getTransacted(
>>>>>> PooledSession.java:256)
>>>>>> at org.springframework.jms.listener.
>>>>>> AbstractMessageListenerContain
>>>>>> er.commitIfNecessary(AbstractMessageListenerContainer.java:571)
>>>>>> at org.springframework.jms.listener.
>>>>>> AbstractMessageListenerContain
>>>>>> er.doExecuteListener(AbstractMessageListenerContainer.java:481)
>>>>>> at org.springframework.jms.listener.
>>>>>> AbstractPollingMessageListener
>>>>>> Container.doReceiveAndExecute(AbstractPollingMessageListener
>>>>>> Container.java:325)
>>>>>> at org.springframework.jms.listener.
>>>>>> AbstractPollingMessageListener
>>>>>> Container.receiveAndExecute(AbstractPollingMessageListener
>>>>>> Container.java:263)
>>>>>> at org.springframework.jms.listener.
>>>>>> DefaultMessageListenerContaine
>>>>>> r$AsyncMessageListenerInvoker.invokeListener(
>>>>>> DefaultMessageListenerContainer.java:1058)
>>>>>> at org.springframework.jms.listener.
>>>>>> DefaultMessageListenerContaine
>>>>>> r$AsyncMessageListenerInvoker.executeOngoingLoop(
>>>>>> DefaultMessageListenerContainer.java:1050)
>>>>>> at org.springframework.jms.listener.
>>>>>> DefaultMessageListenerContaine
>>>>>> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine
>>>>>> r.java:947)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>> ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 22 Mar 2015 07:24:36,617 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: The
>>>>>> Session is closed
>>>>>> 22 Mar 2015 07:26:31,822 14 WARN jms.EndpointMessageListener Execution
>>>>>> of
>>>>>> JMS message listener failed. Caused by: [javax.jms.
>>>>>> IllegalStateException
>>>>>> - The Session is closed]
>>>>>> javax.jms.IllegalStateException: The Session is closed
>>>>>> at org.apache.activemq.ActiveMQSession.checkClosed(
>>>>>> ActiveMQSession.java:731)
>>>>>> at org.apache.activemq.ActiveMQSession.getTransacted(
>>>>>> ActiveMQSession.java:521)
>>>>>> at org.apache.activemq.pool.PooledSession.getTransacted(
>>>>>> PooledSession.java:256)
>>>>>> at org.springframework.jms.listener.
>>>>>> AbstractMessageListenerContain
>>>>>> er.commitIfNecessary(AbstractMessageListenerContainer.java:571)
>>>>>> at org.springframework.jms.listener.
>>>>>> AbstractMessageListenerContain
>>>>>> er.doExecuteListener(AbstractMessageListenerContainer.java:481)
>>>>>> at org.springframework.jms.listener.
>>>>>> AbstractPollingMessageListener
>>>>>> Container.doReceiveAndExecute(AbstractPollingMessageListener
>>>>>> Container.java:325)
>>>>>> at org.springframework.jms.listener.
>>>>>> AbstractPollingMessageListener
>>>>>> Container.receiveAndExecute(AbstractPollingMessageListener
>>>>>> Container.java:263)
>>>>>> at org.springframework.jms.listener.
>>>>>> DefaultMessageListenerContaine
>>>>>> r$AsyncMessageListenerInvoker.invokeListener(
>>>>>> DefaultMessageListenerContainer.java:1058)
>>>>>> at org.springframework.jms.listener.
>>>>>> DefaultMessageListenerContaine
>>>>>> r$AsyncMessageListenerInvoker.executeOngoingLoop(
>>>>>> DefaultMessageListenerContainer.java:1050)
>>>>>> at org.springframework.jms.listener.
>>>>>> DefaultMessageListenerContaine
>>>>>> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine
>>>>>> r.java:947)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>> ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 22 Mar 2015 07:26:32,067 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: The
>>>>>> Session is closed
>>>>>> 22 Mar 2015 07:28:12,075 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 07:30:55,001 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Could not refresh JMS Connection for destination 'ducc.orchestrator.
>>>>>> abbreviated.state'
>>>>>> - retrying in 5000 ms. Cause: Channel was inactive for too (>30000)
>>>>>> long:
>>>>>> tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 08:29:26,669 14 WARN jms.EndpointMessageListener Execution
>>>>>> of
>>>>>> JMS message listener failed. Caused by: [javax.jms.
>>>>>> IllegalStateException
>>>>>> - The Session is closed]
>>>>>> javax.jms.IllegalStateException: The Session is closed
>>>>>> at org.apache.activemq.ActiveMQSession.checkClosed(
>>>>>> ActiveMQSession.java:731)
>>>>>> at org.apache.activemq.ActiveMQSession.getTransacted(
>>>>>> ActiveMQSession.java:521)
>>>>>> at org.apache.activemq.pool.PooledSession.getTransacted(
>>>>>> PooledSession.java:256)
>>>>>> at org.springframework.jms.listener.
>>>>>> AbstractMessageListenerContain
>>>>>> er.commitIfNecessary(AbstractMessageListenerContainer.java:571)
>>>>>> at org.springframework.jms.listener.
>>>>>> AbstractMessageListenerContain
>>>>>> er.doExecuteListener(AbstractMessageListenerContainer.java:481)
>>>>>> at org.springframework.jms.listener.
>>>>>> AbstractPollingMessageListener
>>>>>> Container.doReceiveAndExecute(AbstractPollingMessageListener
>>>>>> Container.java:325)
>>>>>> at org.springframework.jms.listener.
>>>>>> AbstractPollingMessageListener
>>>>>> Container.receiveAndExecute(AbstractPollingMessageListener
>>>>>> Container.java:263)
>>>>>> at org.springframework.jms.listener.
>>>>>> DefaultMessageListenerContaine
>>>>>> r$AsyncMessageListenerInvoker.invokeListener(
>>>>>> DefaultMessageListenerContainer.java:1058)
>>>>>> at org.springframework.jms.listener.
>>>>>> DefaultMessageListenerContaine
>>>>>> r$AsyncMessageListenerInvoker.executeOngoingLoop(
>>>>>> DefaultMessageListenerContainer.java:1050)
>>>>>> at org.springframework.jms.listener.
>>>>>> DefaultMessageListenerContaine
>>>>>> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine
>>>>>> r.java:947)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>> ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 22 Mar 2015 08:29:26,671 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: The
>>>>>> Session is closed
>>>>>> 22 Mar 2015 08:33:06,678 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 08:36:16,841 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 08:37:27,506 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 08:39:07,876 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 08:42:08,029 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 08:42:39,341 15 ERROR processor.DefaultErrorHandler Failed
>>>>>> delivery for (MessageId: ID-S143-52155-1426979058276-0-2895 on
>>>>>> ExchangeId: ID-S143-52155-1426979058276-0-2896). Exhausted after
>>>>>> delivery
>>>>>> attempt: 1 caught: org.springframework.jms.UncategorizedJmsException:
>>>>>> Uncategorized exception occured during JMS processing; nested exception
>>>>>> is
>>>>>> javax.jms.JMSException: Channel was inactive for too (>30000) long:
>>>>>> tcp://
>>>>>> 192.168.10.144:61617
>>>>>> org.springframework.jms.UncategorizedJmsException: Uncategorized
>>>>>> exception occured during JMS processing; nested exception is
>>>>>> javax.jms.JMSException: Channel was inactive for too (>30000) long:
>>>>>> tcp://
>>>>>> 192.168.10.144:61617
>>>>>> at org.springframework.jms.support.JmsUtils.
>>>>>> convertJmsAccessException(JmsUtils.java:316)
>>>>>> at org.springframework.jms.support.JmsAccessor.
>>>>>> convertJmsAccessException(JmsAccessor.java:168)
>>>>>> at org.springframework.jms.core.JmsTemplate.execute(
>>>>>> JmsTemplate.java:469)
>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>> CamelJmsTemplate.send(JmsConfiguration.java:188)
>>>>>> at org.apache.camel.component.jms.JmsProducer.doSend(
>>>>>> JmsProducer.java:398)
>>>>>> at org.apache.camel.component.jms.JmsProducer.processInOnly(
>>>>>> JmsProducer.java:352)
>>>>>> at org.apache.camel.component.jms.JmsProducer.process(
>>>>>> JmsProducer.java:132)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:73)
>>>>>> at org.apache.camel.processor.SendProcessor$2.
>>>>>> doInAsyncProducer(
>>>>>> SendProcessor.java:120)
>>>>>> at org.apache.camel.impl.ProducerCache.doInAsyncProducer(
>>>>>> ProducerCache.java:292)
>>>>>> at org.apache.camel.processor.SendProcessor.process(
>>>>>> SendProcessor.java:115)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:73)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> processNext(
>>>>>> DelegateAsyncProcessor.java:99)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:90)
>>>>>> at org.apache.camel.management.InstrumentationProcessor.
>>>>>> process(
>>>>>> InstrumentationProcessor.java:73)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:73)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> processNext(
>>>>>> DelegateAsyncProcessor.java:99)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:90)
>>>>>> at org.apache.camel.processor.interceptor.TraceInterceptor.
>>>>>> process(TraceInterceptor.java:91)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:73)
>>>>>> at org.apache.camel.processor.RedeliveryErrorHandler.
>>>>>> processErrorHandler(RedeliveryErrorHandler.java:330)
>>>>>> at org.apache.camel.processor.RedeliveryErrorHandler.
>>>>>> process(
>>>>>> RedeliveryErrorHandler.java:220)
>>>>>> at org.apache.camel.processor.RouteContextProcessor.
>>>>>> processNext(
>>>>>> RouteContextProcessor.java:45)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:90)
>>>>>> at org.apache.camel.processor.interceptor.DefaultChannel.
>>>>>> process(DefaultChannel.java:303)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:73)
>>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.
>>>>>> java:117)
>>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.
>>>>>> java:80)
>>>>>> at org.apache.camel.processor.RouteContextProcessor.
>>>>>> processNext(
>>>>>> RouteContextProcessor.java:45)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:90)
>>>>>> at org.apache.camel.processor.UnitOfWorkProcessor.
>>>>>> processAsync(
>>>>>> UnitOfWorkProcessor.java:150)
>>>>>> at org.apache.camel.processor.UnitOfWorkProcessor.process(
>>>>>> UnitOfWorkProcessor.java:117)
>>>>>> at org.apache.camel.processor.RouteInflightRepositoryProcess
>>>>>> or.processNext(RouteInflightRepositoryProcessor.java:48)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:90)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:73)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> processNext(
>>>>>> DelegateAsyncProcessor.java:99)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:90)
>>>>>> at org.apache.camel.management.InstrumentationProcessor.
>>>>>> process(
>>>>>> InstrumentationProcessor.java:73)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:99)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:86)
>>>>>> at org.apache.camel.component.timer.TimerConsumer.
>>>>>> sendTimerExchange(TimerConsumer.java:134)
>>>>>> at org.apache.camel.component.timer.TimerConsumer$1.run(
>>>>>> TimerConsumer.java:63)
>>>>>> at java.util.TimerThread.mainLoop(Timer.java:555)
>>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>> Caused by: javax.jms.JMSException: Channel was inactive for too
>>>>>> (>30000)
>>>>>> long: tcp://192.168.10.144:61617
>>>>>> at org.apache.activemq.util.JMSExceptionSupport.create(
>>>>>> JMSExceptionSupport.java:49)
>>>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket(
>>>>>> ActiveMQConnection.java:1377)
>>>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket(
>>>>>> ActiveMQConnection.java:1305)
>>>>>> at org.apache.activemq.ActiveMQSession.send(
>>>>>> ActiveMQSession.java:1795)
>>>>>> at org.apache.activemq.ActiveMQMessageProducer.send(
>>>>>> ActiveMQMessageProducer.java:277)
>>>>>> at org.apache.activemq.ActiveMQMessageProducer.send(
>>>>>> ActiveMQMessageProducer.java:212)
>>>>>> at org.apache.activemq.pool.PooledProducer.send(
>>>>>> PooledProducer.java:79)
>>>>>> at org.apache.activemq.pool.PooledProducer.send(
>>>>>> PooledProducer.java:62)
>>>>>> at org.springframework.jms.core.JmsTemplate.doSend(
>>>>>> JmsTemplate.java:592)
>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>> CamelJmsTemplate.doSend(JmsConfiguration.java:276)
>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>> CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215)
>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>> CamelJmsTemplate.access$100(JmsConfiguration.java:157)
>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>> CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191)
>>>>>> at org.springframework.jms.core.JmsTemplate.execute(
>>>>>> JmsTemplate.java:466)
>>>>>> ... 41 more
>>>>>> Caused by: org.apache.activemq.transport.InactivityIOException:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> at org.apache.activemq.transport.
>>>>>> AbstractInactivityMonitor$4.
>>>>>> run(AbstractInactivityMonitor.java:209)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>> ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 22 Mar 2015 08:42:39,546 15 WARN timer.TimerConsumer Error processing
>>>>>> exchange. Exchange[Message: org.apache.uima.ducc.transport.event.
>>>>>> JdStateDuccEvent@db25b23]. Caused by: [org.springframework.jms.
>>>>>> UncategorizedJmsException
>>>>>> - Uncategorized exception occured during JMS processing; nested
>>>>>> exception
>>>>>> is javax.jms.JMSException: Channel was inactive for too (>30000) long:
>>>>>> tcp://192.168.10.144:61617]
>>>>>> org.springframework.jms.UncategorizedJmsException: Uncategorized
>>>>>> exception occured during JMS processing; nested exception is
>>>>>> javax.jms.JMSException: Channel was inactive for too (>30000) long:
>>>>>> tcp://
>>>>>> 192.168.10.144:61617
>>>>>> at org.springframework.jms.support.JmsUtils.
>>>>>> convertJmsAccessException(JmsUtils.java:316)
>>>>>> at org.springframework.jms.support.JmsAccessor.
>>>>>> convertJmsAccessException(JmsAccessor.java:168)
>>>>>> at org.springframework.jms.core.JmsTemplate.execute(
>>>>>> JmsTemplate.java:469)
>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>> CamelJmsTemplate.send(JmsConfiguration.java:188)
>>>>>> at org.apache.camel.component.jms.JmsProducer.doSend(
>>>>>> JmsProducer.java:398)
>>>>>> at org.apache.camel.component.jms.JmsProducer.processInOnly(
>>>>>> JmsProducer.java:352)
>>>>>> at org.apache.camel.component.jms.JmsProducer.process(
>>>>>> JmsProducer.java:132)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:73)
>>>>>> at org.apache.camel.processor.SendProcessor$2.
>>>>>> doInAsyncProducer(
>>>>>> SendProcessor.java:120)
>>>>>> at org.apache.camel.impl.ProducerCache.doInAsyncProducer(
>>>>>> ProducerCache.java:292)
>>>>>> at org.apache.camel.processor.SendProcessor.process(
>>>>>> SendProcessor.java:115)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:73)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> processNext(
>>>>>> DelegateAsyncProcessor.java:99)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:90)
>>>>>> at org.apache.camel.management.InstrumentationProcessor.
>>>>>> process(
>>>>>> InstrumentationProcessor.java:73)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:73)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> processNext(
>>>>>> DelegateAsyncProcessor.java:99)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:90)
>>>>>> at org.apache.camel.processor.interceptor.TraceInterceptor.
>>>>>> process(TraceInterceptor.java:91)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:73)
>>>>>> at org.apache.camel.processor.RedeliveryErrorHandler.
>>>>>> processErrorHandler(RedeliveryErrorHandler.java:330)
>>>>>> at org.apache.camel.processor.RedeliveryErrorHandler.
>>>>>> process(
>>>>>> RedeliveryErrorHandler.java:220)
>>>>>> at org.apache.camel.processor.RouteContextProcessor.
>>>>>> processNext(
>>>>>> RouteContextProcessor.java:45)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:90)
>>>>>> at org.apache.camel.processor.interceptor.DefaultChannel.
>>>>>> process(DefaultChannel.java:303)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:73)
>>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.
>>>>>> java:117)
>>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.
>>>>>> java:80)
>>>>>> at org.apache.camel.processor.RouteContextProcessor.
>>>>>> processNext(
>>>>>> RouteContextProcessor.java:45)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:90)
>>>>>> at org.apache.camel.processor.UnitOfWorkProcessor.
>>>>>> processAsync(
>>>>>> UnitOfWorkProcessor.java:150)
>>>>>> at org.apache.camel.processor.UnitOfWorkProcessor.process(
>>>>>> UnitOfWorkProcessor.java:117)
>>>>>> at org.apache.camel.processor.RouteInflightRepositoryProcess
>>>>>> or.processNext(RouteInflightRepositoryProcessor.java:48)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:90)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:73)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> processNext(
>>>>>> DelegateAsyncProcessor.java:99)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:90)
>>>>>> at org.apache.camel.management.InstrumentationProcessor.
>>>>>> process(
>>>>>> InstrumentationProcessor.java:73)
>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>> AsyncProcessorHelper.java:99)
>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>> process(
>>>>>> DelegateAsyncProcessor.java:86)
>>>>>> at org.apache.camel.component.timer.TimerConsumer.
>>>>>> sendTimerExchange(TimerConsumer.java:134)
>>>>>> at org.apache.camel.component.timer.TimerConsumer$1.run(
>>>>>> TimerConsumer.java:63)
>>>>>> at java.util.TimerThread.mainLoop(Timer.java:555)
>>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>> Caused by: javax.jms.JMSException: Channel was inactive for too
>>>>>> (>30000)
>>>>>> long: tcp://192.168.10.144:61617
>>>>>> at org.apache.activemq.util.JMSExceptionSupport.create(
>>>>>> JMSExceptionSupport.java:49)
>>>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket(
>>>>>> ActiveMQConnection.java:1377)
>>>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket(
>>>>>> ActiveMQConnection.java:1305)
>>>>>> at org.apache.activemq.ActiveMQSession.send(
>>>>>> ActiveMQSession.java:1795)
>>>>>> at org.apache.activemq.ActiveMQMessageProducer.send(
>>>>>> ActiveMQMessageProducer.java:277)
>>>>>> at org.apache.activemq.ActiveMQMessageProducer.send(
>>>>>> ActiveMQMessageProducer.java:212)
>>>>>> at org.apache.activemq.pool.PooledProducer.send(
>>>>>> PooledProducer.java:79)
>>>>>> at org.apache.activemq.pool.PooledProducer.send(
>>>>>> PooledProducer.java:62)
>>>>>> at org.springframework.jms.core.JmsTemplate.doSend(
>>>>>> JmsTemplate.java:592)
>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>> CamelJmsTemplate.doSend(JmsConfiguration.java:276)
>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>> CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215)
>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>> CamelJmsTemplate.access$100(JmsConfiguration.java:157)
>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>> CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191)
>>>>>> at org.springframework.jms.core.JmsTemplate.execute(
>>>>>> JmsTemplate.java:466)
>>>>>> ... 41 more
>>>>>> Caused by: org.apache.activemq.transport.InactivityIOException:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> at org.apache.activemq.transport.
>>>>>> AbstractInactivityMonitor$4.
>>>>>> run(AbstractInactivityMonitor.java:209)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>> ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>> ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 22 Mar 2015 08:43:19,482 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 08:44:59,642 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 08:46:10,729 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 08:48:05,731 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>> 22 Mar 2015 08:49:49,723 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>> iner
>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>> Channel
>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>
>>>>>>
>>>>>>
>>>>>> Please help me to find out root cause of this and how can I kill a job
>>>>>> or
>>>>>> start a new one when any job gets stuck after a certain time..
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards,
>>>>>> Satya Nand Kanodia
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>> Thanks and Regards,
>>>> Satya Nand Kanodia
>>>>
>>>>
>>>>
>> --
>> Thanks and Regards,
>> Satya Nand Kanodia
>>
>>
--
Thanks and Regards,
Satya Nand Kanodia
Re: Job gets stuck for infinite time
Posted by Lou DeGenaro <lo...@gmail.com>.
In resources/ducc.classes you could specify something like:
Nodepool jobdriver { nodefile jobdriver.nodes ; parent --default-- }
Class JobDriver reserve-base { nodepool = jobdriver; priority = 0;
enforce = false }
And in resources/jobdriver.nodes:
hostA
hostB
Beware that if your network has a disconnect and an inuse jobdriver node
becomes unreachable, jobs using that jobdriver node will be in trouble.
Lou.
On Wed, Apr 1, 2015 at 7:12 AM, Satya Nand Kanodia <satya.kanodia@orkash.com
> wrote:
> Hi,
>
> yeah It can be done in the way you are telling. But there is a parameter
> in ducc.properties for specifying broker on any system. Why isn't there any
> way for Job Driver? Because in any network of systems there is no guaranty
> that all will be available at certain time. Either we should have a option
> to specify job driver on the system which has highest availability or If a
> system running a job driver goes down , DUCC Head should allocate a new
> job driver.
>
>
>
> On 03/30/2015 06:02 PM, Lou DeGenaro wrote:
>
>> I don't think forcing the JD node and the broker on the same host will
>> help
>> much if your network has issues since all parts of DUCC communicate thru
>> the network, some via the broker.
>>
>> Nevertheless, if you have only two machines then you could first start
>> DUCC
>> only on the machine that you wish to have the JD+broker. After DUCC is up
>> on the single machine, you could start the DUCC agent on the other
>> machine.
>>
>> Lou.
>>
>> On Mon, Mar 30, 2015 at 12:34 AM, Satya Nand Kanodia <
>> satya.kanodia@orkash.com> wrote:
>>
>> Hi,
>>>
>>> Actually I have two systems. One is running broker and other one is
>>> running job driver( I could not find any way to force job driver to be on
>>> any specific system, It automatically goes on 2nd system). I think when
>>> the
>>> connectivity goes, the problem comes because broker and driver are on two
>>> different systems. Is there any way I can force job driver to be on the
>>> same system, which is running broker.
>>>
>>>
>>> On 03/24/2015 06:00 PM, Lou DeGenaro wrote:
>>>
>>> By "serious" I simply meant that if your network is having problems then
>>>> applications that are dependent on the network will be having problems,
>>>> DUCC
>>>> being one such application. Granted, DUCC's Job Driver should not take
>>>> NPE,
>>>> but fixing the NPE may not make your application run much better.
>>>>
>>>> I recommend fixing your network, then it won't matter if you run DUCC
>>>> 1.x
>>>> or DUCC 2.x.
>>>>
>>>> I can make no promises, but my guess is that DUCC 2.0 will be released
>>>> in
>>>> April or May.
>>>>
>>>> If this error is really a problem for you and you need a fix for DUCC
>>>> 1.x,
>>>> I can try to do that - but I'd prefer to leave it as Won't Fix in light
>>>> of
>>>> the re-write that was done for DUCC 2.0.
>>>>
>>>> Lou.
>>>>
>>>> On Tue, Mar 24, 2015 at 12:50 AM, Satya Nand Kanodia <
>>>> satya.kanodia@orkash.com> wrote:
>>>>
>>>> Dear Lou,
>>>>
>>>>> It some time happens in weekends. when It happens we also get some
>>>>> exceptions of networking like exception in connection with database etc
>>>>> etc. So upgrading to DUCC 2.0 is a solution or we can do some thing
>>>>> else
>>>>> ?
>>>>> what did you mean by more serious problems?
>>>>>
>>>>> On 03/23/2015 06:26 PM, Lou DeGenaro wrote:
>>>>>
>>>>> A Jira already exists for this issue https://issues.apache.org/
>>>>>
>>>>>> jira/browse/
>>>>>> UIMA-4225. It has been marked "Won't Fix" because the Job Driver and
>>>>>> corresponding Job Process have been re-written for 2.0 version of DUCC
>>>>>> (not
>>>>>> yet released, but available from trunk).
>>>>>>
>>>>>> Are you able to reproduce this error with regularity? I've only seen
>>>>>> it
>>>>>> once before, which resulted in Jira 4225 and its subsequent closure.
>>>>>> It
>>>>>> also seems that you may have more serious problems than just one job
>>>>>> failing. Based on the message:
>>>>>>
>>>>>> 22 Mar 2015 08:49:49,723 14 WARN jms.DefaultJmsMessageListenerC
>>>>>> ontainer Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover.
>>>>>>
>>>>>> it seems you may have network or broker problems?
>>>>>>
>>>>>> Lou.
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 23, 2015 at 2:16 AM, Satya Nand Kanodia <
>>>>>> satya.kanodia@orkash.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> In DUCC when I submit a job , sometimes it gets hung for infinite
>>>>>>> time.
>>>>>>> Today it happened again. A job started with 2 processes on 22 Mar
>>>>>>> 2015
>>>>>>> 04:34:38 had been continuously running till today.
>>>>>>>
>>>>>>> I used this configuration in job file
>>>>>>>
>>>>>>> process_thread_count=3
>>>>>>> process_failures_limit=50
>>>>>>> process_per_item_time_max=5
>>>>>>> process_deployments_max=2
>>>>>>> attach_console=true
>>>>>>> process_jvm_args=-Xms4g -XX\:+UseConcMarkSweepGC
>>>>>>> process_memory_size=7
>>>>>>> classpath_order=ducc-before-user
>>>>>>>
>>>>>>> Finally I had to terminate it from DUCC-MON. Job usually takes 15
>>>>>>> minutes
>>>>>>> max to process documents. I submitted a batch of 433 documents and
>>>>>>> 342
>>>>>>> documents had been processed. There were two errors of cas timed out.
>>>>>>> There
>>>>>>> were connection exceptions in both processes , which lasted for 10
>>>>>>> seconds.
>>>>>>> exception happened at 22 Mar 2015 08:50:42. After that no response
>>>>>>> from
>>>>>>> both processes.
>>>>>>>
>>>>>>> I got these exceptions from jd.out file
>>>>>>>
>>>>>>> These exception in jd.out file started at 05:09:12.
>>>>>>>
>>>>>>> 22 Mar 2015 05:32:12,352 154 ERROR jd.JobDriver waitForProcessStatus
>>>>>>> N/A process status error? java.lang.NullPointerException
>>>>>>>
>>>>>>> At:
>>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>>> JobDriver.java:1081)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(
>>>>>>> JobDriver.java:1123)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>>> java:1585)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>>> WorkItem.java:196)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>>> java:160)
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:471)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 22 Mar 2015 05:32:12,353 154 ERROR jd.JobDriver exception 19524
>>>>>>> processing error? java.lang.NullPointerException
>>>>>>>
>>>>>>> At:
>>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>>> JobDriver.java:1081)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.isNodeFailure(
>>>>>>> JobDriver.java:1092)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>>> java:1586)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>>> WorkItem.java:196)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>>> java:160)
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:471)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver waitForProcessStatus
>>>>>>> N/A process status error? java.lang.NullPointerException
>>>>>>>
>>>>>>> At:
>>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>>> JobDriver.java:1081)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(
>>>>>>> JobDriver.java:1123)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>>> java:1585)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>>> WorkItem.java:196)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>>> java:160)
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:471)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver exception 19524
>>>>>>> processing error? java.lang.NullPointerException
>>>>>>>
>>>>>>> At:
>>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>>> JobDriver.java:1081)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.isNodeFailure(
>>>>>>> JobDriver.java:1092)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>>> java:1586)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>>> WorkItem.java:196)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>>> java:160)
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:471)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver waitForProcessStatus
>>>>>>> N/A process status error? java.lang.NullPointerException
>>>>>>>
>>>>>>> At:
>>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>>> JobDriver.java:1081)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(
>>>>>>> JobDriver.java:1123)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>>> java:1585)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>>> WorkItem.java:196)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>>> java:160)
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:471)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver exception 19524
>>>>>>> processing error? java.lang.NullPointerException
>>>>>>>
>>>>>>> At:
>>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>>> JobDriver.java:1081)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.isNodeFailure(
>>>>>>> JobDriver.java:1092)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>>> java:1586)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>>> WorkItem.java:196)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>>> java:160)
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:471)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 22 Mar 2015 07:10:59,845 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 07:13:09,855 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 07:16:50,007 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>>
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 07:24:35,994 14 WARN jms.EndpointMessageListener
>>>>>>> Execution
>>>>>>> of
>>>>>>> JMS message listener failed. Caused by: [javax.jms.
>>>>>>> IllegalStateException
>>>>>>> - The Session is closed]
>>>>>>> javax.jms.IllegalStateException: The Session is closed
>>>>>>> at org.apache.activemq.ActiveMQSession.checkClosed(
>>>>>>> ActiveMQSession.java:731)
>>>>>>> at org.apache.activemq.ActiveMQSession.getTransacted(
>>>>>>> ActiveMQSession.java:521)
>>>>>>> at org.apache.activemq.pool.PooledSession.getTransacted(
>>>>>>> PooledSession.java:256)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractMessageListenerContain
>>>>>>> er.commitIfNecessary(AbstractMessageListenerContainer.java:571)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractMessageListenerContain
>>>>>>> er.doExecuteListener(AbstractMessageListenerContainer.java:481)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractPollingMessageListener
>>>>>>> Container.doReceiveAndExecute(AbstractPollingMessageListener
>>>>>>> Container.java:325)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractPollingMessageListener
>>>>>>> Container.receiveAndExecute(AbstractPollingMessageListener
>>>>>>> Container.java:263)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.invokeListener(
>>>>>>> DefaultMessageListenerContainer.java:1058)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.executeOngoingLoop(
>>>>>>> DefaultMessageListenerContainer.java:1050)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine
>>>>>>> r.java:947)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 22 Mar 2015 07:24:36,617 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>>
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> The
>>>>>>> Session is closed
>>>>>>> 22 Mar 2015 07:26:31,822 14 WARN jms.EndpointMessageListener
>>>>>>> Execution
>>>>>>> of
>>>>>>> JMS message listener failed. Caused by: [javax.jms.
>>>>>>> IllegalStateException
>>>>>>> - The Session is closed]
>>>>>>> javax.jms.IllegalStateException: The Session is closed
>>>>>>> at org.apache.activemq.ActiveMQSession.checkClosed(
>>>>>>> ActiveMQSession.java:731)
>>>>>>> at org.apache.activemq.ActiveMQSession.getTransacted(
>>>>>>> ActiveMQSession.java:521)
>>>>>>> at org.apache.activemq.pool.PooledSession.getTransacted(
>>>>>>> PooledSession.java:256)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractMessageListenerContain
>>>>>>> er.commitIfNecessary(AbstractMessageListenerContainer.java:571)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractMessageListenerContain
>>>>>>> er.doExecuteListener(AbstractMessageListenerContainer.java:481)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractPollingMessageListener
>>>>>>> Container.doReceiveAndExecute(AbstractPollingMessageListener
>>>>>>> Container.java:325)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractPollingMessageListener
>>>>>>> Container.receiveAndExecute(AbstractPollingMessageListener
>>>>>>> Container.java:263)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.invokeListener(
>>>>>>> DefaultMessageListenerContainer.java:1058)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.executeOngoingLoop(
>>>>>>> DefaultMessageListenerContainer.java:1050)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine
>>>>>>> r.java:947)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 22 Mar 2015 07:26:32,067 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> The
>>>>>>> Session is closed
>>>>>>> 22 Mar 2015 07:28:12,075 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>>
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 07:30:55,001 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Could not refresh JMS Connection for destination 'ducc.orchestrator.
>>>>>>> abbreviated.state'
>>>>>>> - retrying in 5000 ms. Cause: Channel was inactive for too (>30000)
>>>>>>> long:
>>>>>>> tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:29:26,669 14 WARN jms.EndpointMessageListener
>>>>>>> Execution
>>>>>>> of
>>>>>>> JMS message listener failed. Caused by: [javax.jms.
>>>>>>> IllegalStateException
>>>>>>> - The Session is closed]
>>>>>>> javax.jms.IllegalStateException: The Session is closed
>>>>>>> at org.apache.activemq.ActiveMQSession.checkClosed(
>>>>>>> ActiveMQSession.java:731)
>>>>>>> at org.apache.activemq.ActiveMQSession.getTransacted(
>>>>>>> ActiveMQSession.java:521)
>>>>>>> at org.apache.activemq.pool.PooledSession.getTransacted(
>>>>>>> PooledSession.java:256)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractMessageListenerContain
>>>>>>> er.commitIfNecessary(AbstractMessageListenerContainer.java:571)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractMessageListenerContain
>>>>>>> er.doExecuteListener(AbstractMessageListenerContainer.java:481)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractPollingMessageListener
>>>>>>> Container.doReceiveAndExecute(AbstractPollingMessageListener
>>>>>>> Container.java:325)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractPollingMessageListener
>>>>>>> Container.receiveAndExecute(AbstractPollingMessageListener
>>>>>>> Container.java:263)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.invokeListener(
>>>>>>> DefaultMessageListenerContainer.java:1058)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.executeOngoingLoop(
>>>>>>> DefaultMessageListenerContainer.java:1050)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine
>>>>>>> r.java:947)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 22 Mar 2015 08:29:26,671 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> The
>>>>>>> Session is closed
>>>>>>> 22 Mar 2015 08:33:06,678 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:36:16,841 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:37:27,506 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:39:07,876 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:42:08,029 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>>
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:42:39,341 15 ERROR processor.DefaultErrorHandler
>>>>>>> Failed
>>>>>>> delivery for (MessageId: ID-S143-52155-1426979058276-0-2895 on
>>>>>>> ExchangeId: ID-S143-52155-1426979058276-0-2896). Exhausted after
>>>>>>> delivery
>>>>>>> attempt: 1 caught: org.springframework.jms.
>>>>>>> UncategorizedJmsException:
>>>>>>> Uncategorized exception occured during JMS processing; nested
>>>>>>> exception
>>>>>>> is
>>>>>>> javax.jms.JMSException: Channel was inactive for too (>30000) long:
>>>>>>> tcp://
>>>>>>> 192.168.10.144:61617
>>>>>>> org.springframework.jms.UncategorizedJmsException: Uncategorized
>>>>>>> exception occured during JMS processing; nested exception is
>>>>>>> javax.jms.JMSException: Channel was inactive for too (>30000) long:
>>>>>>> tcp://
>>>>>>> 192.168.10.144:61617
>>>>>>> at org.springframework.jms.support.JmsUtils.
>>>>>>> convertJmsAccessException(JmsUtils.java:316)
>>>>>>> at org.springframework.jms.support.JmsAccessor.
>>>>>>> convertJmsAccessException(JmsAccessor.java:168)
>>>>>>> at org.springframework.jms.core.JmsTemplate.execute(
>>>>>>> JmsTemplate.java:469)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.send(JmsConfiguration.java:188)
>>>>>>> at org.apache.camel.component.jms.JmsProducer.doSend(
>>>>>>> JmsProducer.java:398)
>>>>>>> at org.apache.camel.component.
>>>>>>> jms.JmsProducer.processInOnly(
>>>>>>> JmsProducer.java:352)
>>>>>>> at org.apache.camel.component.jms.JmsProducer.process(
>>>>>>> JmsProducer.java:132)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.SendProcessor$2.
>>>>>>> doInAsyncProducer(
>>>>>>> SendProcessor.java:120)
>>>>>>> at org.apache.camel.impl.ProducerCache.doInAsyncProducer(
>>>>>>> ProducerCache.java:292)
>>>>>>> at org.apache.camel.processor.SendProcessor.process(
>>>>>>> SendProcessor.java:115)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> processNext(
>>>>>>> DelegateAsyncProcessor.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.management.InstrumentationProcessor.
>>>>>>> process(
>>>>>>> InstrumentationProcessor.java:73)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> processNext(
>>>>>>> DelegateAsyncProcessor.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.processor.
>>>>>>> interceptor.TraceInterceptor.
>>>>>>> process(TraceInterceptor.java:91)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.RedeliveryErrorHandler.
>>>>>>> processErrorHandler(RedeliveryErrorHandler.java:330)
>>>>>>> at org.apache.camel.processor.RedeliveryErrorHandler.
>>>>>>> process(
>>>>>>> RedeliveryErrorHandler.java:220)
>>>>>>> at org.apache.camel.processor.RouteContextProcessor.
>>>>>>> processNext(
>>>>>>> RouteContextProcessor.java:45)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.processor.interceptor.DefaultChannel.
>>>>>>> process(DefaultChannel.java:303)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.
>>>>>>> java:117)
>>>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.
>>>>>>> java:80)
>>>>>>> at org.apache.camel.processor.RouteContextProcessor.
>>>>>>> processNext(
>>>>>>> RouteContextProcessor.java:45)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.processor.UnitOfWorkProcessor.
>>>>>>> processAsync(
>>>>>>> UnitOfWorkProcessor.java:150)
>>>>>>> at org.apache.camel.processor.
>>>>>>> UnitOfWorkProcessor.process(
>>>>>>> UnitOfWorkProcessor.java:117)
>>>>>>> at org.apache.camel.processor.
>>>>>>> RouteInflightRepositoryProcess
>>>>>>> or.processNext(RouteInflightRepositoryProcessor.java:48)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> processNext(
>>>>>>> DelegateAsyncProcessor.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.management.InstrumentationProcessor.
>>>>>>> process(
>>>>>>> InstrumentationProcessor.java:73)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:86)
>>>>>>> at org.apache.camel.component.timer.TimerConsumer.
>>>>>>> sendTimerExchange(TimerConsumer.java:134)
>>>>>>> at org.apache.camel.component.timer.TimerConsumer$1.run(
>>>>>>> TimerConsumer.java:63)
>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:555)
>>>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>>> Caused by: javax.jms.JMSException: Channel was inactive for too
>>>>>>> (>30000)
>>>>>>> long: tcp://192.168.10.144:61617
>>>>>>> at org.apache.activemq.util.JMSExceptionSupport.create(
>>>>>>> JMSExceptionSupport.java:49)
>>>>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket(
>>>>>>> ActiveMQConnection.java:1377)
>>>>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket(
>>>>>>> ActiveMQConnection.java:1305)
>>>>>>> at org.apache.activemq.ActiveMQSession.send(
>>>>>>> ActiveMQSession.java:1795)
>>>>>>> at org.apache.activemq.ActiveMQMessageProducer.send(
>>>>>>> ActiveMQMessageProducer.java:277)
>>>>>>> at org.apache.activemq.ActiveMQMessageProducer.send(
>>>>>>> ActiveMQMessageProducer.java:212)
>>>>>>> at org.apache.activemq.pool.PooledProducer.send(
>>>>>>> PooledProducer.java:79)
>>>>>>> at org.apache.activemq.pool.PooledProducer.send(
>>>>>>> PooledProducer.java:62)
>>>>>>> at org.springframework.jms.core.JmsTemplate.doSend(
>>>>>>> JmsTemplate.java:592)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.doSend(JmsConfiguration.java:276)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.access$100(JmsConfiguration.java:157)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191)
>>>>>>> at org.springframework.jms.core.JmsTemplate.execute(
>>>>>>> JmsTemplate.java:466)
>>>>>>> ... 41 more
>>>>>>> Caused by: org.apache.activemq.transport.InactivityIOException:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> at org.apache.activemq.transport.
>>>>>>> AbstractInactivityMonitor$4.
>>>>>>> run(AbstractInactivityMonitor.java:209)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 22 Mar 2015 08:42:39,546 15 WARN timer.TimerConsumer Error
>>>>>>> processing
>>>>>>> exchange. Exchange[Message: org.apache.uima.ducc.transport.event.
>>>>>>> JdStateDuccEvent@db25b23]. Caused by: [org.springframework.jms.
>>>>>>> UncategorizedJmsException
>>>>>>> - Uncategorized exception occured during JMS processing; nested
>>>>>>> exception
>>>>>>> is javax.jms.JMSException: Channel was inactive for too (>30000)
>>>>>>> long:
>>>>>>> tcp://192.168.10.144:61617]
>>>>>>> org.springframework.jms.UncategorizedJmsException: Uncategorized
>>>>>>> exception occured during JMS processing; nested exception is
>>>>>>> javax.jms.JMSException: Channel was inactive for too (>30000) long:
>>>>>>> tcp://
>>>>>>> 192.168.10.144:61617
>>>>>>> at org.springframework.jms.support.JmsUtils.
>>>>>>> convertJmsAccessException(JmsUtils.java:316)
>>>>>>> at org.springframework.jms.support.JmsAccessor.
>>>>>>> convertJmsAccessException(JmsAccessor.java:168)
>>>>>>> at org.springframework.jms.core.JmsTemplate.execute(
>>>>>>> JmsTemplate.java:469)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.send(JmsConfiguration.java:188)
>>>>>>> at org.apache.camel.component.jms.JmsProducer.doSend(
>>>>>>> JmsProducer.java:398)
>>>>>>> at org.apache.camel.component.
>>>>>>> jms.JmsProducer.processInOnly(
>>>>>>> JmsProducer.java:352)
>>>>>>> at org.apache.camel.component.jms.JmsProducer.process(
>>>>>>> JmsProducer.java:132)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.SendProcessor$2.
>>>>>>> doInAsyncProducer(
>>>>>>> SendProcessor.java:120)
>>>>>>> at org.apache.camel.impl.ProducerCache.doInAsyncProducer(
>>>>>>> ProducerCache.java:292)
>>>>>>> at org.apache.camel.processor.SendProcessor.process(
>>>>>>> SendProcessor.java:115)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> processNext(
>>>>>>> DelegateAsyncProcessor.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.management.InstrumentationProcessor.
>>>>>>> process(
>>>>>>> InstrumentationProcessor.java:73)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> processNext(
>>>>>>> DelegateAsyncProcessor.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.processor.
>>>>>>> interceptor.TraceInterceptor.
>>>>>>> process(TraceInterceptor.java:91)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.RedeliveryErrorHandler.
>>>>>>> processErrorHandler(RedeliveryErrorHandler.java:330)
>>>>>>> at org.apache.camel.processor.RedeliveryErrorHandler.
>>>>>>> process(
>>>>>>> RedeliveryErrorHandler.java:220)
>>>>>>> at org.apache.camel.processor.RouteContextProcessor.
>>>>>>> processNext(
>>>>>>> RouteContextProcessor.java:45)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.processor.interceptor.DefaultChannel.
>>>>>>> process(DefaultChannel.java:303)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.
>>>>>>> java:117)
>>>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.
>>>>>>> java:80)
>>>>>>> at org.apache.camel.processor.RouteContextProcessor.
>>>>>>> processNext(
>>>>>>> RouteContextProcessor.java:45)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.processor.UnitOfWorkProcessor.
>>>>>>> processAsync(
>>>>>>> UnitOfWorkProcessor.java:150)
>>>>>>> at org.apache.camel.processor.
>>>>>>> UnitOfWorkProcessor.process(
>>>>>>> UnitOfWorkProcessor.java:117)
>>>>>>> at org.apache.camel.processor.
>>>>>>> RouteInflightRepositoryProcess
>>>>>>> or.processNext(RouteInflightRepositoryProcessor.java:48)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> processNext(
>>>>>>> DelegateAsyncProcessor.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.management.InstrumentationProcessor.
>>>>>>> process(
>>>>>>> InstrumentationProcessor.java:73)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:86)
>>>>>>> at org.apache.camel.component.timer.TimerConsumer.
>>>>>>> sendTimerExchange(TimerConsumer.java:134)
>>>>>>> at org.apache.camel.component.timer.TimerConsumer$1.run(
>>>>>>> TimerConsumer.java:63)
>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:555)
>>>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>>> Caused by: javax.jms.JMSException: Channel was inactive for too
>>>>>>> (>30000)
>>>>>>> long: tcp://192.168.10.144:61617
>>>>>>> at org.apache.activemq.util.JMSExceptionSupport.create(
>>>>>>> JMSExceptionSupport.java:49)
>>>>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket(
>>>>>>> ActiveMQConnection.java:1377)
>>>>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket(
>>>>>>> ActiveMQConnection.java:1305)
>>>>>>> at org.apache.activemq.ActiveMQSession.send(
>>>>>>> ActiveMQSession.java:1795)
>>>>>>> at org.apache.activemq.ActiveMQMessageProducer.send(
>>>>>>> ActiveMQMessageProducer.java:277)
>>>>>>> at org.apache.activemq.ActiveMQMessageProducer.send(
>>>>>>> ActiveMQMessageProducer.java:212)
>>>>>>> at org.apache.activemq.pool.PooledProducer.send(
>>>>>>> PooledProducer.java:79)
>>>>>>> at org.apache.activemq.pool.PooledProducer.send(
>>>>>>> PooledProducer.java:62)
>>>>>>> at org.springframework.jms.core.JmsTemplate.doSend(
>>>>>>> JmsTemplate.java:592)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.doSend(JmsConfiguration.java:276)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.access$100(JmsConfiguration.java:157)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191)
>>>>>>> at org.springframework.jms.core.JmsTemplate.execute(
>>>>>>> JmsTemplate.java:466)
>>>>>>> ... 41 more
>>>>>>> Caused by: org.apache.activemq.transport.InactivityIOException:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> at org.apache.activemq.transport.
>>>>>>> AbstractInactivityMonitor$4.
>>>>>>> run(AbstractInactivityMonitor.java:209)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 22 Mar 2015 08:43:19,482 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:44:59,642 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:46:10,729 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:48:05,731 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:49:49,723 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Please help me to find out root cause of this and how can I kill a
>>>>>>> job
>>>>>>> or
>>>>>>> start a new one when any job gets stuck after a certain time..
>>>>>>>
>>>>>>> --
>>>>>>> Thanks and Regards,
>>>>>>> Satya Nand Kanodia
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>> Thanks and Regards,
>>>>> Satya Nand Kanodia
>>>>>
>>>>>
>>>>>
>>>>> --
>>> Thanks and Regards,
>>> Satya Nand Kanodia
>>>
>>>
>>>
> --
> Thanks and Regards,
> Satya Nand Kanodia
>
>