Camels stop without warning

[davedave <na...@scattered.com.au>]

I'm running Camel (2.14.1) on ServiceMix (5.4.0), and have 21 routes across
two bundles, including 11 timer producing endpoints.

After around 10 hours (~1000 timer messages) of running, camel grinds to a
halt, and only restarting the bundles resolves the issue. I have placed .log
processors between each point in the route, and all running routes seems to
halt at the same time (at various points in the route), with just a few
rouge exchanges making it a few more hops over the following minutes. No
more timers are fired, and in flight exchanges finish their current process
but never make it to the next step in the route.

There is nothing telltale in the logs, even set to DEBUG, everything just
seems to stop, and I just get checkpoint debugs in the servicemix.log from
activemq, until I restart the bundles. Once the bundles get the stop
instruction, they actually begin routing again, but they also begin shutting
down.

I haven't tried running TRACE yet, as those 1000 messages get split in to
many more thousands, and there'd be hundreds of megs of logs every minute
being produced. And with it failing between 8 and 12 hours, there's a 4 hour
window where it would need to be watched closely.

Connecting to the JMX port, I can't see and deadlocked threads.

Any ideas on how best to best debug this?





--
View this message in context: http://camel.465427.n5.nabble.com/Camels-stop-without-warning-tp5765046.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camels stop without warning

[davedave <na...@scattered.com.au>]

Just prior to posting my question, I changed some things with my DB
connection pool, and the system has been running for more than a day. Which
is 100% better than it has run (over the last month).

Could this have been caused by a poorly configured (default) thread pool
profile? In turn, hiding the real problem?



--
View this message in context: http://camel.465427.n5.nabble.com/Camels-stop-without-warning-tp5765046p5765108.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camels stop without warning

[davedave <na...@scattered.com.au>]

These are just a couple, the first is an example of a producing route, the
second it one of the processing routes.

Thanks for looking!




Mark Webb wrote
> Could you post your route definitions?





--
View this message in context: http://camel.465427.n5.nabble.com/Camels-stop-without-warning-tp5765046p5765091.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camels stop without warning

[Mark Webb <el...@gmail.com>]

Could you post your route definitions?


On Tue, Mar 31, 2015 at 12:57 AM, davedave <na...@scattered.com.au> wrote:

> I'm running Camel (2.14.1) on ServiceMix (5.4.0), and have 21 routes across
> two bundles, including 11 timer producing endpoints.
>
> After around 10 hours (~1000 timer messages) of running, camel grinds to a
> halt, and only restarting the bundles resolves the issue. I have placed
> .log
> processors between each point in the route, and all running routes seems to
> halt at the same time (at various points in the route), with just a few
> rouge exchanges making it a few more hops over the following minutes. No
> more timers are fired, and in flight exchanges finish their current process
> but never make it to the next step in the route.
>
> There is nothing telltale in the logs, even set to DEBUG, everything just
> seems to stop, and I just get checkpoint debugs in the servicemix.log from
> activemq, until I restart the bundles. Once the bundles get the stop
> instruction, they actually begin routing again, but they also begin
> shutting
> down.
>
> I haven't tried running TRACE yet, as those 1000 messages get split in to
> many more thousands, and there'd be hundreds of megs of logs every minute
> being produced. And with it failing between 8 and 12 hours, there's a 4
> hour
> window where it would need to be watched closely.
>
> Connecting to the JMX port, I can't see and deadlocked threads.
>
> Any ideas on how best to best debug this?
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Camels-stop-without-warning-tp5765046.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>