You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by jnupponen <ju...@gmail.com> on 2015/07/13 11:18:32 UTC
Quartz2 problem with AsyncDelayedDelivery
Hi!
I've run into a problem with camel-quartz2 (at least on 2.13.2 and 2.15.2)
cron scheduling and AsyncDelayedDelivery-option in the DefaultErrorHandler.
If I want to shutdown/restart routes during DefaultErrorHandlers
redeliveries quartz scheduling gets deadlocked and quietly fails all future
fire times. Without AsyncDelayedDelivery-option this problem does not occur.
I'm running my routes in bundles in Karaf 3.0.3 but the problem also occurs
when running the code on jvm (at least on 1.7) without Karaf.
I have configured quite long redelivery times for my routes (e.g. 8 times in
2 hour interval) but at times I would need to be able to restart bundles
running in Karaf even when there would be redeliveries still left (e.g. when
installing new version of the bundle).
This seems like a bug to me but what's your take on this? I've attached few
lines of log and some example route code to the end of this message.
Best regards,
- Jussi
Logs on org.quartz debug level show every few seconds:
[artzScheduler-camel-1_Worker-2] SimpleThreadPool DEBUG
Waiting for thread DefaultQuartzScheduler-camel-1_Worker-2 to shut down
and jstack shows for DefaultQuartzScheduler-camel-1_Worker-2:
"DefaultQuartzScheduler-camel-1_Worker-2" prio=10 tid=0x00007fd2f8201000
nid=0x2a16 in Object.wait() [0x00007fd3039dc000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007ae410420> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.shutdown(SimpleThreadPool.java:371)
- locked <0x00000007ae410420> (a java.lang.Object)
at org.quartz.core.QuartzScheduler.shutdown(QuartzScheduler.java:740)
at org.quartz.impl.StdScheduler.shutdown(StdScheduler.java:206)
at
org.apache.camel.component.quartz2.QuartzComponent.doStop(QuartzComponent.java:352)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
at org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:141)
at
org.apache.camel.util.ServiceHelper.stopAndShutdownService(ServiceHelper.java:204)
at
org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2827)
at
org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2852)
at
org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2840)
at
org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:2704)
- locked <0x00000007ad1abd88> (a org.apache.camel.impl.DefaultCamelContext)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
at
com.example.route.MyAsyncDelayedDeliveryTesting$1.process(MyAsyncDelayedDeliveryTesting.java:43)
at
org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at
org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:44)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at
org.apache.camel.processor.loadbalancer.LoadBalancerSupport.process(LoadBalancerSupport.java:87)
at org.apache.camel.component.quartz2.CamelJob.execute(CamelJob.java:56)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000007ae539450> (a java.lang.Object)
Example of this situation looks like this:
package com.example.route;
import static org.apache.camel.LoggingLevel.ERROR;
import static org.apache.camel.LoggingLevel.WARN;
import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.joda.time.DateTime;
public class MyAsyncDelayedDeliveryTesting extends RouteBuilder {
@Override
public void configure() throws Exception {
// Set lower timeout for convenience (default 300 seconds).
getContext().getShutdownStrategy().setTimeout(10);
// Start some route1 cron scheduled.
DateTime now = DateTime.now().plusSeconds(2);
from("quartz2://my_route_quartz_timer?cron="
+now.getSecondOfMinute()+"+"
+now.getMinuteOfHour()+"+"
+now.getHourOfDay()+"+*+*+?")
.id("scheduling1")
.to("direct://my-route");
// Start other route that shutdowns the context thus shutting down the
route1.
// Simulate the situation where you want to restart your route when there
are still
// ongoing redelivery tasks.
now = DateTime.now().plusSeconds(4);
from("quartz2://my_shutdown_route_quartz_timer?cron="
+now.getSecondOfMinute()+"+"
+now.getMinuteOfHour()+"+"
+now.getHourOfDay()+"+*+*+?")
.id("scheduling2")
.process(new Processor() {
@Override
public void process(Exchange arg0) throws Exception {
getContext().stop();
}
});
// Do some logic that causes exception and redeliveries for 1 minute.
from("direct://my-route")
.id("route1")
.errorHandler(defaultErrorHandler()
.maximumRedeliveries(10)
.redeliveryDelay(1000*6)
// asyncDelayedRedelivery causes the problem.
.asyncDelayedRedelivery()
.retryAttemptedLogLevel(WARN)
.retriesExhaustedLogLevel(ERROR))
.process(new Processor() {
@Override
public void process(Exchange exchange) throws Exception {
throw new RuntimeException("Error");
}
});
}
}
Full example is visible here:
https://github.com/jnupponen/camel-quartz2-asyncDelayedDelivery-problem
--
View this message in context: http://camel.465427.n5.nabble.com/Quartz2-problem-with-AsyncDelayedDelivery-tp5769210.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Quartz2 problem with AsyncDelayedDelivery
Posted by ravi narayanan <ge...@gmail.com>.
Hi Claus,
When you say about persistent store can you tell me how can I achieve it
using active mq.
Some link where they have implemented will probably help me.
On Tuesday, 14 July 2015, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> Its bad practice to have so long redelvery intervals. Its is intended
> for short time processing. If you want so long then off load the
> messages to some persistent store and process from there.
>
> Though we have a ticket in JIRA to allow Camel's error handler to
> discard those pending redeliveries in can of shutdown etc. But its not
> implemented yet.
>
> On Mon, Jul 13, 2015 at 11:18 AM, jnupponen <jussi.nupponen@gmail.com
> <javascript:;>> wrote:
> > Hi!
> >
> > I've run into a problem with camel-quartz2 (at least on 2.13.2 and
> 2.15.2)
> > cron scheduling and AsyncDelayedDelivery-option in the
> DefaultErrorHandler.
> > If I want to shutdown/restart routes during DefaultErrorHandlers
> > redeliveries quartz scheduling gets deadlocked and quietly fails all
> future
> > fire times. Without AsyncDelayedDelivery-option this problem does not
> occur.
> > I'm running my routes in bundles in Karaf 3.0.3 but the problem also
> occurs
> > when running the code on jvm (at least on 1.7) without Karaf.
> >
> > I have configured quite long redelivery times for my routes (e.g. 8
> times in
> > 2 hour interval) but at times I would need to be able to restart bundles
> > running in Karaf even when there would be redeliveries still left (e.g.
> when
> > installing new version of the bundle).
> >
> > This seems like a bug to me but what's your take on this? I've attached
> few
> > lines of log and some example route code to the end of this message.
> >
> > Best regards,
> > - Jussi
> >
> >
> >
> > Logs on org.quartz debug level show every few seconds:
> > [artzScheduler-camel-1_Worker-2] SimpleThreadPool DEBUG
> > Waiting for thread DefaultQuartzScheduler-camel-1_Worker-2 to shut down
> >
> > and jstack shows for DefaultQuartzScheduler-camel-1_Worker-2:
> > "DefaultQuartzScheduler-camel-1_Worker-2" prio=10 tid=0x00007fd2f8201000
> > nid=0x2a16 in Object.wait() [0x00007fd3039dc000]
> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <0x00000007ae410420> (a java.lang.Object)
> > at
> org.quartz.simpl.SimpleThreadPool.shutdown(SimpleThreadPool.java:371)
> > - locked <0x00000007ae410420> (a java.lang.Object)
> > at
> org.quartz.core.QuartzScheduler.shutdown(QuartzScheduler.java:740)
> > at org.quartz.impl.StdScheduler.shutdown(StdScheduler.java:206)
> > at
> >
> org.apache.camel.component.quartz2.QuartzComponent.doStop(QuartzComponent.java:352)
> > at
> org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
> > at
> org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:141)
> > at
> >
> org.apache.camel.util.ServiceHelper.stopAndShutdownService(ServiceHelper.java:204)
> > at
> >
> org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2827)
> > at
> >
> org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2852)
> > at
> >
> org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2840)
> > at
> >
> org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:2704)
> > - locked <0x00000007ad1abd88> (a
> org.apache.camel.impl.DefaultCamelContext)
> > at
> org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
> > at
> >
> com.example.route.MyAsyncDelayedDeliveryTesting$1.process(MyAsyncDelayedDeliveryTesting.java:43)
> > at
> >
> org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
> > at
> >
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> > at
> >
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
> > at
> >
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> > at
> >
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> > at
> >
> org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:44)
> > at
> >
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
> > at
> >
> org.apache.camel.processor.loadbalancer.LoadBalancerSupport.process(LoadBalancerSupport.java:87)
> > at
> org.apache.camel.component.quartz2.CamelJob.execute(CamelJob.java:56)
> > at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> > at
> >
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> > - locked <0x00000007ae539450> (a java.lang.Object)
> >
> >
> > Example of this situation looks like this:
> >
> > package com.example.route;
> >
> > import static org.apache.camel.LoggingLevel.ERROR;
> > import static org.apache.camel.LoggingLevel.WARN;
> >
> > import org.apache.camel.Exchange;
> > import org.apache.camel.Processor;
> > import org.apache.camel.builder.RouteBuilder;
> > import org.joda.time.DateTime;
> >
> > public class MyAsyncDelayedDeliveryTesting extends RouteBuilder {
> >
> >
> > @Override
> > public void configure() throws Exception {
> >
> > // Set lower timeout for convenience (default 300
> seconds).
> > getContext().getShutdownStrategy().setTimeout(10);
> >
> > // Start some route1 cron scheduled.
> > DateTime now = DateTime.now().plusSeconds(2);
> > from("quartz2://my_route_quartz_timer?cron="
> > +now.getSecondOfMinute()+"+"
> > +now.getMinuteOfHour()+"+"
> > +now.getHourOfDay()+"+*+*+?")
> > .id("scheduling1")
> > .to("direct://my-route");
> >
> >
> > // Start other route that shutdowns the context thus
> shutting down the
> > route1.
> > // Simulate the situation where you want to restart your
> route when there
> > are still
> > // ongoing redelivery tasks.
> > now = DateTime.now().plusSeconds(4);
> > from("quartz2://my_shutdown_route_quartz_timer?cron="
> > +now.getSecondOfMinute()+"+"
> > +now.getMinuteOfHour()+"+"
> > +now.getHourOfDay()+"+*+*+?")
> > .id("scheduling2")
> > .process(new Processor() {
> >
> > @Override
> > public void process(Exchange arg0) throws
> Exception {
> > getContext().stop();
> >
> > }
> > });
> >
> > // Do some logic that causes exception and redeliveries
> for 1 minute.
> > from("direct://my-route")
> > .id("route1")
> > .errorHandler(defaultErrorHandler()
> > .maximumRedeliveries(10)
> > .redeliveryDelay(1000*6)
> > // asyncDelayedRedelivery causes the problem.
> > .asyncDelayedRedelivery()
> > .retryAttemptedLogLevel(WARN)
> > .retriesExhaustedLogLevel(ERROR))
> >
> > .process(new Processor() {
> >
> > @Override
> > public void process(Exchange exchange) throws
> Exception {
> > throw new RuntimeException("Error");
> > }
> > });
> > }
> > }
> >
> > Full example is visible here:
> > https://github.com/jnupponen/camel-quartz2-asyncDelayedDelivery-problem
> >
> >
> >
> > --
> > View this message in context:
> http://camel.465427.n5.nabble.com/Quartz2-problem-with-AsyncDelayedDelivery-tp5769210.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com <javascript:;>
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
>
Re: Quartz2 problem with AsyncDelayedDelivery
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Its bad practice to have so long redelvery intervals. Its is intended
for short time processing. If you want so long then off load the
messages to some persistent store and process from there.
Though we have a ticket in JIRA to allow Camel's error handler to
discard those pending redeliveries in can of shutdown etc. But its not
implemented yet.
On Mon, Jul 13, 2015 at 11:18 AM, jnupponen <ju...@gmail.com> wrote:
> Hi!
>
> I've run into a problem with camel-quartz2 (at least on 2.13.2 and 2.15.2)
> cron scheduling and AsyncDelayedDelivery-option in the DefaultErrorHandler.
> If I want to shutdown/restart routes during DefaultErrorHandlers
> redeliveries quartz scheduling gets deadlocked and quietly fails all future
> fire times. Without AsyncDelayedDelivery-option this problem does not occur.
> I'm running my routes in bundles in Karaf 3.0.3 but the problem also occurs
> when running the code on jvm (at least on 1.7) without Karaf.
>
> I have configured quite long redelivery times for my routes (e.g. 8 times in
> 2 hour interval) but at times I would need to be able to restart bundles
> running in Karaf even when there would be redeliveries still left (e.g. when
> installing new version of the bundle).
>
> This seems like a bug to me but what's your take on this? I've attached few
> lines of log and some example route code to the end of this message.
>
> Best regards,
> - Jussi
>
>
>
> Logs on org.quartz debug level show every few seconds:
> [artzScheduler-camel-1_Worker-2] SimpleThreadPool DEBUG
> Waiting for thread DefaultQuartzScheduler-camel-1_Worker-2 to shut down
>
> and jstack shows for DefaultQuartzScheduler-camel-1_Worker-2:
> "DefaultQuartzScheduler-camel-1_Worker-2" prio=10 tid=0x00007fd2f8201000
> nid=0x2a16 in Object.wait() [0x00007fd3039dc000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000007ae410420> (a java.lang.Object)
> at org.quartz.simpl.SimpleThreadPool.shutdown(SimpleThreadPool.java:371)
> - locked <0x00000007ae410420> (a java.lang.Object)
> at org.quartz.core.QuartzScheduler.shutdown(QuartzScheduler.java:740)
> at org.quartz.impl.StdScheduler.shutdown(StdScheduler.java:206)
> at
> org.apache.camel.component.quartz2.QuartzComponent.doStop(QuartzComponent.java:352)
> at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
> at org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:141)
> at
> org.apache.camel.util.ServiceHelper.stopAndShutdownService(ServiceHelper.java:204)
> at
> org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2827)
> at
> org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2852)
> at
> org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2840)
> at
> org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:2704)
> - locked <0x00000007ad1abd88> (a org.apache.camel.impl.DefaultCamelContext)
> at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
> at
> com.example.route.MyAsyncDelayedDeliveryTesting$1.process(MyAsyncDelayedDeliveryTesting.java:43)
> at
> org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at
> org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:44)
> at
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
> at
> org.apache.camel.processor.loadbalancer.LoadBalancerSupport.process(LoadBalancerSupport.java:87)
> at org.apache.camel.component.quartz2.CamelJob.execute(CamelJob.java:56)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> - locked <0x00000007ae539450> (a java.lang.Object)
>
>
> Example of this situation looks like this:
>
> package com.example.route;
>
> import static org.apache.camel.LoggingLevel.ERROR;
> import static org.apache.camel.LoggingLevel.WARN;
>
> import org.apache.camel.Exchange;
> import org.apache.camel.Processor;
> import org.apache.camel.builder.RouteBuilder;
> import org.joda.time.DateTime;
>
> public class MyAsyncDelayedDeliveryTesting extends RouteBuilder {
>
>
> @Override
> public void configure() throws Exception {
>
> // Set lower timeout for convenience (default 300 seconds).
> getContext().getShutdownStrategy().setTimeout(10);
>
> // Start some route1 cron scheduled.
> DateTime now = DateTime.now().plusSeconds(2);
> from("quartz2://my_route_quartz_timer?cron="
> +now.getSecondOfMinute()+"+"
> +now.getMinuteOfHour()+"+"
> +now.getHourOfDay()+"+*+*+?")
> .id("scheduling1")
> .to("direct://my-route");
>
>
> // Start other route that shutdowns the context thus shutting down the
> route1.
> // Simulate the situation where you want to restart your route when there
> are still
> // ongoing redelivery tasks.
> now = DateTime.now().plusSeconds(4);
> from("quartz2://my_shutdown_route_quartz_timer?cron="
> +now.getSecondOfMinute()+"+"
> +now.getMinuteOfHour()+"+"
> +now.getHourOfDay()+"+*+*+?")
> .id("scheduling2")
> .process(new Processor() {
>
> @Override
> public void process(Exchange arg0) throws Exception {
> getContext().stop();
>
> }
> });
>
> // Do some logic that causes exception and redeliveries for 1 minute.
> from("direct://my-route")
> .id("route1")
> .errorHandler(defaultErrorHandler()
> .maximumRedeliveries(10)
> .redeliveryDelay(1000*6)
> // asyncDelayedRedelivery causes the problem.
> .asyncDelayedRedelivery()
> .retryAttemptedLogLevel(WARN)
> .retriesExhaustedLogLevel(ERROR))
>
> .process(new Processor() {
>
> @Override
> public void process(Exchange exchange) throws Exception {
> throw new RuntimeException("Error");
> }
> });
> }
> }
>
> Full example is visible here:
> https://github.com/jnupponen/camel-quartz2-asyncDelayedDelivery-problem
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Quartz2-problem-with-AsyncDelayedDelivery-tp5769210.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Quartz2 problem with AsyncDelayedDelivery
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Jul 14, 2015 at 2:41 PM, jnupponen <ju...@gmail.com> wrote:
> Hi!
>
> Thanks for the information, I'll take another approach to the long
> redelivery times then. Can you say some number that would still be
> acceptable waiting time in redeliveries?
>
> I don't actually call getContext.stop() in my code but it was in the example
> to demonstrate the same effect that typing bundle:stop in Karaf will have.
> After that camel-quartz2 bundle needs to be restarted since the
> DefaultQuartzScheduler is unable to shutdown.
>
Ah yeah sure that triggers the shutdown from an independent thread. So
there is a little bug in camel-quartz2. You are welcome to log a JIRA
ticket.
> Thanks!
> -Jussi
>
>
>
> Claus Ibsen-2 wrote
>> Hi
>>
>> Its bad practice to have so long redelvery intervals. Its is intended for
>> short time processing. If you want so long then off load the messages to
>> some persistent store and process from there.
>>
>> Though we have a ticket in JIRA to allow Camel's error handler to discard
>> those pending redeliveries in can of shutdown etc. But its not implemented
>> yet.
>
>
> Claus Ibsen-2 wrote
>> Hi
>>
>> Also dont stop Camel from the same thread you process a message. You
>> should spin that off to a separate thread, or use controlbus in async mode
>> etc.
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Quartz2-problem-with-AsyncDelayedDelivery-tp5769210p5769288.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Quartz2 problem with AsyncDelayedDelivery
Posted by jnupponen <ju...@gmail.com>.
Hi!
Thanks for the information, I'll take another approach to the long
redelivery times then. Can you say some number that would still be
acceptable waiting time in redeliveries?
I don't actually call getContext.stop() in my code but it was in the example
to demonstrate the same effect that typing bundle:stop in Karaf will have.
After that camel-quartz2 bundle needs to be restarted since the
DefaultQuartzScheduler is unable to shutdown.
Thanks!
-Jussi
Claus Ibsen-2 wrote
> Hi
>
> Its bad practice to have so long redelvery intervals. Its is intended for
> short time processing. If you want so long then off load the messages to
> some persistent store and process from there.
>
> Though we have a ticket in JIRA to allow Camel's error handler to discard
> those pending redeliveries in can of shutdown etc. But its not implemented
> yet.
Claus Ibsen-2 wrote
> Hi
>
> Also dont stop Camel from the same thread you process a message. You
> should spin that off to a separate thread, or use controlbus in async mode
> etc.
--
View this message in context: http://camel.465427.n5.nabble.com/Quartz2-problem-with-AsyncDelayedDelivery-tp5769210p5769288.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Quartz2 problem with AsyncDelayedDelivery
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Also dont stop Camel from the same thread you process a message. You
should spin that off to a separate thread, or use controlbus in async
mode etc.
On Mon, Jul 13, 2015 at 11:18 AM, jnupponen <ju...@gmail.com> wrote:
> Hi!
>
> I've run into a problem with camel-quartz2 (at least on 2.13.2 and 2.15.2)
> cron scheduling and AsyncDelayedDelivery-option in the DefaultErrorHandler.
> If I want to shutdown/restart routes during DefaultErrorHandlers
> redeliveries quartz scheduling gets deadlocked and quietly fails all future
> fire times. Without AsyncDelayedDelivery-option this problem does not occur.
> I'm running my routes in bundles in Karaf 3.0.3 but the problem also occurs
> when running the code on jvm (at least on 1.7) without Karaf.
>
> I have configured quite long redelivery times for my routes (e.g. 8 times in
> 2 hour interval) but at times I would need to be able to restart bundles
> running in Karaf even when there would be redeliveries still left (e.g. when
> installing new version of the bundle).
>
> This seems like a bug to me but what's your take on this? I've attached few
> lines of log and some example route code to the end of this message.
>
> Best regards,
> - Jussi
>
>
>
> Logs on org.quartz debug level show every few seconds:
> [artzScheduler-camel-1_Worker-2] SimpleThreadPool DEBUG
> Waiting for thread DefaultQuartzScheduler-camel-1_Worker-2 to shut down
>
> and jstack shows for DefaultQuartzScheduler-camel-1_Worker-2:
> "DefaultQuartzScheduler-camel-1_Worker-2" prio=10 tid=0x00007fd2f8201000
> nid=0x2a16 in Object.wait() [0x00007fd3039dc000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000007ae410420> (a java.lang.Object)
> at org.quartz.simpl.SimpleThreadPool.shutdown(SimpleThreadPool.java:371)
> - locked <0x00000007ae410420> (a java.lang.Object)
> at org.quartz.core.QuartzScheduler.shutdown(QuartzScheduler.java:740)
> at org.quartz.impl.StdScheduler.shutdown(StdScheduler.java:206)
> at
> org.apache.camel.component.quartz2.QuartzComponent.doStop(QuartzComponent.java:352)
> at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
> at org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:141)
> at
> org.apache.camel.util.ServiceHelper.stopAndShutdownService(ServiceHelper.java:204)
> at
> org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2827)
> at
> org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2852)
> at
> org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2840)
> at
> org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:2704)
> - locked <0x00000007ad1abd88> (a org.apache.camel.impl.DefaultCamelContext)
> at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
> at
> com.example.route.MyAsyncDelayedDeliveryTesting$1.process(MyAsyncDelayedDeliveryTesting.java:43)
> at
> org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at
> org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:44)
> at
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
> at
> org.apache.camel.processor.loadbalancer.LoadBalancerSupport.process(LoadBalancerSupport.java:87)
> at org.apache.camel.component.quartz2.CamelJob.execute(CamelJob.java:56)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> - locked <0x00000007ae539450> (a java.lang.Object)
>
>
> Example of this situation looks like this:
>
> package com.example.route;
>
> import static org.apache.camel.LoggingLevel.ERROR;
> import static org.apache.camel.LoggingLevel.WARN;
>
> import org.apache.camel.Exchange;
> import org.apache.camel.Processor;
> import org.apache.camel.builder.RouteBuilder;
> import org.joda.time.DateTime;
>
> public class MyAsyncDelayedDeliveryTesting extends RouteBuilder {
>
>
> @Override
> public void configure() throws Exception {
>
> // Set lower timeout for convenience (default 300 seconds).
> getContext().getShutdownStrategy().setTimeout(10);
>
> // Start some route1 cron scheduled.
> DateTime now = DateTime.now().plusSeconds(2);
> from("quartz2://my_route_quartz_timer?cron="
> +now.getSecondOfMinute()+"+"
> +now.getMinuteOfHour()+"+"
> +now.getHourOfDay()+"+*+*+?")
> .id("scheduling1")
> .to("direct://my-route");
>
>
> // Start other route that shutdowns the context thus shutting down the
> route1.
> // Simulate the situation where you want to restart your route when there
> are still
> // ongoing redelivery tasks.
> now = DateTime.now().plusSeconds(4);
> from("quartz2://my_shutdown_route_quartz_timer?cron="
> +now.getSecondOfMinute()+"+"
> +now.getMinuteOfHour()+"+"
> +now.getHourOfDay()+"+*+*+?")
> .id("scheduling2")
> .process(new Processor() {
>
> @Override
> public void process(Exchange arg0) throws Exception {
> getContext().stop();
>
> }
> });
>
> // Do some logic that causes exception and redeliveries for 1 minute.
> from("direct://my-route")
> .id("route1")
> .errorHandler(defaultErrorHandler()
> .maximumRedeliveries(10)
> .redeliveryDelay(1000*6)
> // asyncDelayedRedelivery causes the problem.
> .asyncDelayedRedelivery()
> .retryAttemptedLogLevel(WARN)
> .retriesExhaustedLogLevel(ERROR))
>
> .process(new Processor() {
>
> @Override
> public void process(Exchange exchange) throws Exception {
> throw new RuntimeException("Error");
> }
> });
> }
> }
>
> Full example is visible here:
> https://github.com/jnupponen/camel-quartz2-asyncDelayedDelivery-problem
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Quartz2-problem-with-AsyncDelayedDelivery-tp5769210.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/