You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "lakshmi.prashant" <la...@sap.com> on 2013/09/22 11:59:46 UTC
Camel quartz misfires and route not getting run / triggered after
exception:ObjectAlreadyExistsException
Hi, I am running camel quartz (2.10.4) & quartz has been set-up in clustered
mode. The clocks in the cluster are synchronized. I have set-up a trigger
to run every 1 minute, via the camel-quartz end-point in my route. a) it
works fine if quartz is not set up in clustered mode (uses RAMJobstore). b)
If quartz is set up in clustred mode, it was working fine when there were
only 2 nodes in the cluster. Sometimes, instead of every 1 minute, the
runs used to get fired every 2 minutes over a time-window, but it seemed to
stabilize and run fine. c) After I increased the number of VM's in the
cluster (to ~7 VM's), I faced misfires.. i) The routes used to be
triggered correctly for sometime and then stop working altogether.. ii)
The below exception has been logged in 1 (or) more VM's in the cluster,
after which the camel endpoints seemed to not fire at all. PFA the
depiction of misfires from camel.Note: This issue is noticed in 2.11 camel
as well. Is this related to
https://issues.apache.org/jira/browse/CAMEL-5994... And has this not been
fixed till now?Kindly help.Thanks,LakshmiException
Trace:ERROR#org.apache.camel.blueprint.BlueprintCamelContext##anonymous#Blueprint
Extender: 3##avatarcl#aq4appaq4t#iflmap#null#null#Error occurred during
starting Camel: CamelContext(context4) due Unable to store Trigger with
name: 'schedule3' and group: 'Camel', because one already exists with this
identification.org.quartz.ObjectAlreadyExistsException: Unable to store
Trigger with name: 'schedule3' and group: 'Camel', because one already
exists with this identification.at
org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1200)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$3.execute(JobStoreSupport.java:1072)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:3716)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3788)
at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:90)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInLock(JobStoreSupport.java:3712)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJobAndTrigger(JobStoreSupport.java:1059)
at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:822) at
org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243) at
org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232)
at
org.apache.camel.component.quartz.QuartzComponent.addJob(QuartzComponent.java:222)
at
org.apache.camel.component.quartz.QuartzEndpoint.addTrigger(QuartzEndpoint.java:81)
at
org.apache.camel.component.quartz.QuartzEndpoint.consumerStarted(QuartzEndpoint.java:213)
at
org.apache.camel.component.quartz.QuartzConsumer.doStart(QuartzConsumer.java:39)
at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) at
org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:1819)
at
org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:2113)
at
org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:2049)
at
org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:1979)
at
org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:1758)
at
org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:1633)
at
org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:1500)
<http://camel.465427.n5.nabble.com/file/n5739997/FailedRuns.jpg>
<http://camel.465427.n5.nabble.com/file/n5739997/runs_successful_with_2_VMs.jpg>
<http://camel.465427.n5.nabble.com/file/n5739997/some_runs_misfiring_fire_in_2_mins.jpg>
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-quartz-misfires-and-route-not-getting-run-triggered-after-exception-ObjectAlreadyExistsExceptin-tp5739997.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel quartz misfires and route not getting run / triggered
after exception:ObjectAlreadyExistsException
Posted by Babak Vahdat <ba...@swissonline.ch>.
Hi
As you've already reported by your first post, only increasing the number of
running VM's makes this exception to happen (from 2 to 7). This CAN be a
hint for a race-condition somewhere. There was a similar problem being
reported while ago:
https://issues.apache.org/jira/browse/CAMEL-6686
The only reason why we could fix that was because we could reproduce the
same exception inside a test, which is now added to the code base of this
Camel module. In your case I was NOT able to do the same. If you could
somehow provide a test or a sample project then we could dig into this to
see if we can find the root cause of the problem and probably fix it.
There's also the following Camel test module you could make use of for this:
http://camel.apache.org/blueprint-testing.html
Babak
lakshmi.prashant wrote
> Hi,
>
> Can you kindly explain why the route starts failing after running for
> sometime..
> Why is the doAddJob() getting called, after few runs of a schedule have
> already tun, as reported in the exception?
> The race condition can happen only at the start of the route, while
> scheduling the quartz job. But it gets reported after few runs in the
> below exception trace:
<http://camel.465427.n5.nabble.com/file/n5743232/Route_Stoping_After_fewruns.png>
>
> Thanks,
> Lakshmi
>
> Exception Trace:
> ERROR#org.apache.camel.blueprint.BlueprintCamelContext##anonymous#Blueprint
> Extender: 3##avatarcl#aq4appaq4t#iflmap#null#null#
> Error occurred during starting Camel: CamelContext(context4) due
> Unable to store Trigger with name: 'schedule3' and group: 'Camel', because
> one already exists with this
> identification.org.quartz.ObjectAlreadyExistsException:
> Unable to store Trigger with name: 'schedule3' and group: 'Camel', because
> one already exists with this identification. at
> org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1200)
> at
> org.quartz.impl.jdbcjobstore.JobStoreSupport$3.execute(JobStoreSupport.java:1072)
> at
> org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:3716)
> at
> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3788)
> at
> org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:90)
> at
> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInLock(JobStoreSupport.java:3712)
> at
> org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJobAndTrigger(JobStoreSupport.java:1059)
> at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:822)
> at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243) at
> org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232)
> at
> org.apache.camel.component.quartz.QuartzComponent.addJob(QuartzComponent.java:222)
> at
> org.apache.camel.component.quartz.QuartzEndpoint.addTrigger(QuartzEndpoint.java:81)
> at
> org.apache.camel.component.quartz.QuartzEndpoint.consumerStarted(QuartzEndpoint.java:213)
> at
> org.apache.camel.component.quartz.QuartzConsumer.doStart(QuartzConsumer.java:39)
> at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
> at
> org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:1819)
> at
> org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:2113)
> at
> org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:2049)
> at
> org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:1979)
> at
> org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:1758)
> at
> org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:1633)
> at
> org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:1500)
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-quartz-misfires-and-route-not-getting-run-triggered-after-exception-ObjectAlreadyExistsExceptin-tp5739997p5743238.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel quartz misfires and route not getting run / triggered
after exception:ObjectAlreadyExistsException
Posted by "lakshmi.prashant" <la...@sap.com>.
Hi,
Can you kindly explain why the route starts failing after running for
sometime..
Why is the doAddJob() getting called, after few runs of a schedule have
already tun, as reported in the exception?
The race condition can happen only at the start of the route, while
scheduling the quartz job. But it gets reported after few runs in the below
exception trace:
<http://camel.465427.n5.nabble.com/file/n5743232/Route_Stoping_After_fewruns.png>
Thanks,
Lakshmi
Exception Trace:
ERROR#org.apache.camel.blueprint.BlueprintCamelContext##anonymous#Blueprint
Extender: 3##avatarcl#aq4appaq4t#iflmap#null#null#
Error occurred during starting Camel: CamelContext(context4) due
Unable to store Trigger with name: 'schedule3' and group: 'Camel', because
one already exists with this
identification.org.quartz.ObjectAlreadyExistsException:
Unable to store Trigger with name: 'schedule3' and group: 'Camel', because
one already exists with this identification. at
org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1200)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$3.execute(JobStoreSupport.java:1072)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:3716)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3788)
at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:90)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInLock(JobStoreSupport.java:3712)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJobAndTrigger(JobStoreSupport.java:1059)
at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:822)
at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243) at
org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232)
at
org.apache.camel.component.quartz.QuartzComponent.addJob(QuartzComponent.java:222)
at
org.apache.camel.component.quartz.QuartzEndpoint.addTrigger(QuartzEndpoint.java:81)
at
org.apache.camel.component.quartz.QuartzEndpoint.consumerStarted(QuartzEndpoint.java:213)
at
org.apache.camel.component.quartz.QuartzConsumer.doStart(QuartzConsumer.java:39)
at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) at
org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:1819)
at
org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:2113)
at
org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:2049)
at
org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:1979)
at
org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:1758)
at
org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:1633)
at
org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:1500)
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-quartz-misfires-and-route-not-getting-run-triggered-after-exception-ObjectAlreadyExistsExceptin-tp5739997p5743232.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel quartz misfires and route not getting run / triggered
after exception:ObjectAlreadyExistsException
Posted by Babak Vahdat <ba...@swissonline.ch>.
Hi
Thanks for getting back. Your Quartz configuration seems fine.
To me this sounds like a bug in Quartz itself as Quartz claims that the
trigger is null/non-existing but when we try to schedule the Job using that
given trigger it blows up with ObjectAlreadyExistsException.
That said there's a race condition between the time window as we ask for a
pre-existing trigger and the point where we try to schedule the Job. What if
in this time window another JVM (a cluster member) schedules a Job using the
same trigger name/group. IMHO what actually would come handy here would be a
*atomic* Quartz-API for this, something like what we already have in Java
itself:
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ConcurrentHashMap.html#putIfAbsent(K,%20V)
Which as it's Javadoc already says is equivalent to:
if (!map.containsKey(key))
return map.put(key, value);
else
return map.get(key);
Except that the action is performed *atomically*.
Maybe you want to ask for help using the Quartz user-forum itself and refer
to this thread?
Babak
lakshmi.prashant wrote
> Hi,
>
> Sorry for the delay in response. I did not get notified in my mail box
> and hence had not followed your reply to the mail thread.
>
> Please find below my answers and the blueprint xml used in the cluster
> with 7 VM's.
>
> Thanks,
> Lakshmi
>
>
> - Which exact Quartz 1.x version do you make use of? :
/
> Ans: I am using servicemix quartz 1.8.6_1 bundle.
/
>
> - Can you reproduce this with the latest Camel 2.12.1 version?
/
> Ans: Yes. It happens in big clusters. Has any fix been provided with camel
> 2.12.1?
/
>
> - And CAMEL-5994 is already fixed so this should not cause any problem for
> you
> - Also looking at the stacktrace I think Quartz has not been setup
> properly in Cluster-Mode (see
> org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232)
> on the call-stack)... Did you set the org.quartz.jobStore.isClustered and
> org.quartz.jobStore.clusterCheckinInterval properties appropriately?
/
> Ans: Please find the blueprint XML that I had used. I have set isClustered
> as true and the logs showed that the route got triggered by quartz in
> different cluster VM's for different runs, till the exception got
> reported. But I was not sure of the right value for the checkin interval
> in the cluster and had not specified any specific value.
/
>
> - As you run inside OSGi but assign the id of
> <camelContext>
> by yourself (e.g. context4), have you seen the blue-box about this here
> http://camel.apache.org/quartz
/
> Ans: I am assigning an unique id for each bundle with the camel route. As
> of now, I had tested with only 1 bundle deployed in multiple cluster nodes
> & hence I guess that the context id should not be a problem.
>
> I will also check by removing the context id, as well.
/
>
>
> I had a look at the quartz component source code: The error trace shows
> that existingTrigger is null. It looks like that a race condition is
> occuring between the different VM's. Both of them are likely getting the
> trigger name as null and try to schedule the job..but the job is already
> scheduled
>
> private void doAddJob(JobDetail job, Trigger trigger) throws
> SchedulerException {
> incrementJobCounter(getScheduler());
>
> Trigger existingTrigger =
> getScheduler().getTrigger(trigger.getName(), trigger.getGroup());
> if (existingTrigger == null) {
> LOG.debug("Adding job using trigger: {}/{}",
> trigger.getGroup(), trigger.getName());
>
*
> getScheduler().scheduleJob(job, trigger);
*
> } else if (hasTriggerChanged(existingTrigger, trigger)) {
> LOG.debug("Trigger: {}/{} already exists and will be updated
> by Quartz.", trigger.getGroup(), trigger.getName());
> // fast forward start time to now, as we do not want any
> misfire to kick in
> trigger.setStartTime(new Date());
> // replace job, and relate trigger to previous job name, which
> is needed to reschedule job
> scheduler.addJob(job, true);
> trigger.setJobName(existingTrigger.getJobName());
> scheduler.rescheduleJob(trigger.getName(), trigger.getGroup(),
> trigger);
> } else {
> if (!isClustered()) {
> LOG.debug("Trigger: {}/{} already exists and will be
> resumed by Quartz.", trigger.getGroup(), trigger.getName());
> // fast forward start time to now, as we do not want any
> misfire to kick in
> trigger.setStartTime(new Date());
> // replace job, and relate trigger to previous job name,
> which is needed to reschedule job
> scheduler.addJob(job, true);
> trigger.setJobName(existingTrigger.getJobName());
> scheduler.rescheduleJob(trigger.getName(),
> trigger.getGroup(), trigger);
> } else {
> LOG.debug("Trigger: {}/{} already exists and is already
> scheduled by clustered JobStore.", trigger.getGroup(), trigger.getName());
> }
> }
> }
>
> ____________________________________________________________________________________
>
*
> Blueprint XML used by me:
*
>
> <?xml version="1.0" encoding="UTF-8"?>
> <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
>
> xmlns:Generator="com.sap.it.gnb.ifl.common.gen.pluggability.api.GeneratorHandler"
> xmlns:str="http://exslt.org/strings" xmlns:exsl="http://exslt.org/common"
> xmlns:ext="http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0"
> xmlns:idocsoap="urn:sap-com:document:sap:idoc:soap:messages"
> xmlns:bsn="http://sapcd.com/bsnagt"
> xmlns:ns="https://bsnschemas.netweaver.neo.com/bsnflow"
> xmlns:jaxws="http://cxf.apache.org/blueprint/jaxws"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:sec="http://cxf.apache.org/configuration/security"
> xmlns:wsrm-policy="http://schemas.xmlsoap.org/ws/2005/02/rm/policy"
> xmlns:wsrm-mgr="http://cxf.apache.org/ws/rm/manager"
> xmlns:wsa="http://cxf.apache.org/ws/addressing"
> xmlns:http-conf="http://cxf.apache.org/transports/http/configuration"
> xmlns:cxfcore="http://cxf.apache.org/blueprint/core"
> xmlns:cxf="http://camel.apache.org/schema/blueprint/cxf"
> xmlns:camel="http://camel.apache.org/schema/blueprint"
> xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0
> http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
> http://camel.apache.org/schema/blueprint
> http://camel.apache.org/schema/blueprint/camel-blueprint.xsd
> http://camel.apache.org/schema/blueprint/cxf
> http://camel.apache.org/schema/blueprint/cxf/camel-cxf.xsd
> http://cxf.apache.org/blueprint/core
> http://cxf.apache.org/schemas/blueprint/core.xsd
> http://cxf.apache.org/transports/http/configuration
> http://cxf.apache.org/schemas/configuration/http-conf.xsd
> http://cxf.apache.org/ws/rm/manager
> http://cxf.apache.org/schemas/configuration/wsrm-manager.xsd
> http://schemas.xmlsoap.org/ws/2005/02/rm/policy
> http://schemas.xmlsoap.org/ws/2005/02/rm/wsrm-policy.xsd
> http://cxf.apache.org/configuration/security
> http://cxf.apache.org/schemas/configuration/security.xsd
> http://cxf.apache.org/blueprint/jaxws
> http://cxf.apache.org/schemas/blueprint/jaxws.xsd
> http://camel.apache.org/schema/spring
> http://camel.apache.org/schema/spring/camel-spring.xsd">
>
>
>
> <reference id="dataSource" interface="javax.sql.DataSource"
> filter="(dataSourceName=default)" />
>
> <bean id="stdSchedulerFactory"
> class="org.quartz.impl.StdSchedulerFactory">
>
> <argument type="java.util.Properties" >
>
> <props>
>
> <prop key="org.quartz.jobStore.class">
> org.quartz.impl.jdbcjobstore.JobStoreTX
> </prop>
>
> <prop key="org.quartz.dataSource.default.jndiURL">
> osgi:service/javax.sql.DataSource/(dataSourceName=default)
> </prop>
>
> <prop key="org.quartz.jobStore.dataSource">
> default
> </prop>
>
>
> <prop key="org.quartz.scheduler.instanceId">
> AUTO
> </prop>
>
> <prop key="org.quartz.jobStore.misfireThreshold">
> 60000
> </prop>
>
> <prop key="org.quartz.jobStore.class">
> org.quartz.impl.jdbcjobstore.JobStoreTX
> </prop>
>
> <prop key="org.quartz.jobStore.driverDelegateClass">
> org.quartz.impl.jdbcjobstore.StdJDBCDelegate
> </prop>
>
> <prop key="org.quartz.dataSource.default.jndiAlwaysLookup">
> false
> </prop>
>
>
> <prop key="org.quartz.scheduler.skipUpdateCheck">
> true
> </prop>
>
> <prop key="org.quartz.jobStore.isClustered">
> true
> </prop>
>
> <prop key="org.quartz.threadPool.class">
> org.quartz.simpl.SimpleThreadPool
> </prop>
>
> <prop key="org.quartz.threadPool.threadCount">
> 50
> </prop>
>
> <prop key="org.quartz.plugin.triggHistory.class">
> org.quartz.plugins.history.LoggingTriggerHistoryPlugin
> </prop>
>
> <prop key="org.quartz.plugin.triggHistory.triggerFiredMessage">
> Trigger {1}.{0} fired job {6}.{5} at {4,date,yyyy-MM-dd HH:mm:ss}
> </prop>
>
> <prop key="org.quartz.plugin.triggHistory.triggerCompleteMessage">
> Trigger {1}.{0} completed firing job {6}.{5} at
> {4,date, yyyy-MM-dd HH:mm:ss} with resulting trigger instructioncode
> {9}
>
> </prop>
>
>
> </props>
>
> </argument>
>
> </bean>
>
>
> <bean id="quartz"
> class="org.apache.camel.component.quartz.QuartzComponent">
>
>
> <property name="factory" ref="stdSchedulerFactory"/>
>
> </bean>
>
>
> <bean id="mockProcessor" class="com.sap.test.route.TestProcessor" />
>
>
>
> <camelContext id="context4"
> xmlns="http://camel.apache.org/schema/blueprint" >
>
>
> <route>
>
> <from uri="quartz://schedule3?cron=0+0/1+0-23+*+*+?+*" />
>
> <process ref="mockProcessor" />
>
>
> <camel:to uri="file:data/outbox" />
>
> </route>
> </camelContext>
>
> </blueprint>
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-quartz-misfires-and-route-not-getting-run-triggered-after-exception-ObjectAlreadyExistsExceptin-tp5739997p5741637.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel quartz misfires and route not getting run / triggered
after exception:ObjectAlreadyExistsException
Posted by "lakshmi.prashant" <la...@sap.com>.
Hi,
Sorry for the delay in response. I did not get notified in my mail box and
hence had not followed your reply to the mail thread.
Please find below my answers and the blueprint xml used in the cluster with
7 VM's.
Thanks,
Lakshmi
- Which exact Quartz 1.x version do you make use of? :
/Ans: I am using servicemix quartz 1.8.6_1 bundle./
- Can you reproduce this with the latest Camel 2.12.1 version?
/Ans: Yes. It happens in big clusters. Has any fix been provided with camel
2.12.1?/
- And CAMEL-5994 is already fixed so this should not cause any problem for
you
- Also looking at the stacktrace I think Quartz has not been setup properly
in Cluster-Mode (see
org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232)
on the call-stack)... Did you set the org.quartz.jobStore.isClustered and
org.quartz.jobStore.clusterCheckinInterval properties appropriately?
/Ans: Please find the blueprint XML that I had used. I have set isClustered
as true and the logs showed that the route got triggered by quartz in
different cluster VM's for different runs, till the exception got reported.
But I was not sure of the right value for the checkin interval in the
cluster and had not specified any specific value./
- As you run inside OSGi but assign the id of <camelContext> by yourself
(e.g. context4), have you seen the blue-box about this here
http://camel.apache.org/quartz
/Ans: I am assigning an unique id for each bundle with the camel route. As
of now, I had tested with only 1 bundle deployed in multiple cluster nodes &
hence I guess that the context id should not be a problem.
I will also check by removing the context id, as well.
/
I had a look at the quartz component source code: The error trace shows
that existingTrigger is null. It looks like that a race condition is
occuring between the different VM's. Both of them are likely getting the
trigger name as null and try to schedule the job..but the job is already
scheduled
private void doAddJob(JobDetail job, Trigger trigger) throws
SchedulerException {
incrementJobCounter(getScheduler());
Trigger existingTrigger =
getScheduler().getTrigger(trigger.getName(), trigger.getGroup());
if (existingTrigger == null) {
LOG.debug("Adding job using trigger: {}/{}", trigger.getGroup(),
trigger.getName());
*getScheduler().scheduleJob(job, trigger);*
} else if (hasTriggerChanged(existingTrigger, trigger)) {
LOG.debug("Trigger: {}/{} already exists and will be updated by
Quartz.", trigger.getGroup(), trigger.getName());
// fast forward start time to now, as we do not want any misfire
to kick in
trigger.setStartTime(new Date());
// replace job, and relate trigger to previous job name, which
is needed to reschedule job
scheduler.addJob(job, true);
trigger.setJobName(existingTrigger.getJobName());
scheduler.rescheduleJob(trigger.getName(), trigger.getGroup(),
trigger);
} else {
if (!isClustered()) {
LOG.debug("Trigger: {}/{} already exists and will be resumed
by Quartz.", trigger.getGroup(), trigger.getName());
// fast forward start time to now, as we do not want any
misfire to kick in
trigger.setStartTime(new Date());
// replace job, and relate trigger to previous job name,
which is needed to reschedule job
scheduler.addJob(job, true);
trigger.setJobName(existingTrigger.getJobName());
scheduler.rescheduleJob(trigger.getName(),
trigger.getGroup(), trigger);
} else {
LOG.debug("Trigger: {}/{} already exists and is already
scheduled by clustered JobStore.", trigger.getGroup(), trigger.getName());
}
}
}
____________________________________________________________________________________
*Blueprint XML used by me:*
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:Generator="com.sap.it.gnb.ifl.common.gen.pluggability.api.GeneratorHandler"
xmlns:str="http://exslt.org/strings" xmlns:exsl="http://exslt.org/common"
xmlns:ext="http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0"
xmlns:idocsoap="urn:sap-com:document:sap:idoc:soap:messages"
xmlns:bsn="http://sapcd.com/bsnagt"
xmlns:ns="https://bsnschemas.netweaver.neo.com/bsnflow"
xmlns:jaxws="http://cxf.apache.org/blueprint/jaxws"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:sec="http://cxf.apache.org/configuration/security"
xmlns:wsrm-policy="http://schemas.xmlsoap.org/ws/2005/02/rm/policy"
xmlns:wsrm-mgr="http://cxf.apache.org/ws/rm/manager"
xmlns:wsa="http://cxf.apache.org/ws/addressing"
xmlns:http-conf="http://cxf.apache.org/transports/http/configuration"
xmlns:cxfcore="http://cxf.apache.org/blueprint/core"
xmlns:cxf="http://camel.apache.org/schema/blueprint/cxf"
xmlns:camel="http://camel.apache.org/schema/blueprint"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0
http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
http://camel.apache.org/schema/blueprint
http://camel.apache.org/schema/blueprint/camel-blueprint.xsd
http://camel.apache.org/schema/blueprint/cxf
http://camel.apache.org/schema/blueprint/cxf/camel-cxf.xsd
http://cxf.apache.org/blueprint/core
http://cxf.apache.org/schemas/blueprint/core.xsd
http://cxf.apache.org/transports/http/configuration
http://cxf.apache.org/schemas/configuration/http-conf.xsd
http://cxf.apache.org/ws/rm/manager
http://cxf.apache.org/schemas/configuration/wsrm-manager.xsd
http://schemas.xmlsoap.org/ws/2005/02/rm/policy
http://schemas.xmlsoap.org/ws/2005/02/rm/wsrm-policy.xsd
http://cxf.apache.org/configuration/security
http://cxf.apache.org/schemas/configuration/security.xsd
http://cxf.apache.org/blueprint/jaxws
http://cxf.apache.org/schemas/blueprint/jaxws.xsd
http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd">
<reference id="dataSource" interface="javax.sql.DataSource"
filter="(dataSourceName=default)" />
<bean id="stdSchedulerFactory" class="org.quartz.impl.StdSchedulerFactory">
<argument type="java.util.Properties" >
<props>
<prop
key="org.quartz.jobStore.class">org.quartz.impl.jdbcjobstore.JobStoreTX</prop>
<prop
key="org.quartz.dataSource.default.jndiURL">osgi:service/javax.sql.DataSource/(dataSourceName=default)</prop>
<prop key="org.quartz.jobStore.dataSource">default</prop>
<prop key="org.quartz.scheduler.instanceId">AUTO</prop>
<prop key="org.quartz.jobStore.misfireThreshold">60000</prop>
<prop
key="org.quartz.jobStore.class">org.quartz.impl.jdbcjobstore.JobStoreTX</prop>
<prop
key="org.quartz.jobStore.driverDelegateClass">org.quartz.impl.jdbcjobstore.StdJDBCDelegate</prop>
<prop key="org.quartz.dataSource.default.jndiAlwaysLookup">false</prop>
<prop key="org.quartz.scheduler.skipUpdateCheck">true</prop>
<prop key="org.quartz.jobStore.isClustered">true</prop>
<prop
key="org.quartz.threadPool.class">org.quartz.simpl.SimpleThreadPool</prop>
<prop key="org.quartz.threadPool.threadCount">50</prop>
<prop
key="org.quartz.plugin.triggHistory.class">org.quartz.plugins.history.LoggingTriggerHistoryPlugin
</prop>
<prop key="org.quartz.plugin.triggHistory.triggerFiredMessage">Trigger
{1}.{0} fired job {6}.{5} at {4,date,yyyy-MM-dd HH:mm:ss} </prop>
<prop
key="org.quartz.plugin.triggHistory.triggerCompleteMessage">Trigger {1}.{0}
completed firing job {6}.{5} at
{4,date, yyyy-MM-dd HH:mm:ss} with resulting trigger instructioncode
{9}
</prop>
</props>
</argument>
</bean>
<bean id="quartz"
class="org.apache.camel.component.quartz.QuartzComponent">
<property name="factory" ref="stdSchedulerFactory"/>
</bean>
<bean id="mockProcessor" class="com.sap.test.route.TestProcessor" />
<camelContext id="context4"
xmlns="http://camel.apache.org/schema/blueprint" >
<route>
<from uri="quartz://schedule3?cron=0+0/1+0-23+*+*+?+*" />
<process ref="mockProcessor" />
<camel:to uri="file:data/outbox" />
</route>
</camelContext>
</blueprint>
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-quartz-misfires-and-route-not-getting-run-triggered-after-exception-ObjectAlreadyExistsExceptin-tp5739997p5741632.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel quartz misfires and route not getting run / triggered
after exception:ObjectAlreadyExistsException
Posted by Babak Vahdat <ba...@swissonline.ch>.
Hi
Were you able to find the root cause of the problem here? If not:
- Which exact Quartz 1.x version do you make use of?
- Can you reproduce this with the latest Camel 2.12.1 version?
- And CAMEL-5994 is already fixed so this should not cause any problem for
you
- Also looking at the stacktrace I think Quartz has not been setup properly
in Cluster-Mode (see
org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232)
on the call-stack)... Did you set the org.quartz.jobStore.isClustered and
org.quartz.jobStore.clusterCheckinInterval properties appropriately?
- As you run inside OSGi but assign the id of <camelContext> by yourself
(e.g. context4), have you seen the blue-box about this here
http://camel.apache.org/quartz
Babak
lakshmi.prashant wrote
> Hi,
>
> I am running camel quartz (2.10.4) & quartz has been set-up in clustered
> mode. The clocks in the cluster are synchronized.
>
> I have set-up a trigger to run every 1 minute, via the camel-quartz
> end-point in my route.
>
> a) it works fine if quartz is not set up in clustered mode (uses
> RAMJobstore).
>
> b) If quartz is set up in clustred mode, it was working fine when there
> were only 2 nodes in the cluster.
> Sometimes, instead of every 1 minute, the runs used to get fired
> every 2 minutes over a time-window, but it seemed to stabilize and run
> fine.
>
> c) After I increased the number of VM's in the cluster (to ~7 VM's), I
> faced misfires..
>
> i) The routes used to be triggered correctly for sometime and then
> stop working altogether..
> ii) The below exception has been logged in 1 (or) more VM's in the
> cluster, after which the camel endpoints seemed to not fire at all. PFA
> the depiction of misfires from camel.
>
> Note: This issue is noticed in 2.11 camel as well. Is this related to
> https://issues.apache.org/jira/browse/CAMEL-5994... And has this not been
> fixed till now?
>
> Kindly help.
>
> Thanks,
> Lakshmi
>
> Exception Trace:
>
> ERROR#org.apache.camel.blueprint.BlueprintCamelContext##anonymous#Blueprint
> Extender: 3##avatarcl#aq4appaq4t#iflmap#null#null#Error occurred during
> starting Camel: CamelContext(context4) due Unable to store Trigger with
> name: 'schedule3' and group: 'Camel', because one already exists with this
> identification.org.quartz.ObjectAlreadyExistsException: Unable to store
> Trigger with name: 'schedule3' and group: 'Camel', because one already
> exists with this identification.
> at
> org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1200)
> at
> org.quartz.impl.jdbcjobstore.JobStoreSupport$3.execute(JobStoreSupport.java:1072)
> at
> org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:3716)
> at
> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3788)
> at
> org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:90)
> at
> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInLock(JobStoreSupport.java:3712)
> at
> org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJobAndTrigger(JobStoreSupport.java:1059)
> at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:822)
> at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243)
> at
> org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232)
> at
> org.apache.camel.component.quartz.QuartzComponent.addJob(QuartzComponent.java:222)
> at
> org.apache.camel.component.quartz.QuartzEndpoint.addTrigger(QuartzEndpoint.java:81)
> at
> org.apache.camel.component.quartz.QuartzEndpoint.consumerStarted(QuartzEndpoint.java:213)
> at
> org.apache.camel.component.quartz.QuartzConsumer.doStart(QuartzConsumer.java:39)
> at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
> at
> org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:1819)
> at
> org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:2113)
> at
> org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:2049)
> at
> org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:1979)
> at
> org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:1758)
> at
> org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:1633)
> at
> org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:1500)
>
>
<http://camel.465427.n5.nabble.com/file/n5739997/FailedRuns.jpg>
<http://camel.465427.n5.nabble.com/file/n5739997/runs_successful_with_2_VMs.jpg>
<http://camel.465427.n5.nabble.com/file/n5739997/some_runs_misfiring_fire_in_2_mins.jpg>
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-quartz-misfires-and-route-not-getting-run-triggered-after-exception-ObjectAlreadyExistsExceptin-tp5739997p5740345.html
Sent from the Camel - Users mailing list archive at Nabble.com.