You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by selva <sg...@gmail.com> on 2014/11/25 14:43:17 UTC
Quartz clustering in camel spring DSL - JIRA CAMEL-8076
Hi Willem,
I have tested with camel 2.14.1-SNAPSHOT and the exception is gone but
QueryBuilderProcessor is not getting called from quartz end point.
Please find below configuration and logging information for both recovery
and success scenario. In success scenario QueryBuilderProcessor is getting
called from quartz endpoint and able see some system.out.printl statement in
success case but its failing in recovery case.
Please help me to solve this issue.
Route Configuration:
<route id="quartz" trace="true">
<from
uri="quartz2://cluster/quartz?cron=0+0/4+++*+?&durableJob=true&stateful=true&recoverableJob=true">
<to uri="direct:queryProcessor" />//calling sub rout here.
...
</route>
<route id="queryProcessor" startupOrder="1">
<from uri="direct:queryProcessor" />
<to uri="bean:QueryBuilderProcessor" />
</route>
Please find the below logs for Calling QueryBuilderProcessor from quartz
endpoint:
=================================================================================
Case 1 : //Logs for Immediate recovery : failure with recovery
---------------------------------------------------------------
2014-11-25 12:28:47,319 [_ClusterManager] INFO JobStoreTX
- ClusterManager: detected 1 failed or restarted instances.
2014-11-25 12:28:47,319 [_ClusterManager] INFO JobStoreTX
- ClusterManager: Scanning for instance "16898502964"'s failed in-progress
jobs.
2014-11-25 12:28:56,675 [_ClusterManager] INFO JobStoreTX
- ClusterManager: ......Scheduled 1 recoverable job(s) for recovery.
2014-11-25 12:29:08,599 [ontext_Worker-1] INFO LoggingTriggerHistoryPlugin
- Trigger [RECOVERING_JOBS.recover_16898502964_1416898727319] fired job
[cluster.quartz] scheduled at: 25-11-2014 12:28:00.000, next scheduled at:
null
2014-11-25 12:29:08,600 [ontext_Worker-1] INFO LoggingJobHistoryPlugin
- Job [cluster.quartz] to be fired by trigger
[RECOVERING_JOBS.recover_16898502964_1416898727319], re-fire: 0
2014-11-25 12:29:08,604 [ontext_Worker-1] WARN CamelJob
- Cannot find existing QuartzEndpoint with uri:
quartz2://cluster/quartz?cron=0+0%2F4+*+*+*+%3F&recoverableJob=true&stateful=true.
Creating new endpoint instance.
2014-11-25 12:29:09,514 [ontext_Worker-1] INFO QuartzEndpoint
- Job cluster.quartz (triggerType=CronTriggerImpl,
jobClass=StatefulCamelJob) is scheduled. Next fire date is null
2014-11-25 12:29:09,526 [ontext_Worker-1] INFO LoggingJobHistoryPlugin
- Job [cluster.quartz] execution complete and reports: null
2014-11-25 12:29:09,528 [ontext_Worker-1] INFO LoggingTriggerHistoryPlugin
- Trigger [RECOVERING_JOBS.recover_16898502964_1416898727319] completed
firing job [cluster.quartz] with resulting trigger instruction code: DELETE
TRIGGER. Next scheduled at: null
case 2: //Normal flow : success
-------------------------------
2014-11-25 12:32:04,901 [ontext_Worker-2] INFO LoggingTriggerHistoryPlugin
- Trigger [cluster.quartz] fired job [cluster.quartz] scheduled at:
25-11-2014 12:32:00.000, next scheduled at: 25-11-2014 12:36:00.000
2014-11-25 12:32:04,902 [ontext_Worker-2] INFO LoggingJobHistoryPlugin
- Job [cluster.quartz] to be fired by trigger [cluster.quartz], re-fire: 0
2014-11-25 12:32:04,958 [ontext_Worker-2] INFO QueryBuilderProcessor -
Started
****processor started******//system.out line from QueryBuilderProcessor
****processor Ends******
2014-11-25 12:32:16,658 [ontext_Worker-2] INFO LoggingJobHistoryPlugin
- Job [cluster.quartz] execution complete and reports: null
2014-11-25 12:32:16,658 [ontext_Worker-2] INFO LoggingTriggerHistoryPlugin
- Trigger [cluster.quartz] completed firing job [cluster.quartz] with
resulting trigger instruction code: DO NOTHING. Next scheduled at:
25-11-2014 12:36:00.000
Captured in qrtz_triggers table:
S.NO. TRIGGER_GROUP JOB_NAME JOB_GROUP NEXT_FIRE_TIME
PREV_FIRE_TIME PRIORITY TRIGGER_STATE TRIGGER_TYPE
1. RECOVERING_JOBS quartz cluster 1416905400000 -1
5 SIMPLE WAITING
2. cluster quartZ cluster 1416905640000
1416905400000 5 CRON WAITING
Question:
1.In recovery scenario QueryBuilderProcessor bean is not getting called from
quartz endpoint but in next interval QueryBuilderProcessor bean is getting
called . Did I miss anything here?
*jar downloaded from below link:*
http://repository.apache.org/content/groups/snapshots/org/apache/camel/camel-quartz2/2.14.1-SNAPSHOT/
Please correct me If I did any mistake.
--
View this message in context: http://camel.465427.n5.nabble.com/Quartz-clustering-in-camel-spring-DSL-JIRA-CAMEL-8076-tp5759589.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Quartz clustering in camel spring DSL - JIRA CAMEL-8076
Posted by selva <sg...@gmail.com>.
Sorry Willem , its typo error I could not copy the row from DB so manually
formed the table format.
*quartZ* -> small 'z' i.e quartz
--
View this message in context: http://camel.465427.n5.nabble.com/Quartz-clustering-in-camel-spring-DSL-JIRA-CAMEL-8076-tp5759589p5759639.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Quartz clustering in camel spring DSL - JIRA CAMEL-8076
Posted by Willem Jiang <wi...@gmail.com>.
Hi Selva,
Thanks for sharing your patch with us. Your patch looks good to me, and I managed to reproduce the error by doing some change in the SpringQuartzConsumerTwoAppsClusteredRecoveryTest.
I will commit the patch shortly.
Regards,
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On December 2, 2014 at 7:23:03 PM, selva (sgpselvam@gmail.com) wrote:
> Hi Willem,
>
> I have downloaded(2.14.x from github) and debugged the camel quartz
> component source code and found the reason for the issue i am facing.
>
> As i already posted i am facing issue in the cluster environment, deployed
> the camel application as stand alone program running 2 instances. While
> executing a route if one instance goes down the other instance should
> trigger the same route immediately as we have configured
> *recoverableJob=true* in the quartz2 endpoint .
>
> The issue i was facing like the second instance is triggering quartz2
> endpoint but not executing my process(QueryBuilderProcessor) .
>
> Example :
>
>
> > uri="quartz2://cluster/quartz?cron=0+0/4+++*+?&durableJob=true&stateful=true&recoverableJob=true">
>
>
>
>
> While debugging i found that in camelJob.java, the triggerkey in the normal
> and the triggerykey during recovery is different,so the below condition is
> failing and going else block(Please check the highlighted else block).
>
>
> public void execute(JobExecutionContext context) throws
> JobExecutionException {
> Exchange exchange = null;
> try {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Running CamelJob jobExecutionContext={}",
> context);
> }
>
> CamelContext camelContext = getCamelContext(context);
> QuartzEndpoint endpoint = lookupQuartzEndpoint(camelContext,
> context);//* Step 1 : *
> exchange = endpoint.createExchange();
> exchange.setIn(new QuartzMessage(exchange, context));
> endpoint.getConsumerLoadBalancer().process(exchange);//*Step :2*
>
>
>
> * Step 1 : * Quartz endpoint creation
>
>
> if (triggerKey.equals(checkTriggerKey){
> return quartzEndpoint;
> }
>
>
> if (camelContext.hasEndpoint(endpointUri) != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Getting Endpoint from camelContext.");
> }
> result = camelContext.getEndpoint(endpointUri,
> QuartzEndpoint.class);
> } else {
> LOG.warn("Cannot find existing QuartzEndpoint with uri: {}.*
> Creating new endpoint instance.", endpointUri);*
> result = camelContext.getEndpoint(endpointUri,
> QuartzEndpoint.class);
> }
>
>
>
> Since its creating new quartz endpoint , the consumerLoadBalancer Object is
> null.
>
> So as next step its executing the below code and creating *New *
> consumerLoadBalancer Object
>
> *Step :2*
>
>
> public LoadBalancer getConsumerLoadBalancer() {
> if (consumerLoadBalancer == null) {
> consumerLoadBalancer = new RoundRobinLoadBalancer();
> }
> return consumerLoadBalancer;
> }
>
> Since its creating as a new Object ,the processors property is empty so
> not calling my QueryBuilderProcessor Processor.
>
> In normal flow(not recovery) i can see list of processors details in the
> processors property of consumerLoadBalancer .
>
> *Temporary Fix for Testing:* Below Code working fine in Recovery flow with
> immediate retry.
>
> As a temporary fix we have modified code in the camelJob.java ,
> lookupQuartzEndpoint method like below
> (Please check the highlighted text)
>
> protected QuartzEndpoint lookupQuartzEndpoint(CamelContext camelContext,
> JobExecutionContext quartzContext) throws JobExecutionException {
> TriggerKey triggerKey = quartzContext.getTrigger().getKey();
> * JobDetail jobDetail = quartzContext.getJobDetail();
> JobKey jobKey = jobDetail.getKey();
> *
> if (LOG.isDebugEnabled()) {
> LOG.debug("Looking up existing QuartzEndpoint with
> triggerKey={}", triggerKey);
> }
>
> // check all active routes for the quartz endpoint this task matches
> // as we prefer to use the existing endpoint from the routes
> for (Route route : camelContext.getRoutes()) {
> if (route.getEndpoint() instanceof QuartzEndpoint) {
> QuartzEndpoint quartzEndpoint = (QuartzEndpoint)
> route.getEndpoint();
> TriggerKey checkTriggerKey = quartzEndpoint.getTriggerKey();
> if (LOG.isTraceEnabled()) {
> LOG.trace("Checking route endpoint={} with
> checkTriggerKey={}", quartzEndpoint, checkTriggerKey);
> }
> if
> (triggerKey.equals(checkTriggerKey)*||(jobDetail.requestsRecovery()== true
> &&
> jobKey.getGroup().equals(checkTriggerKey.getGroup())&&jobKey.getName().equals(checkTriggerKey.getName())))
> {*
> return quartzEndpoint;
> }
> }
> }
> return quartzEndpoint;
> }
>
>
> Except the triggerKey remaning all jobdetails are same in the existing
> quartzEndpoint so we put the condition check to return the existing quartz
> endpoint instead creating new quartz endpoint in the recoveryflow.
>
> * Summary :*
> The problem we found is while creating new quartz endpoint the
> consumerLoadBalancer is null.
>
> Please let us know the right way to get the consumerLoadBalancer values
> while creating new quartz endpoint in case of recovery flow.
>
> Thanks,
> selva
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Quartz-clustering-in-camel-spring-DSL-JIRA-CAMEL-8076-tp5759589p5759928.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Quartz clustering in camel spring DSL - JIRA CAMEL-8076
Posted by selva <sg...@gmail.com>.
Hi Willem,
I have downloaded(2.14.x from github) and debugged the camel quartz
component source code and found the reason for the issue i am facing.
As i already posted i am facing issue in the cluster environment, deployed
the camel application as stand alone program running 2 instances. While
executing a route if one instance goes down the other instance should
trigger the same route immediately as we have configured
*recoverableJob=true* in the quartz2 endpoint .
The issue i was facing like the second instance is triggering quartz2
endpoint but not executing my process(QueryBuilderProcessor) .
Example :
<route id="quartz" trace="true">
<from
uri="quartz2://cluster/quartz?cron=0+0/4+++*+?&durableJob=true&stateful=true&recoverableJob=true">
<to uri="bean:QueryBuilderProcessor" />
</route>
While debugging i found that in camelJob.java, the triggerkey in the normal
and the triggerykey during recovery is different,so the below condition is
failing and going else block(Please check the highlighted else block).
public void execute(JobExecutionContext context) throws
JobExecutionException {
Exchange exchange = null;
try {
if (LOG.isDebugEnabled()) {
LOG.debug("Running CamelJob jobExecutionContext={}",
context);
}
CamelContext camelContext = getCamelContext(context);
QuartzEndpoint endpoint = lookupQuartzEndpoint(camelContext,
context);//* Step 1 : *
exchange = endpoint.createExchange();
exchange.setIn(new QuartzMessage(exchange, context));
endpoint.getConsumerLoadBalancer().process(exchange);//*Step :2*
* Step 1 : * Quartz endpoint creation
if (triggerKey.equals(checkTriggerKey){
return quartzEndpoint;
}
if (camelContext.hasEndpoint(endpointUri) != null) {
if (LOG.isDebugEnabled()) {
LOG.debug("Getting Endpoint from camelContext.");
}
result = camelContext.getEndpoint(endpointUri,
QuartzEndpoint.class);
} else {
LOG.warn("Cannot find existing QuartzEndpoint with uri: {}.*
Creating new endpoint instance.", endpointUri);*
result = camelContext.getEndpoint(endpointUri,
QuartzEndpoint.class);
}
Since its creating new quartz endpoint , the consumerLoadBalancer Object is
null.
So as next step its executing the below code and creating *New *
consumerLoadBalancer Object
*Step :2*
public LoadBalancer getConsumerLoadBalancer() {
if (consumerLoadBalancer == null) {
consumerLoadBalancer = new RoundRobinLoadBalancer();
}
return consumerLoadBalancer;
}
Since its creating as a new Object ,the processors property is empty so
not calling my QueryBuilderProcessor Processor.
In normal flow(not recovery) i can see list of processors details in the
processors property of consumerLoadBalancer .
*Temporary Fix for Testing:* Below Code working fine in Recovery flow with
immediate retry.
As a temporary fix we have modified code in the camelJob.java ,
lookupQuartzEndpoint method like below
(Please check the highlighted text)
protected QuartzEndpoint lookupQuartzEndpoint(CamelContext camelContext,
JobExecutionContext quartzContext) throws JobExecutionException {
TriggerKey triggerKey = quartzContext.getTrigger().getKey();
* JobDetail jobDetail = quartzContext.getJobDetail();
JobKey jobKey = jobDetail.getKey();
*
if (LOG.isDebugEnabled()) {
LOG.debug("Looking up existing QuartzEndpoint with
triggerKey={}", triggerKey);
}
// check all active routes for the quartz endpoint this task matches
// as we prefer to use the existing endpoint from the routes
for (Route route : camelContext.getRoutes()) {
if (route.getEndpoint() instanceof QuartzEndpoint) {
QuartzEndpoint quartzEndpoint = (QuartzEndpoint)
route.getEndpoint();
TriggerKey checkTriggerKey = quartzEndpoint.getTriggerKey();
if (LOG.isTraceEnabled()) {
LOG.trace("Checking route endpoint={} with
checkTriggerKey={}", quartzEndpoint, checkTriggerKey);
}
if
(triggerKey.equals(checkTriggerKey)*||(jobDetail.requestsRecovery()== true
&&
jobKey.getGroup().equals(checkTriggerKey.getGroup())&&jobKey.getName().equals(checkTriggerKey.getName())))
{*
return quartzEndpoint;
}
}
}
return quartzEndpoint;
}
Except the triggerKey remaning all jobdetails are same in the existing
quartzEndpoint so we put the condition check to return the existing quartz
endpoint instead creating new quartz endpoint in the recoveryflow.
* Summary :*
The problem we found is while creating new quartz endpoint the
consumerLoadBalancer is null.
Please let us know the right way to get the consumerLoadBalancer values
while creating new quartz endpoint in case of recovery flow.
Thanks,
selva
--
View this message in context: http://camel.465427.n5.nabble.com/Quartz-clustering-in-camel-spring-DSL-JIRA-CAMEL-8076-tp5759589p5759928.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Quartz clustering in camel spring DSL - JIRA CAMEL-8076
Posted by Willem Jiang <wi...@gmail.com>.
Did you clean up your database after testing?
S.NO. TRIGGER_GROUP JOB_NAME JOB_GROUP NEXT_FIRE_TIME PREV_FIRE_TIME PRIORITY TRIGGER_STATE TRIGGER_TYPE
1. RECOVERING_JOBS quartz cluster 1416905400000 -1 5 SIMPLE WAITING
2. cluster quartZ cluster 1416905640000 1416905400000 5 CRON WAITING
The RECOVERING_JOBS JOB_Name is quartz, but the cluster JOB_NAME is quartZ.
The laster character name is different.
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On November 26, 2014 at 2:28:07 PM, selva (sgpselvam@gmail.com) wrote:
> Hi Willem,
>
> I have defined one quartz2 endpoint route and one sub route in
> camel-context.xml file . Whenever the quartz scheduler triggers , the quartz
> end point will call the sub route using direct component. I am printing some
> text in console using bean processor(QueryBuilderProcessor) which is defined
> in sub route.
>
> > class="cluster.quartz.QueryBuilderProcessor" />
>
> > uri="quartz2://cluster/quartz?cron=0+0/4+++*+?&durableJob=true&stateful=true&recoverableJob=true">
> //calling sub rout
> here.
> ...
>
>
>
>
>
>
>
>
>
> My cluster configuration:
> I have followed below Link to set up Clustering with the JDBC-Jobstore.
>
> http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigJDBCJobStoreClustering
> I have quartz.properties file in resource folder like
> org/quartz/quartz.properties.
>
> org.quartz.scheduler.instanceName = ClusteredScheduler
> org.quartz.scheduler.instanceId = AUTO
> org.quartz.jobStore.isClustered = true
> ...
> org.quartz.plugin.triggerHistory.class=org.quartz.plugins.history.LoggingTriggerHistoryPlugin
> org.quartz.plugin.triggerHistory.triggerFiredMessage=Trigger [{1}.{0}] fired
> job [{6}.{5}] scheduled at: {2, date, dd-MM-yyyy HH:mm:ss.SSS}, next
> scheduled at: {3, date, dd-MM-yyyy HH:mm:ss.SSS}
> org.quartz.plugin.triggerHistory.triggerCompleteMessage=Trigger [{1}.{0}]
> completed firing job [{6}.{5}] with resulting trigger instruction code: {9}.
> Next scheduled at: {3, date, dd-MM-yyyy HH:mm:ss.SSS}
> org.quartz.plugin.triggerHistory.triggerMisfiredMessage=Trigger [{1}.{0}]
> misfired job [{6}.{5}]. Should have fired at: {3, date, dd-MM-yyyy
> HH:mm:ss.SSS}
> org.quartz.plugin.jobHistory.class=org.quartz.plugins.history.LoggingJobHistoryPlugin
> org.quartz.plugin.jobHistory.jobToBeFiredMessage=Job [{1}.{0}] to be fired
> by trigger [{4}.{3}], re-fire: {7}
> org.quartz.plugin.jobHistory.jobSuccessMessage=Job [{1}.{0}] execution
> complete and reports: {8}
> org.quartz.plugin.jobHistory.jobFailedMessage=Job [{1}.{0}] execution failed
> with exception: {8}
> org.quartz.plugin.jobHistory.jobWasVetoedMessage=Job [{1}.{0}] was vetoed.
> It was to be fired by trigger [{4}.{3}] at: {2, date, dd-MM-yyyy
> HH:mm:ss.SSS}
>
>
> QueryBuilderProcessor.java
>
> public void process(Exchange exchange) throws Exception {
> LOGGER.info("Started");
> System.out.println(****processor started******");
> Thread.sleep(10000);
> System.out.println(****processor Ends******);
>
> }
>
> Test case:
>
> I am running the same camel application as standalone using
> org.apache.camel.spring.Main class in two eclipse instance in same machine,
> So by default clock is sync.
>
> Two Node:
> 1. Eclipse Instance 1:(Node1) First I am starting application from eclipse
> instance 1 , based on quartz properties file Quartz tables are populated
> with the job details.
>
> 2.Eclipse Instance 2:(Node2) I am starting same application from eclipse
> instance 2.
>
> Both instance are started and could see their scheduler entries (Node1 &
> Node2) in qrtz_scheduler_state table.
>
> Stopping Job:
>
> Constantly I watching the TRIGGER_STATE column of job in *qrtz_triggers*
> table & console for two camel application.
>
> step1 : Initially Trigger state will be "WAITING"
> stept2: Once the Trigger state changed to "AQUIRED" , I go back to console
> find the which node picked the job.
> step3: Assume ****processor started******" printed from
> QueryBuilderProcessor.java(Node1) and Immediately terminate the Node1 by
> clicking terminate button in eclipse.
> step4: Now current TRIGGER state is "BLOCKED".
> step5: I could see the recovery logs in Node2 console.(Which is mentioned in
> the above question).
>
> *My doubt is , if you see the Recovering job trigger details in
> *qrtz_triggers* table above question, TRIGGER_TYPE is SIMPLE, but we defined
> in our route is CRON type*.
>
> Please refer the section "*Captured in qrtz_triggers table*" in above
> question.
>
> I think the problem is CRON type, Recovery Job Tigger type should be CRON
> right?
>
>
> Thanks,
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Quartz-clustering-in-camel-spring-DSL-JIRA-CAMEL-8076-tp5759589p5759614.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Quartz clustering in camel spring DSL - JIRA CAMEL-8076
Posted by selva <sg...@gmail.com>.
Hi Willem,
I have defined one quartz2 endpoint route and one sub route in
camel-context.xml file . Whenever the quartz scheduler triggers , the quartz
end point will call the sub route using direct component. I am printing some
text in console using bean processor(QueryBuilderProcessor) which is defined
in sub route.
<bean id="QueryBuilderProcessor"
class="cluster.quartz.QueryBuilderProcessor" />
<route id="quartz" trace="true">
<from
uri="quartz2://cluster/quartz?cron=0+0/4+++*+?&durableJob=true&stateful=true&recoverableJob=true">
<to uri="direct:queryProcessor" />//calling sub rout
here.
...
</route>
<route id="queryProcessor" startupOrder="1">
<from uri="direct:queryProcessor" />
<to uri="bean:QueryBuilderProcessor" />
</route>
My cluster configuration:
I have followed below Link to set up Clustering with the JDBC-Jobstore.
http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigJDBCJobStoreClustering
I have quartz.properties file in resource folder like
org/quartz/quartz.properties.
org.quartz.scheduler.instanceName = ClusteredScheduler
org.quartz.scheduler.instanceId = AUTO
org.quartz.jobStore.isClustered = true
...
org.quartz.plugin.triggerHistory.class=org.quartz.plugins.history.LoggingTriggerHistoryPlugin
org.quartz.plugin.triggerHistory.triggerFiredMessage=Trigger [{1}.{0}] fired
job [{6}.{5}] scheduled at: {2, date, dd-MM-yyyy HH:mm:ss.SSS}, next
scheduled at: {3, date, dd-MM-yyyy HH:mm:ss.SSS}
org.quartz.plugin.triggerHistory.triggerCompleteMessage=Trigger [{1}.{0}]
completed firing job [{6}.{5}] with resulting trigger instruction code: {9}.
Next scheduled at: {3, date, dd-MM-yyyy HH:mm:ss.SSS}
org.quartz.plugin.triggerHistory.triggerMisfiredMessage=Trigger [{1}.{0}]
misfired job [{6}.{5}]. Should have fired at: {3, date, dd-MM-yyyy
HH:mm:ss.SSS}
org.quartz.plugin.jobHistory.class=org.quartz.plugins.history.LoggingJobHistoryPlugin
org.quartz.plugin.jobHistory.jobToBeFiredMessage=Job [{1}.{0}] to be fired
by trigger [{4}.{3}], re-fire: {7}
org.quartz.plugin.jobHistory.jobSuccessMessage=Job [{1}.{0}] execution
complete and reports: {8}
org.quartz.plugin.jobHistory.jobFailedMessage=Job [{1}.{0}] execution failed
with exception: {8}
org.quartz.plugin.jobHistory.jobWasVetoedMessage=Job [{1}.{0}] was vetoed.
It was to be fired by trigger [{4}.{3}] at: {2, date, dd-MM-yyyy
HH:mm:ss.SSS}
QueryBuilderProcessor.java
public void process(Exchange exchange) throws Exception {
LOGGER.info("Started");
System.out.println(****processor started******");
Thread.sleep(10000);
System.out.println(****processor Ends******);
}
Test case:
I am running the same camel application as standalone using
org.apache.camel.spring.Main class in two eclipse instance in same machine,
So by default clock is sync.
Two Node:
1. Eclipse Instance 1:(Node1) First I am starting application from eclipse
instance 1 , based on quartz properties file Quartz tables are populated
with the job details.
2.Eclipse Instance 2:(Node2) I am starting same application from eclipse
instance 2.
Both instance are started and could see their scheduler entries (Node1 &
Node2) in qrtz_scheduler_state table.
Stopping Job:
Constantly I watching the TRIGGER_STATE column of job in *qrtz_triggers*
table & console for two camel application.
step1 : Initially Trigger state will be "WAITING"
stept2: Once the Trigger state changed to "AQUIRED" , I go back to console
find the which node picked the job.
step3: Assume ****processor started******" printed from
QueryBuilderProcessor.java(Node1) and Immediately terminate the Node1 by
clicking terminate button in eclipse.
step4: Now current TRIGGER state is "BLOCKED".
step5: I could see the recovery logs in Node2 console.(Which is mentioned in
the above question).
*My doubt is , if you see the Recovering job trigger details in
*qrtz_triggers* table above question, TRIGGER_TYPE is SIMPLE, but we defined
in our route is CRON type*.
Please refer the section "*Captured in qrtz_triggers table*" in above
question.
I think the problem is CRON type, Recovery Job Tigger type should be CRON
right?
Thanks,
--
View this message in context: http://camel.465427.n5.nabble.com/Quartz-clustering-in-camel-spring-DSL-JIRA-CAMEL-8076-tp5759589p5759614.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Quartz clustering in camel spring DSL - JIRA CAMEL-8076
Posted by Willem Jiang <wi...@gmail.com>.
I did some test on my side, I cannot reproduce the error that you meet.
From the failure with recovery log that you showed, The next scheduled is null and Camel Job cannot find the QuartzEndpoint.
2014-11-25 12:29:08,599 [ontext_Worker-1] INFO LoggingTriggerHistoryPlugin
- Trigger [RECOVERING_JOBS.recover_16898502964_1416898727319] fired job
[cluster.quartz] scheduled at: 25-11-2014 12:28:00.000, next scheduled at:
null
2014-11-25 12:29:08,600 [ontext_Worker-1] INFO LoggingJobHistoryPlugin
- Job [cluster.quartz] to be fired by trigger
[RECOVERING_JOBS.recover_16898502964_1416898727319], re-fire: 0
2014-11-25 12:29:08,604 [ontext_Worker-1] WARN CamelJob
- Cannot find existing QuartzEndpoint with uri:
quartz2://cluster/quartz?cron=0+0%2F4+*+*+*+%3F&recoverableJob=true&stateful=true.
Creating new endpoint instance.
Can you tell me how did you stop the job ?
What’s your quartz cluster looks like ?
If you have a test case to share, it could help us to resolve the issue more easily.
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On November 25, 2014 at 9:56:46 PM, selva (sgpselvam@gmail.com) wrote:
> Hi Willem,
>
> I have tested with camel 2.14.1-SNAPSHOT and the exception is gone but
> QueryBuilderProcessor is not getting called from quartz end point.
>
> Please find below configuration and logging information for both recovery
> and success scenario. In success scenario QueryBuilderProcessor is getting
> called from quartz endpoint and able see some system.out.printl statement in
> success case but its failing in recovery case.
>
> Please help me to solve this issue.
>
> Route Configuration:
>
>
> > uri="quartz2://cluster/quartz?cron=0+0/4+++*+?&durableJob=true&stateful=true&recoverableJob=true">
> //calling sub rout here.
> ...
>
>
>
>
>
>
>
> Please find the below logs for Calling QueryBuilderProcessor from quartz
> endpoint:
> =================================================================================
>
> Case 1 : //Logs for Immediate recovery : failure with recovery
> ---------------------------------------------------------------
> 2014-11-25 12:28:47,319 [_ClusterManager] INFO JobStoreTX
> - ClusterManager: detected 1 failed or restarted instances.
> 2014-11-25 12:28:47,319 [_ClusterManager] INFO JobStoreTX
> - ClusterManager: Scanning for instance "16898502964"'s failed in-progress
> jobs.
> 2014-11-25 12:28:56,675 [_ClusterManager] INFO JobStoreTX
> - ClusterManager: ......Scheduled 1 recoverable job(s) for recovery.
> 2014-11-25 12:29:08,599 [ontext_Worker-1] INFO LoggingTriggerHistoryPlugin
> - Trigger [RECOVERING_JOBS.recover_16898502964_1416898727319] fired job
> [cluster.quartz] scheduled at: 25-11-2014 12:28:00.000, next scheduled at:
> null
> 2014-11-25 12:29:08,600 [ontext_Worker-1] INFO LoggingJobHistoryPlugin
> - Job [cluster.quartz] to be fired by trigger
> [RECOVERING_JOBS.recover_16898502964_1416898727319], re-fire: 0
> 2014-11-25 12:29:08,604 [ontext_Worker-1] WARN CamelJob
> - Cannot find existing QuartzEndpoint with uri:
> quartz2://cluster/quartz?cron=0+0%2F4+*+*+*+%3F&recoverableJob=true&stateful=true.
> Creating new endpoint instance.
> 2014-11-25 12:29:09,514 [ontext_Worker-1] INFO QuartzEndpoint
> - Job cluster.quartz (triggerType=CronTriggerImpl,
> jobClass=StatefulCamelJob) is scheduled. Next fire date is null
> 2014-11-25 12:29:09,526 [ontext_Worker-1] INFO LoggingJobHistoryPlugin
> - Job [cluster.quartz] execution complete and reports: null
> 2014-11-25 12:29:09,528 [ontext_Worker-1] INFO LoggingTriggerHistoryPlugin
> - Trigger [RECOVERING_JOBS.recover_16898502964_1416898727319] completed
> firing job [cluster.quartz] with resulting trigger instruction code: DELETE
> TRIGGER. Next scheduled at: null
>
>
> case 2: //Normal flow : success
> -------------------------------
> 2014-11-25 12:32:04,901 [ontext_Worker-2] INFO LoggingTriggerHistoryPlugin
> - Trigger [cluster.quartz] fired job [cluster.quartz] scheduled at:
> 25-11-2014 12:32:00.000, next scheduled at: 25-11-2014 12:36:00.000
> 2014-11-25 12:32:04,902 [ontext_Worker-2] INFO LoggingJobHistoryPlugin
> - Job [cluster.quartz] to be fired by trigger [cluster.quartz], re-fire: 0
> 2014-11-25 12:32:04,958 [ontext_Worker-2] INFO QueryBuilderProcessor -
> Started
> ****processor started******//system.out line from QueryBuilderProcessor
> ****processor Ends******
> 2014-11-25 12:32:16,658 [ontext_Worker-2] INFO LoggingJobHistoryPlugin
> - Job [cluster.quartz] execution complete and reports: null
> 2014-11-25 12:32:16,658 [ontext_Worker-2] INFO LoggingTriggerHistoryPlugin
> - Trigger [cluster.quartz] completed firing job [cluster.quartz] with
> resulting trigger instruction code: DO NOTHING. Next scheduled at:
> 25-11-2014 12:36:00.000
>
>
> Captured in qrtz_triggers table:
>
> S.NO. TRIGGER_GROUP JOB_NAME JOB_GROUP NEXT_FIRE_TIME
> PREV_FIRE_TIME PRIORITY TRIGGER_STATE TRIGGER_TYPE
> 1. RECOVERING_JOBS quartz cluster 1416905400000 -1
> 5 SIMPLE WAITING
> 2. cluster quartZ cluster 1416905640000
> 1416905400000 5 CRON WAITING
>
>
> Question:
> 1.In recovery scenario QueryBuilderProcessor bean is not getting called from
> quartz endpoint but in next interval QueryBuilderProcessor bean is getting
> called . Did I miss anything here?
>
> *jar downloaded from below link:*
> http://repository.apache.org/content/groups/snapshots/org/apache/camel/camel-quartz2/2.14.1-SNAPSHOT/
>
>
> Please correct me If I did any mistake.
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Quartz-clustering-in-camel-spring-DSL-JIRA-CAMEL-8076-tp5759589.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>