You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "lakshmi.prashant" <la...@gmail.com> on 2014/11/06 12:58:24 UTC
Re: Quartz job data deletion in clustered quartz2
Hi,
That does not help.
If we have a shared scheduler instance (by exposing the
StdSchedulerFactory as a OSGi service) used by the different camel quartz
components / routes, we face the following issue:
After 1 camel quartz route is un-deployed & removed, the scheduler
instance starts misfiring, due to ClassLoader issues in loading the CamelJob
class.
The other camel quartz routes / bundles sharing the scheduler instance
start misfiring after the CamelJob class in the I camel route bundle gets
uninstalled (when that bundle is undeployed).
When the scheduler instance tries to acquire the next triggers & load
the Job class, the Quartz CascadingClassLoaderHelper tries to remember the
scheme that was last used to load the CamelJob class & reports the following
exception.
Caused by: java.lang.ClassNotFoundException: Unable to load class
org.apache.camel.component.quartz2.CamelJob by any known loaders.
at
org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:126)
at
org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138)
at
org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2824)
... 5 common frames omitted
Caused by: java.lang.IllegalStateException: Bundle "Quartz2_Camel_Test_5Min"
has been uninstalled
I have raised this issue in Quartz forum
<https://groups.google.com/forum/#!topic/quartz-scheduler/Ptek0hAhQJw> as
well.
Can you please let me know if I can configure in quartz.properties any
recommended value for the org.quartz.scheduler.classLoadHelper.class, so
that quartz will load the CamelJob class correctly, when multiple camel
quartz bundles share the same scheduler instance...
<http://camel.465427.n5.nabble.com/file/n5758609/Acquire_Triggers_After_Undeploy_Route2.png>
Thanks,
Lakshmi
--
View this message in context: http://camel.465427.n5.nabble.com/Quartz-job-data-deletion-in-clustered-quartz2-tp5757508p5758609.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Quartz job data deletion in clustered quartz2
Posted by Willem Jiang <wi...@gmail.com>.
Hi Lakshmi,
We did some work in camel-quartz2 recently such as reschedule the job for cluster[1], do you mind test them to see if it fix some of your cluster issue?
You can use Camel 2.14.1-SNAPSHOT for verification.
[1]https://issues.apache.org/jira/browse/CAMEL-7627
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On November 24, 2014 at 12:37:37 PM, lakshmi.prashant (lakshmi.prashant@gmail.com) wrote:
> Hi Willem,
>
> Scheduler is not created from outside, but on deployment of camel blueprint
> bundles.
> But the scheduler jobs are not deleted on un-deployment of the camel
> bundles, in clustered mode and this is what needs to be handled - while
> taking care of durable jobs and job recovery'.
>
> Thanks,
> Lakshmi
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Quartz-job-data-deletion-in-clustered-quartz2-tp5757508p5759519.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Quartz job data deletion in clustered quartz2
Posted by "lakshmi.prashant" <la...@gmail.com>.
Hi Willem,
Scheduler is not created from outside, but on deployment of camel blueprint
bundles.
But the scheduler jobs are not deleted on un-deployment of the camel
bundles, in clustered mode and this is what needs to be handled - while
taking care of durable jobs and job recovery'.
Thanks,
Lakshmi
--
View this message in context: http://camel.465427.n5.nabble.com/Quartz-job-data-deletion-in-clustered-quartz2-tp5757508p5759519.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Quartz job data deletion in clustered quartz2
Posted by Willem Jiang <wi...@gmail.com>.
camel-quartz component is managed the scheduler rightly if the scheduler is created by itself. But it is not his job to clean up job data if the scheduler is created from outside.
Quartz scheduler should be more resilient by keeping processing other triggers as you suggested.
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On November 21, 2014 at 4:07:24 PM, lakshmi.prashant (lakshmi.prashant@gmail.com) wrote:
> Hi Willem,
>
> We are listening to the un-deployment event ourselves.
>
> 1. Actually, if the job is deleted from any UI (that is used to schedule
> jobs) - that UI will have to take care to remove the job data from the
> scheduler.
>
> 2. But, in the camel quartz scenarios, the jobs are created at the start of
> routes. Once the routes get removed (un-deployed), the quartz Job data will
> also have to be removed.
>
> Quartz scheduler will not automatically come to know of this. I had
> already raised an issue in quartz forum that they should make some
> improvement:
>
> (i.e.) Quartz scheduler has to handle the following situation while
> acquiring triggers to be run: if the job class related to any job is
> missing, it should remove that job data (or) log the issue & ignore that &
> continue processing the other triggers.
>
> 3. Unless the issue is resolved either in camel or in quartz, users of
> clustered camel quartz will continue to face this issue, as all camel quartz
> routes will stop running.
>
> Thanks,
> Lakshmi
>
>
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Quartz-job-data-deletion-in-clustered-quartz2-tp5757508p5759412.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Quartz job data deletion in clustered quartz2
Posted by "lakshmi.prashant" <la...@gmail.com>.
Hi Willem,
We are listening to the un-deployment event ourselves.
1. Actually, if the job is deleted from any UI (that is used to schedule
jobs) - that UI will have to take care to remove the job data from the
scheduler.
2. But, in the camel quartz scenarios, the jobs are created at the start of
routes. Once the routes get removed (un-deployed), the quartz Job data will
also have to be removed.
Quartz scheduler will not automatically come to know of this. I had
already raised an issue in quartz forum that they should make some
improvement:
(i.e.) Quartz scheduler has to handle the following situation while
acquiring triggers to be run: if the job class related to any job is
missing, it should remove that job data (or) log the issue & ignore that &
continue processing the other triggers.
3. Unless the issue is resolved either in camel or in quartz, users of
clustered camel quartz will continue to face this issue, as all camel quartz
routes will stop running.
Thanks,
Lakshmi
--
View this message in context: http://camel.465427.n5.nabble.com/Quartz-job-data-deletion-in-clustered-quartz2-tp5757508p5759412.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Quartz job data deletion in clustered quartz2
Posted by Willem Jiang <wi...@gmail.com>.
Hi Lakshmi,
I just have some time to revisit the issue of clustered quartz2.
It’s the scheduler work to avoid triggers the job data when the bundle is stop, camel-quartz cannot monitor the OSGi bundle event for that.
But I think I can catch ObjectAlreadyExistsException to do a double check for it.
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On November 10, 2014 at 12:34:50 PM, lakshmi.prashant (lakshmi.prashant@gmail.com) wrote:
> Hi Claus,
>
> There is a mis-communication - we need not have a special classloader
> helper, I think.
>
> The issue was because on the un-deployment of 1 camel blueprint bundle
> (with camel quartz2 route),* the quartz job data is not deleted from db - if
> it is clustered quartz.*
>
> Unfortunately, we do not want to delete the job data, when the route is
> stopped using RoutePolicySupport class, as the main intent from clustered
> quartz is job recovery.
> - The scheduler will be shut down (QuartzComponent: doStop()) if there are
> no more jobs (if the scheduler is not shared across camel context bundles) &
> it works fine.
> - But if the scheduler configuration / scheduler instance is shared across
> camel quartz routes / bundles, the scheduler continues to run.
> - When the scheduler acquires next trigger, the trigger related to
> undeployed bundle is also obtained & then it tries to execute that trigger
> by executing CamelJob class from uninstalled bundle, using
> CascadingClassLoaderHelper.
> - If it cannot load the class for that trigger, it throws exception and
> the rest of the triggers do not get executed at that time - So we get
> misfires.
>
> Please refer to line no. 876 in
> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.java - this quartz class throws
> exception, if job class is not loaded and does not proceed further.
> job.setJobClass(loadHelper.loadClass(rs .getString(COL_JOB_CLASS)));
>
> 1. I have written an osgi EventHandler service that will listen to 'bundle
> undeploy' events, that get published.
> 2. If the osgi bundle related to camel quartz2 is undeployed, it will
> remove the corresponding job data from DB.
>
> If this can be handled by camel quartz2, it will become simple for
> end-users.
>
> a) There is an issue in camel QuartzEndpoint.java in addJobInScheduler(). We
> were getting misfires in some nodes of the cluster, due to below issue.
>
> a) If the trigger does not exist in DB, it tries to schedule the job
> b) But this is not an atomic transaction - After the call to find a
> trigger from DB is made, some other node in the cluster could have created
> the trigger, resulting in ObjectAlreadyExistsException when call to schedule
> job is made
> c) Then misfires happen in that cluster node, as the Quartz component /
> camel context itself does not get started.
>
> private void addJobInScheduler() throws Exception {
> // Add or use existing trigger to/from scheduler
> Scheduler scheduler = getComponent().getScheduler();
> JobDetail jobDetail;
> Trigger trigger = scheduler.getTrigger(triggerKey);
> if (trigger == null) {
> jobDetail = createJobDetail();
> trigger = createTrigger(jobDetail);
>
> updateJobDataMap(jobDetail);
>
> // Schedule it now. Remember that scheduler might not be started
> it, but we can schedule now.
> try{
> Date nextFireDate = scheduler.scheduleJob(jobDetail, trigger);
> if (LOG.isInfoEnabled()) {
> LOG.info("Job {} (triggerType={}, jobClass={}) is
> scheduled. Next fire date is {}",
> new Object[] {trigger.getKey(),
> trigger.getClass().getSimpleName(),
>
> jobDetail.getJobClass().getSimpleName(), nextFireDate});
> }
> }
> * catch(ObjectAlreadyExistsException e){
> //double-check if Some other VM might has already stored the
> job & trigger in clustered mode
> if(!(getComponent().isClustered())){
> throw e;
> }
> else{
> trigger = scheduler.getTrigger(triggerKey);
> if(trigger==null){
> throw new SchedulerException("Trigger could not be found in
> quartz scheduler.");
> }
> }
> }*
> } else {
> ensureNoDupTriggerKey();
> }
>
> Can the above correction in QuartzComponent.java be made?
>
> Thanks,
> Lakshmi
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Quartz-job-data-deletion-in-clustered-quartz2-tp5757508p5758806.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Quartz job data deletion in clustered quartz2
Posted by "lakshmi.prashant" <la...@gmail.com>.
Hi Claus,
There is a mis-communication - we need not have a special classloader
helper, I think.
The issue was because on the un-deployment of 1 camel blueprint bundle
(with camel quartz2 route),* the quartz job data is not deleted from db - if
it is clustered quartz.*
Unfortunately, we do not want to delete the job data, when the route is
stopped using RoutePolicySupport class, as the main intent from clustered
quartz is job recovery.
- The scheduler will be shut down (QuartzComponent: doStop()) if there are
no more jobs (if the scheduler is not shared across camel context bundles) &
it works fine.
- But if the scheduler configuration / scheduler instance is shared across
camel quartz routes / bundles, the scheduler continues to run.
- When the scheduler acquires next trigger, the trigger related to
undeployed bundle is also obtained & then it tries to execute that trigger
by executing CamelJob class from uninstalled bundle, using
CascadingClassLoaderHelper.
- If it cannot load the class for that trigger, it throws exception and
the rest of the triggers do not get executed at that time - So we get
misfires.
Please refer to line no. 876 in
org.quartz.impl.jdbcjobstore.StdJDBCDelegate.java - this quartz class throws
exception, if job class is not loaded and does not proceed further.
job.setJobClass(loadHelper.loadClass(rs .getString(COL_JOB_CLASS)));
1. I have written an osgi EventHandler service that will listen to 'bundle
undeploy' events, that get published.
2. If the osgi bundle related to camel quartz2 is undeployed, it will
remove the corresponding job data from DB.
If this can be handled by camel quartz2, it will become simple for
end-users.
a) There is an issue in camel QuartzEndpoint.java in addJobInScheduler(). We
were getting misfires in some nodes of the cluster, due to below issue.
a) If the trigger does not exist in DB, it tries to schedule the job
b) But this is not an atomic transaction - After the call to find a
trigger from DB is made, some other node in the cluster could have created
the trigger, resulting in ObjectAlreadyExistsException when call to schedule
job is made
c) Then misfires happen in that cluster node, as the Quartz component /
camel context itself does not get started.
private void addJobInScheduler() throws Exception {
// Add or use existing trigger to/from scheduler
Scheduler scheduler = getComponent().getScheduler();
JobDetail jobDetail;
Trigger trigger = scheduler.getTrigger(triggerKey);
if (trigger == null) {
jobDetail = createJobDetail();
trigger = createTrigger(jobDetail);
updateJobDataMap(jobDetail);
// Schedule it now. Remember that scheduler might not be started
it, but we can schedule now.
try{
Date nextFireDate = scheduler.scheduleJob(jobDetail, trigger);
if (LOG.isInfoEnabled()) {
LOG.info("Job {} (triggerType={}, jobClass={}) is
scheduled. Next fire date is {}",
new Object[] {trigger.getKey(),
trigger.getClass().getSimpleName(),
jobDetail.getJobClass().getSimpleName(), nextFireDate});
}
}
* catch(ObjectAlreadyExistsException e){
//double-check if Some other VM might has already stored the
job & trigger in clustered mode
if(!(getComponent().isClustered())){
throw e;
}
else{
trigger = scheduler.getTrigger(triggerKey);
if(trigger==null){
throw new SchedulerException("Trigger could not be found in
quartz scheduler.");
}
}
}*
} else {
ensureNoDupTriggerKey();
}
Can the above correction in QuartzComponent.java be made?
Thanks,
Lakshmi
--
View this message in context: http://camel.465427.n5.nabble.com/Quartz-job-data-deletion-in-clustered-quartz2-tp5757508p5758806.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Quartz job data deletion in clustered quartz2
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I found an api, and logged a ticket
https://issues.apache.org/jira/browse/CAMEL-8020
On Sun, Nov 9, 2014 at 7:59 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> Yeah unfortunately class loading in OSGi and using 3rd party libraries
> that are NOT osgi friendly is a challenge, and you can hit these kind
> of issues here.
>
> I am not sure if quartz offer an api where you can provide a custom
> classloader, so we can better control this when the store want's to
> load a class.
>
>
>
> On Thu, Nov 6, 2014 at 12:58 PM, lakshmi.prashant
> <la...@gmail.com> wrote:
>> Hi,
>>
>> That does not help.
>>
>> If we have a shared scheduler instance (by exposing the
>> StdSchedulerFactory as a OSGi service) used by the different camel quartz
>> components / routes, we face the following issue:
>>
>> After 1 camel quartz route is un-deployed & removed, the scheduler
>> instance starts misfiring, due to ClassLoader issues in loading the CamelJob
>> class.
>> The other camel quartz routes / bundles sharing the scheduler instance
>> start misfiring after the CamelJob class in the I camel route bundle gets
>> uninstalled (when that bundle is undeployed).
>> When the scheduler instance tries to acquire the next triggers & load
>> the Job class, the Quartz CascadingClassLoaderHelper tries to remember the
>> scheme that was last used to load the CamelJob class & reports the following
>> exception.
>>
>> Caused by: java.lang.ClassNotFoundException: Unable to load class
>> org.apache.camel.component.quartz2.CamelJob by any known loaders.
>> at
>> org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:126)
>> at
>> org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138)
>> at
>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
>> at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2824)
>> ... 5 common frames omitted
>> Caused by: java.lang.IllegalStateException: Bundle "Quartz2_Camel_Test_5Min"
>> has been uninstalled
>>
>>
>> I have raised this issue in Quartz forum
>> <https://groups.google.com/forum/#!topic/quartz-scheduler/Ptek0hAhQJw> as
>> well.
>>
>>
>>
>> Can you please let me know if I can configure in quartz.properties any
>> recommended value for the org.quartz.scheduler.classLoadHelper.class, so
>> that quartz will load the CamelJob class correctly, when multiple camel
>> quartz bundles share the same scheduler instance...
>>
>> <http://camel.465427.n5.nabble.com/file/n5758609/Acquire_Triggers_After_Undeploy_Route2.png>
>>
>>
>> Thanks,
>> Lakshmi
>>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/Quartz-job-data-deletion-in-clustered-quartz2-tp5757508p5758609.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Quartz job data deletion in clustered quartz2
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Yeah unfortunately class loading in OSGi and using 3rd party libraries
that are NOT osgi friendly is a challenge, and you can hit these kind
of issues here.
I am not sure if quartz offer an api where you can provide a custom
classloader, so we can better control this when the store want's to
load a class.
On Thu, Nov 6, 2014 at 12:58 PM, lakshmi.prashant
<la...@gmail.com> wrote:
> Hi,
>
> That does not help.
>
> If we have a shared scheduler instance (by exposing the
> StdSchedulerFactory as a OSGi service) used by the different camel quartz
> components / routes, we face the following issue:
>
> After 1 camel quartz route is un-deployed & removed, the scheduler
> instance starts misfiring, due to ClassLoader issues in loading the CamelJob
> class.
> The other camel quartz routes / bundles sharing the scheduler instance
> start misfiring after the CamelJob class in the I camel route bundle gets
> uninstalled (when that bundle is undeployed).
> When the scheduler instance tries to acquire the next triggers & load
> the Job class, the Quartz CascadingClassLoaderHelper tries to remember the
> scheme that was last used to load the CamelJob class & reports the following
> exception.
>
> Caused by: java.lang.ClassNotFoundException: Unable to load class
> org.apache.camel.component.quartz2.CamelJob by any known loaders.
> at
> org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:126)
> at
> org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138)
> at
> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
> at
> org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2824)
> ... 5 common frames omitted
> Caused by: java.lang.IllegalStateException: Bundle "Quartz2_Camel_Test_5Min"
> has been uninstalled
>
>
> I have raised this issue in Quartz forum
> <https://groups.google.com/forum/#!topic/quartz-scheduler/Ptek0hAhQJw> as
> well.
>
>
>
> Can you please let me know if I can configure in quartz.properties any
> recommended value for the org.quartz.scheduler.classLoadHelper.class, so
> that quartz will load the CamelJob class correctly, when multiple camel
> quartz bundles share the same scheduler instance...
>
> <http://camel.465427.n5.nabble.com/file/n5758609/Acquire_Triggers_After_Undeploy_Route2.png>
>
>
> Thanks,
> Lakshmi
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Quartz-job-data-deletion-in-clustered-quartz2-tp5757508p5758609.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/