You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by "Tobias Bocanegra (JIRA)" <ji...@apache.org> on 2008/05/17 21:39:55 UTC
[jira] Created: (SLING-462) Endless loop in JobEventHandler
Endless loop in JobEventHandler
-------------------------------
Key: SLING-462
URL: https://issues.apache.org/jira/browse/SLING-462
Project: Sling
Issue Type: Bug
Components: Event
Affects Versions: 2.0.0
Reporter: Tobias Bocanegra
Priority: Critical
the runInBackground method has an endless loop, when
- the queue is not empty,
- the item still exists
- parallelProcessing is false
- the processing map already has the topic
- the eventNode is not locked
i don't know how exactly this state was reached, most probably by a failing JobProcessor.
note: after the respective nodes are deleted, the endless loop ends :-)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SLING-462) Endless loop in JobEventHandler
Posted by "Tobias Bocanegra (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SLING-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599922#action_12599922 ]
Tobias Bocanegra commented on SLING-462:
----------------------------------------
how about adding a nice mechanism for a failing job? i would not mind if a failing job is not dequeued automatically, but without a 'sleep-time', this just kills the CPU :-)
> Endless loop in JobEventHandler
> -------------------------------
>
> Key: SLING-462
> URL: https://issues.apache.org/jira/browse/SLING-462
> Project: Sling
> Issue Type: Bug
> Components: Event
> Affects Versions: 2.0.0
> Reporter: Tobias Bocanegra
> Assignee: Carsten Ziegeler
> Priority: Critical
>
> the runInBackground method has an endless loop, when
> - the queue is not empty,
> - the item still exists
> - parallelProcessing is false
> - the processing map already has the topic
> - the eventNode is not locked
> i don't know how exactly this state was reached, most probably by a failing JobProcessor.
> note: after the respective nodes are deleted, the endless loop ends :-)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SLING-462) Endless loop in JobEventHandler
Posted by "Carsten Ziegeler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SLING-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599829#action_12599829 ]
Carsten Ziegeler commented on SLING-462:
----------------------------------------
Looking further at the code, I can't see a problem with the code itself :)
However, it's true that if a job fails it is rescheduled for ever, if then more than one job is in the pipeline this results in an endless loop.
I created and implemented SLING-477 to set a limit for rescheduling.
> Endless loop in JobEventHandler
> -------------------------------
>
> Key: SLING-462
> URL: https://issues.apache.org/jira/browse/SLING-462
> Project: Sling
> Issue Type: Bug
> Components: Event
> Affects Versions: 2.0.0
> Reporter: Tobias Bocanegra
> Assignee: Carsten Ziegeler
> Priority: Critical
>
> the runInBackground method has an endless loop, when
> - the queue is not empty,
> - the item still exists
> - parallelProcessing is false
> - the processing map already has the topic
> - the eventNode is not locked
> i don't know how exactly this state was reached, most probably by a failing JobProcessor.
> note: after the respective nodes are deleted, the endless loop ends :-)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (SLING-462) Endless loop in JobEventHandler
Posted by "Carsten Ziegeler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SLING-462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carsten Ziegeler reassigned SLING-462:
--------------------------------------
Assignee: Carsten Ziegeler
> Endless loop in JobEventHandler
> -------------------------------
>
> Key: SLING-462
> URL: https://issues.apache.org/jira/browse/SLING-462
> Project: Sling
> Issue Type: Bug
> Components: Event
> Affects Versions: 2.0.0
> Reporter: Tobias Bocanegra
> Assignee: Carsten Ziegeler
> Priority: Critical
>
> the runInBackground method has an endless loop, when
> - the queue is not empty,
> - the item still exists
> - parallelProcessing is false
> - the processing map already has the topic
> - the eventNode is not locked
> i don't know how exactly this state was reached, most probably by a failing JobProcessor.
> note: after the respective nodes are deleted, the endless loop ends :-)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SLING-462) Endless loop in JobEventHandler
Posted by "Carsten Ziegeler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SLING-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599577#action_12599577 ]
Carsten Ziegeler commented on SLING-462:
----------------------------------------
Looking at the code, I curently can't see how this state could be reached.
Can you turn on debug logging and try to reproduce this state? If something goes wrong there should be an exception in the log.
> Endless loop in JobEventHandler
> -------------------------------
>
> Key: SLING-462
> URL: https://issues.apache.org/jira/browse/SLING-462
> Project: Sling
> Issue Type: Bug
> Components: Event
> Affects Versions: 2.0.0
> Reporter: Tobias Bocanegra
> Assignee: Carsten Ziegeler
> Priority: Critical
>
> the runInBackground method has an endless loop, when
> - the queue is not empty,
> - the item still exists
> - parallelProcessing is false
> - the processing map already has the topic
> - the eventNode is not locked
> i don't know how exactly this state was reached, most probably by a failing JobProcessor.
> note: after the respective nodes are deleted, the endless loop ends :-)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (SLING-462) Endless loop in JobEventHandler
Posted by Carsten Ziegeler <cz...@apache.org>.
Felix Meschberger wrote:
> Hi,
>
> In this case, I agree that everything is in place and we just need to
> change the default behaviour in case of an uncaught Throwable. I think,
> if a job is not able to self-control, it should not be allowed to be
> rescheduled -- misbehaviour is not accepted ;-)
>
Hehe, yepp, I think you're right - I'll change the implemenation
accordingly.
Carsten
--
Carsten Ziegeler
cziegeler@apache.org
Re: [jira] Commented: (SLING-462) Endless loop in JobEventHandler
Posted by Felix Meschberger <fm...@gmail.com>.
Hi,
In this case, I agree that everything is in place and we just need to
change the default behaviour in case of an uncaught Throwable. I think,
if a job is not able to self-control, it should not be allowed to be
rescheduled -- misbehaviour is not accepted ;-)
Regards
Felix
Am Dienstag, den 27.05.2008, 09:01 +0200 schrieb Carsten Ziegeler:
> Felix Meschberger wrote:
> > Hi,
> >
> > I think, there are two situations with failing jobs: Either it can be
> > retried or it can't. IMHO the default should be to NOT retry. We might
> > add a mechanism for a job to signal, that it might be retried after a
> > certain time interval comparable to the
> > javax.servlet.UnavailableException, which may signal an interval at
> > which the servlet may be retried.
> >
> > Sort of like:
> >
> > * Job failure with RetryException (or so) without a time interval:
> > use default
> > (configurable) value
> > * Job failure with RetryException (or so) with time interval: use the
> > given interval
> > provided it is not too small
> > * Otherwise fail the job alltogether and don't reschedule
> >
> Hmm, ok, you might be right :)
> Currently, the job can signal to be rescheduled by a boolean return
> value, so a job implementation should imho catch all exceptions and then
> decide whether it should be rescheduled or not.
> If the job implementation does not catch an exception, the job handler
> does this and reschedules the job. Maybe we should change this default
> handling.
>
> If the job itself signals that it wants to be rescheduled, it can also
> signal a time interval for rescheduling.
>
> I don't want to use an exception as this binds the job implementation to
> the sling job bundle which is currently not required.
>
> So, if we change the default handling for a job raising an exception we
> should be fine.
>
> WDYT?
> Carsten
Re: [jira] Commented: (SLING-462) Endless loop in JobEventHandler
Posted by Carsten Ziegeler <cz...@apache.org>.
Felix Meschberger wrote:
> Hi,
>
> I think, there are two situations with failing jobs: Either it can be
> retried or it can't. IMHO the default should be to NOT retry. We might
> add a mechanism for a job to signal, that it might be retried after a
> certain time interval comparable to the
> javax.servlet.UnavailableException, which may signal an interval at
> which the servlet may be retried.
>
> Sort of like:
>
> * Job failure with RetryException (or so) without a time interval:
> use default
> (configurable) value
> * Job failure with RetryException (or so) with time interval: use the
> given interval
> provided it is not too small
> * Otherwise fail the job alltogether and don't reschedule
>
Hmm, ok, you might be right :)
Currently, the job can signal to be rescheduled by a boolean return
value, so a job implementation should imho catch all exceptions and then
decide whether it should be rescheduled or not.
If the job implementation does not catch an exception, the job handler
does this and reschedules the job. Maybe we should change this default
handling.
If the job itself signals that it wants to be rescheduled, it can also
signal a time interval for rescheduling.
I don't want to use an exception as this binds the job implementation to
the sling job bundle which is currently not required.
So, if we change the default handling for a job raising an exception we
should be fine.
WDYT?
Carsten
--
Carsten Ziegeler
cziegeler@apache.org
Re: [jira] Commented: (SLING-462) Endless loop in JobEventHandler
Posted by Felix Meschberger <fm...@gmail.com>.
Hi,
I think, there are two situations with failing jobs: Either it can be
retried or it can't. IMHO the default should be to NOT retry. We might
add a mechanism for a job to signal, that it might be retried after a
certain time interval comparable to the
javax.servlet.UnavailableException, which may signal an interval at
which the servlet may be retried.
Sort of like:
* Job failure with RetryException (or so) without a time interval:
use default
(configurable) value
* Job failure with RetryException (or so) with time interval: use the
given interval
provided it is not too small
* Otherwise fail the job alltogether and don't reschedule
WDYT ?
Regards
Felix
Am Montag, den 26.05.2008, 23:37 -0700 schrieb Carsten Ziegeler (JIRA):
> [ https://issues.apache.org/jira/browse/SLING-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600040#action_12600040 ]
>
> Carsten Ziegeler commented on SLING-462:
> ----------------------------------------
>
> Yes, I agree - however scheduling a failing job forever doesn't look like a good idea either :)
> On the other hand, there is a sleep-time for a failing job which is actually not very effective :) Currently if there's only one job in the queue, the thread sleeps for a configured time - so if there is one failing job, it doesn't kill the CPU.
> If there's more than one failing job, no sleep time is used and these failing jobs are retried in sequence - which then of course kills the CPU - maybe we should add a per job sleep time for retrying.
>
> > Endless loop in JobEventHandler
> > -------------------------------
> >
> > Key: SLING-462
> > URL: https://issues.apache.org/jira/browse/SLING-462
> > Project: Sling
> > Issue Type: Bug
> > Components: Event
> > Affects Versions: 2.0.0
> > Reporter: Tobias Bocanegra
> > Assignee: Carsten Ziegeler
> > Priority: Critical
> >
> > the runInBackground method has an endless loop, when
> > - the queue is not empty,
> > - the item still exists
> > - parallelProcessing is false
> > - the processing map already has the topic
> > - the eventNode is not locked
> > i don't know how exactly this state was reached, most probably by a failing JobProcessor.
> > note: after the respective nodes are deleted, the endless loop ends :-)
>
[jira] Commented: (SLING-462) Endless loop in JobEventHandler
Posted by "Carsten Ziegeler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SLING-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600040#action_12600040 ]
Carsten Ziegeler commented on SLING-462:
----------------------------------------
Yes, I agree - however scheduling a failing job forever doesn't look like a good idea either :)
On the other hand, there is a sleep-time for a failing job which is actually not very effective :) Currently if there's only one job in the queue, the thread sleeps for a configured time - so if there is one failing job, it doesn't kill the CPU.
If there's more than one failing job, no sleep time is used and these failing jobs are retried in sequence - which then of course kills the CPU - maybe we should add a per job sleep time for retrying.
> Endless loop in JobEventHandler
> -------------------------------
>
> Key: SLING-462
> URL: https://issues.apache.org/jira/browse/SLING-462
> Project: Sling
> Issue Type: Bug
> Components: Event
> Affects Versions: 2.0.0
> Reporter: Tobias Bocanegra
> Assignee: Carsten Ziegeler
> Priority: Critical
>
> the runInBackground method has an endless loop, when
> - the queue is not empty,
> - the item still exists
> - parallelProcessing is false
> - the processing map already has the topic
> - the eventNode is not locked
> i don't know how exactly this state was reached, most probably by a failing JobProcessor.
> note: after the respective nodes are deleted, the endless loop ends :-)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (SLING-462) Endless loop in JobEventHandler
Posted by "Carsten Ziegeler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SLING-462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carsten Ziegeler closed SLING-462.
----------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
We don't reschedule a job if it fails with an exception per default now - this should solve the problems.
> Endless loop in JobEventHandler
> -------------------------------
>
> Key: SLING-462
> URL: https://issues.apache.org/jira/browse/SLING-462
> Project: Sling
> Issue Type: Bug
> Components: Event
> Affects Versions: 2.0.0
> Reporter: Tobias Bocanegra
> Assignee: Carsten Ziegeler
> Priority: Critical
> Fix For: 2.0.0
>
>
> the runInBackground method has an endless loop, when
> - the queue is not empty,
> - the item still exists
> - parallelProcessing is false
> - the processing map already has the topic
> - the eventNode is not locked
> i don't know how exactly this state was reached, most probably by a failing JobProcessor.
> note: after the respective nodes are deleted, the endless loop ends :-)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.