You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2009/09/23 19:00:15 UTC
[jira] Created: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
DebuggableScheduledThreadPoolExecutor only schedules a task once
----------------------------------------------------------------
Key: CASSANDRA-455
URL: https://issues.apache.org/jira/browse/CASSANDRA-455
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.5
Reporter: Jun Rao
DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-455:
-------------------------------------
Attachment: 455.patch
Replace DebuggableScheduledThreadPoolExecutor with non-Scheduled Executors and Timers. This allows logging
exceptions from repeated tasks, which is basically impossible with STPE.
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.4, 0.5
> Reporter: Jun Rao
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 455.patch
>
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759738#action_12759738 ]
Jonathan Ellis commented on CASSANDRA-455:
------------------------------------------
performed the revert on the 0.4 branch.
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.4, 0.5
> Reporter: Jun Rao
> Fix For: 0.5
>
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760307#action_12760307 ]
Jun Rao commented on CASSANDRA-455:
-----------------------------------
Looks good to me. Perhaps we should add some logging for the starting of each scheduled task.
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.4, 0.5
> Reporter: Jun Rao
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 455.patch
>
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759754#action_12759754 ]
Jonathan Ellis commented on CASSANDRA-455:
------------------------------------------
I took a stab at using STPE.decorateTask to make a new ScheduledFuture that doesn't eat exceptions but it's going to be messy and fragile.
Ripping that crap out and using old-school Timers looks like a better option.
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.4, 0.5
> Reporter: Jun Rao
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-455:
----------------------------------------
Assignee: Jonathan Ellis
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.4, 0.5
> Reporter: Jun Rao
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760333#action_12760333 ]
Jonathan Ellis commented on CASSANDRA-455:
------------------------------------------
both flushing and HHO log as soon as they start so I think adding logging at the timer level clutters things up more than clarifies. (maybe if we decide we have too many Timer threads around and consolidate into a global service, though.)
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.4, 0.5
> Reporter: Jun Rao
> Assignee: Jonathan Ellis
> Fix For: 0.5
>
> Attachments: 455.patch
>
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-455:
-------------------------------------
Fix Version/s: 0.5
Affects Version/s: 0.4
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.4, 0.5
> Reporter: Jun Rao
> Fix For: 0.5
>
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759150#action_12759150 ]
Jonathan Ellis commented on CASSANDRA-455:
------------------------------------------
then i guess we should always reschedule after checking for exceptions.
but i'd be more comfortable if i saw the code that is un-scheduling it, so i could be sure there are no conditions under which we could end up w/ 2 copies of the same task scheduled.
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Reporter: Jun Rao
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759139#action_12759139 ]
Jun Rao commented on CASSANDRA-455:
-----------------------------------
It seems that one needs to call DebuggableThreadPoolExecutor.logFutureExceptions(r) to catch any exception while running the scheduled task. However, once get() is called on the FutureTask, the task is no longer scheduled any more. Anyone knows how do address this?
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Reporter: Jun Rao
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759734#action_12759734 ]
Jonathan Ellis commented on CASSANDRA-455:
------------------------------------------
ScheduledFutureTask is private. @Q#%$
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.4, 0.5
> Reporter: Jun Rao
> Fix For: 0.5
>
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759143#action_12759143 ]
Jonathan Ellis commented on CASSANDRA-455:
------------------------------------------
scheduleAtFixedRate "Creates and executes a periodic action that becomes enabled first after the given initial delay, and subsequently with the given period; that is executions will commence after initialDelay then initialDelay+period, then initialDelay + 2 * period, and so on. If any execution of the task encounters an exception, subsequent executions are suppressed..."
so get() shouldn't be causing a cancel, but if an exception is found then we need to re-schedule it manually. (If you cast the Runnable in afterExecute to ScheduledFutureTask you can get access to the scheduling info.)
if get is causing the cancel even w/o any exceptions being involved then I guess you'll need to source dive in ScheduledThreadPoolExecutor to see what is going on.
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Reporter: Jun Rao
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759728#action_12759728 ]
Jonathan Ellis commented on CASSANDRA-455:
------------------------------------------
Okay, here's what's going on.
ScheduledFutureTask overrides FT.run as follows:
public void run() {
if (isPeriodic())
runPeriodic();
else
ScheduledFutureTask.super.run();
}
we are scheduling periodic tasks, so the normal run() method, which just wrapps sync.innerRun(), is not called. Instead we call runPeriodic, of which the important part here is
boolean ok = ScheduledFutureTask.super.runAndReset();
not run()! runAndReset()! which says
/**
* Executes the computation without setting its result, and then
* resets this Future to initial state, failing to do so if the
* computation encounters an exception or is cancelled. This is
* designed for use with tasks that intrinsically execute more
* than once.
* @return true if successfully run and reset
*/
protected boolean runAndReset() {
return sync.innerRunAndReset();
}
key point is "... without setting its result." sure enough, sync.innerRAR sets the state back to 0 when it is done -- but get() will block until state is RAN or CANCELLED, i.e. until there is a result. So the reason get() makes the task stop executing is it deadlocks the executor thread that it's running on.
as far as i can see there is no public way to get an exception out of a FutureTask.sync w/o calling get. I guess we can get it out using reflection tho.
For 0.4 I will just revert the commit that added logFutureExceptions to DebuggableScheduledThreadPoolExecutor.
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.4, 0.5
> Reporter: Jun Rao
> Fix For: 0.5
>
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758776#action_12758776 ]
Jun Rao commented on CASSANDRA-455:
-----------------------------------
The problem seems to be in DebuggableScheduledThreadPoolExecutor.afterExecute(). If I remove the line
DebuggableThreadPoolExecutor.logFutureExceptions(r);
then all tasks are scheduled periodically as expected. Not sure if this is the right fix though.
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Reporter: Jun Rao
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-455)
DebuggableScheduledThreadPoolExecutor only schedules a task once
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759146#action_12759146 ]
Jun Rao commented on CASSANDRA-455:
-----------------------------------
"if get is causing the cancel even w/o any exceptions being involved then I guess you'll need to source dive in ScheduledThreadPoolExecutor to see what is going on."
That seems to be the case. Here is what happens.
Without exception:
When DebuggableThreadPoolExecutor.logFutureExceptions(r) is called in afterExecute(), tasks are no longer executed afterwards. When DebuggableThreadPoolExecutor.logFutureExceptions(r) is removed, tasks are scheduled as expected.
With exception:
If DebuggableThreadPoolExecutor.logFutureExceptions(r) is removed, exception is not logged.
> DebuggableScheduledThreadPoolExecutor only schedules a task once
> ----------------------------------------------------------------
>
> Key: CASSANDRA-455
> URL: https://issues.apache.org/jira/browse/CASSANDRA-455
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Reporter: Jun Rao
>
> DebuggableScheduledThreadPoolExecutor only schedules a task exactly once, instead of periodically. This affects scheduled flushers and periodic hints delivery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.