You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2007/10/16 11:54:51 UTC
[jira] Created: (HADOOP-2062) Standardize long-running,
daemon-like, threads in hadoop daemons
Standardize long-running, daemon-like, threads in hadoop daemons
----------------------------------------------------------------
Key: HADOOP-2062
URL: https://issues.apache.org/jira/browse/HADOOP-2062
Project: Hadoop
Issue Type: Improvement
Components: dfs, mapred
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Fix For: 0.16.0
There are several long-running, independent, threads in hadoop daemons (atleast in the JobTracker - e.g. ExpireLaunchingTasks, ExpireTrackers, TaskCommitQueue etc.) which need to be alive as long as the daemon itself and hence should be impervious to various errors and exceptions (e.g. HADOOP-2051).
Currently, each of them seem to be hand-crafted (again, specifically the JobTracker) and different from the other.
I propose we standardize on an implementation of a long-running, impervious, daemon-thread which can be used all over the shop. That thread should be explicitly shut-down by the hadoop daemon and shouldn't be vulnerable to any exceptions/errors.
This mostly likely will look like this:
{noformat}
public abstract class DaemonThread extends Thread {
public static final Log LOG = LogFactory.getLog(DaemonThread.class);
{
setDaemon(true); // always a daemon
}
public abstract void innerLoop() throws InterruptedException;
public final void run() {
while (!isInterrupted()) {
try {
innerLoop();
} catch (InterruptedException ie) {
LOG.warn(getName() + " interrupted, exiting...");
} catch (Throwable t) {
LOG.error(getName() + " got an exception: " +
StringUtils.stringifyException(t));
}
}
}
}
{noformat}
In fact, we could probably hijack org.apache.hadoop.util.Daemon since it isn't used anywhere (Doug is it still used in nutch?) or atleast sub-class that.
Thoughts? Could someone from hdfs/hbase chime in?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2062) Standardize long-running,
daemon-like, threads in hadoop daemons
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535249 ]
Enis Soztutar commented on HADOOP-2062:
---------------------------------------
I think we should use java.util.concurrent classes to handle deamon functionality. Especially there are lots of scheduled deamons, which might benefit from a common super type which uses ScheduledExecutorService.
> Standardize long-running, daemon-like, threads in hadoop daemons
> ----------------------------------------------------------------
>
> Key: HADOOP-2062
> URL: https://issues.apache.org/jira/browse/HADOOP-2062
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs, mapred
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.16.0
>
>
> There are several long-running, independent, threads in hadoop daemons (atleast in the JobTracker - e.g. ExpireLaunchingTasks, ExpireTrackers, TaskCommitQueue etc.) which need to be alive as long as the daemon itself and hence should be impervious to various errors and exceptions (e.g. HADOOP-2051).
> Currently, each of them seem to be hand-crafted (again, specifically the JobTracker) and different from the other.
> I propose we standardize on an implementation of a long-running, impervious, daemon-thread which can be used all over the shop. That thread should be explicitly shut-down by the hadoop daemon and shouldn't be vulnerable to any exceptions/errors.
> This mostly likely will look like this:
> {noformat}
> public abstract class DaemonThread extends Thread {
> public static final Log LOG = LogFactory.getLog(DaemonThread.class);
> {
> setDaemon(true); // always a daemon
> }
> public abstract void innerLoop() throws InterruptedException;
>
> public final void run() {
> while (!isInterrupted()) {
> try {
> innerLoop();
> } catch (InterruptedException ie) {
> LOG.warn(getName() + " interrupted, exiting...");
> } catch (Throwable t) {
> LOG.error(getName() + " got an exception: " +
> StringUtils.stringifyException(t));
> }
> }
> }
> }
> {noformat}
> In fact, we could probably hijack org.apache.hadoop.util.Daemon since it isn't used anywhere (Doug is it still used in nutch?) or atleast sub-class that.
> Thoughts? Could someone from hdfs/hbase chime in?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2062) Standardize long-running,
daemon-like, threads in hadoop daemons
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535309 ]
Doug Cutting commented on HADOOP-2062:
--------------------------------------
> we could probably hijack org.apache.hadoop.util.Daemon since it isn't used anywhere
It does not appear to be used in Nutch, but it is used extensively in the dfs package. To be safest we might deprecate it and replace it with DaemonThread.
> Standardize long-running, daemon-like, threads in hadoop daemons
> ----------------------------------------------------------------
>
> Key: HADOOP-2062
> URL: https://issues.apache.org/jira/browse/HADOOP-2062
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs, mapred
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.16.0
>
>
> There are several long-running, independent, threads in hadoop daemons (atleast in the JobTracker - e.g. ExpireLaunchingTasks, ExpireTrackers, TaskCommitQueue etc.) which need to be alive as long as the daemon itself and hence should be impervious to various errors and exceptions (e.g. HADOOP-2051).
> Currently, each of them seem to be hand-crafted (again, specifically the JobTracker) and different from the other.
> I propose we standardize on an implementation of a long-running, impervious, daemon-thread which can be used all over the shop. That thread should be explicitly shut-down by the hadoop daemon and shouldn't be vulnerable to any exceptions/errors.
> This mostly likely will look like this:
> {noformat}
> public abstract class DaemonThread extends Thread {
> public static final Log LOG = LogFactory.getLog(DaemonThread.class);
> {
> setDaemon(true); // always a daemon
> }
> public abstract void innerLoop() throws InterruptedException;
>
> public final void run() {
> while (!isInterrupted()) {
> try {
> innerLoop();
> } catch (InterruptedException ie) {
> LOG.warn(getName() + " interrupted, exiting...");
> } catch (Throwable t) {
> LOG.error(getName() + " got an exception: " +
> StringUtils.stringifyException(t));
> }
> }
> }
> }
> {noformat}
> In fact, we could probably hijack org.apache.hadoop.util.Daemon since it isn't used anywhere (Doug is it still used in nutch?) or atleast sub-class that.
> Thoughts? Could someone from hdfs/hbase chime in?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2062) Standardize long-running,
daemon-like, threads in hadoop daemons
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535323 ]
Raghu Angadi commented on HADOOP-2062:
--------------------------------------
Just one more : How about naming it DaemonLoop() which makes it obvious to the user that it runs their code in a loop.. and clearly distinguishes it from Daemon().
> Standardize long-running, daemon-like, threads in hadoop daemons
> ----------------------------------------------------------------
>
> Key: HADOOP-2062
> URL: https://issues.apache.org/jira/browse/HADOOP-2062
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs, mapred
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.16.0
>
>
> There are several long-running, independent, threads in hadoop daemons (atleast in the JobTracker - e.g. ExpireLaunchingTasks, ExpireTrackers, TaskCommitQueue etc.) which need to be alive as long as the daemon itself and hence should be impervious to various errors and exceptions (e.g. HADOOP-2051).
> Currently, each of them seem to be hand-crafted (again, specifically the JobTracker) and different from the other.
> I propose we standardize on an implementation of a long-running, impervious, daemon-thread which can be used all over the shop. That thread should be explicitly shut-down by the hadoop daemon and shouldn't be vulnerable to any exceptions/errors.
> This mostly likely will look like this:
> {noformat}
> public abstract class DaemonThread extends Thread {
> public static final Log LOG = LogFactory.getLog(DaemonThread.class);
> {
> setDaemon(true); // always a daemon
> }
> public abstract void innerLoop() throws InterruptedException;
>
> public final void run() {
> while (!isInterrupted()) {
> try {
> innerLoop();
> } catch (InterruptedException ie) {
> LOG.warn(getName() + " interrupted, exiting...");
> } catch (Throwable t) {
> LOG.error(getName() + " got an exception: " +
> StringUtils.stringifyException(t));
> }
> }
> }
> }
> {noformat}
> In fact, we could probably hijack org.apache.hadoop.util.Daemon since it isn't used anywhere (Doug is it still used in nutch?) or atleast sub-class that.
> Thoughts? Could someone from hdfs/hbase chime in?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2062) Standardize long-running,
daemon-like, threads in hadoop daemons
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-2062:
----------------------------------
Fix Version/s: (was: 0.16.0)
0.17.0
I'm moving this to 0.17.0.
> Standardize long-running, daemon-like, threads in hadoop daemons
> ----------------------------------------------------------------
>
> Key: HADOOP-2062
> URL: https://issues.apache.org/jira/browse/HADOOP-2062
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs, mapred
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.17.0
>
>
> There are several long-running, independent, threads in hadoop daemons (atleast in the JobTracker - e.g. ExpireLaunchingTasks, ExpireTrackers, TaskCommitQueue etc.) which need to be alive as long as the daemon itself and hence should be impervious to various errors and exceptions (e.g. HADOOP-2051).
> Currently, each of them seem to be hand-crafted (again, specifically the JobTracker) and different from the other.
> I propose we standardize on an implementation of a long-running, impervious, daemon-thread which can be used all over the shop. That thread should be explicitly shut-down by the hadoop daemon and shouldn't be vulnerable to any exceptions/errors.
> This mostly likely will look like this:
> {noformat}
> public abstract class DaemonThread extends Thread {
> public static final Log LOG = LogFactory.getLog(DaemonThread.class);
> {
> setDaemon(true); // always a daemon
> }
> public abstract void innerLoop() throws InterruptedException;
>
> public final void run() {
> while (!isInterrupted()) {
> try {
> innerLoop();
> } catch (InterruptedException ie) {
> LOG.warn(getName() + " interrupted, exiting...");
> } catch (Throwable t) {
> LOG.error(getName() + " got an exception: " +
> StringUtils.stringifyException(t));
> }
> }
> }
> }
> {noformat}
> In fact, we could probably hijack org.apache.hadoop.util.Daemon since it isn't used anywhere (Doug is it still used in nutch?) or atleast sub-class that.
> Thoughts? Could someone from hdfs/hbase chime in?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2062) Standardize long-running,
daemon-like, threads in hadoop daemons
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535318 ]
Raghu Angadi commented on HADOOP-2062:
--------------------------------------
oops! this is not a drop in replacement for Daemon.. because when run() returns, it is invoked again. Should innerLoop() be called something like innerBlock() :) ? Also similar to Thread(), it could take a DaemonRunnable (that requires implementation of innerLoop()) as opposed to requiring classes to extend a Thread.
> Standardize long-running, daemon-like, threads in hadoop daemons
> ----------------------------------------------------------------
>
> Key: HADOOP-2062
> URL: https://issues.apache.org/jira/browse/HADOOP-2062
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs, mapred
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.16.0
>
>
> There are several long-running, independent, threads in hadoop daemons (atleast in the JobTracker - e.g. ExpireLaunchingTasks, ExpireTrackers, TaskCommitQueue etc.) which need to be alive as long as the daemon itself and hence should be impervious to various errors and exceptions (e.g. HADOOP-2051).
> Currently, each of them seem to be hand-crafted (again, specifically the JobTracker) and different from the other.
> I propose we standardize on an implementation of a long-running, impervious, daemon-thread which can be used all over the shop. That thread should be explicitly shut-down by the hadoop daemon and shouldn't be vulnerable to any exceptions/errors.
> This mostly likely will look like this:
> {noformat}
> public abstract class DaemonThread extends Thread {
> public static final Log LOG = LogFactory.getLog(DaemonThread.class);
> {
> setDaemon(true); // always a daemon
> }
> public abstract void innerLoop() throws InterruptedException;
>
> public final void run() {
> while (!isInterrupted()) {
> try {
> innerLoop();
> } catch (InterruptedException ie) {
> LOG.warn(getName() + " interrupted, exiting...");
> } catch (Throwable t) {
> LOG.error(getName() + " got an exception: " +
> StringUtils.stringifyException(t));
> }
> }
> }
> }
> {noformat}
> In fact, we could probably hijack org.apache.hadoop.util.Daemon since it isn't used anywhere (Doug is it still used in nutch?) or atleast sub-class that.
> Thoughts? Could someone from hdfs/hbase chime in?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2062) Standardize long-running,
daemon-like, threads in hadoop daemons
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535312 ]
Raghu Angadi commented on HADOOP-2062:
--------------------------------------
> It does not appear to be used in Nutch, but it is used extensively in the dfs package. To be safest we might deprecate it and replace it with DaemonThread.
I think DaemonThread could just extend Daemon (or include getRunnable()) and not require changing implementation of classes (to rename run() to innerLoop(), extend DaemonThread instead of Runnable). This way one can just replace Daemon(...) with DaemonThread(...).
> Standardize long-running, daemon-like, threads in hadoop daemons
> ----------------------------------------------------------------
>
> Key: HADOOP-2062
> URL: https://issues.apache.org/jira/browse/HADOOP-2062
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs, mapred
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.16.0
>
>
> There are several long-running, independent, threads in hadoop daemons (atleast in the JobTracker - e.g. ExpireLaunchingTasks, ExpireTrackers, TaskCommitQueue etc.) which need to be alive as long as the daemon itself and hence should be impervious to various errors and exceptions (e.g. HADOOP-2051).
> Currently, each of them seem to be hand-crafted (again, specifically the JobTracker) and different from the other.
> I propose we standardize on an implementation of a long-running, impervious, daemon-thread which can be used all over the shop. That thread should be explicitly shut-down by the hadoop daemon and shouldn't be vulnerable to any exceptions/errors.
> This mostly likely will look like this:
> {noformat}
> public abstract class DaemonThread extends Thread {
> public static final Log LOG = LogFactory.getLog(DaemonThread.class);
> {
> setDaemon(true); // always a daemon
> }
> public abstract void innerLoop() throws InterruptedException;
>
> public final void run() {
> while (!isInterrupted()) {
> try {
> innerLoop();
> } catch (InterruptedException ie) {
> LOG.warn(getName() + " interrupted, exiting...");
> } catch (Throwable t) {
> LOG.error(getName() + " got an exception: " +
> StringUtils.stringifyException(t));
> }
> }
> }
> }
> {noformat}
> In fact, we could probably hijack org.apache.hadoop.util.Daemon since it isn't used anywhere (Doug is it still used in nutch?) or atleast sub-class that.
> Thoughts? Could someone from hdfs/hbase chime in?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.