You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Faisal Khan <fa...@gmail.com> on 2009/07/01 20:38:52 UTC
Task tracker shutdown given inactivity
Hi everyone,
Is it possible to configure Hadoop's TaskTracker in a way that it
shutdown itself once being idle (no task to run) for a configurable
amount of time? I guess it
is not possible in current implementation. This feature is very useful
in situations (like of ours) where a MapReduce jobs can be scheduled
by a batch processing
system. In other words, the ability to schedule and run a single
MapReduce job on a computing cluster that is also handling
non-MapReduce job and vacating
resources as soon as possible is very desirable for our high
throughput community.
Please let me know If you think this is already possible in Hadoop.
Otherwise, we would be happy to work on contributing a patch to
be included in Hadoop to server this purpose.
Thanks for your help.
--
Faisal
Condor Project
Re: Task tracker shutdown given inactivity
Posted by Dhruba Borthakur <dh...@gmail.com>.
You can also decommission taskTrackers when you know that you do not need
it anymore.:
http://issues.apache.org/jira/browse/HADOOP-5643
thanks,
dhruba
On Wed, Jul 1, 2009 at 1:13 PM, Allen Wittenauer <aw...@yahoo-inc.com> wrote:
>
>
>
> On 7/1/09 12:51 PM, "Faisal Khan" <fa...@gmail.com> wrote:
> In other words, the ability to schedule and run a single
> > MapReduce job on a computing cluster that is also handling
> > non-MapReduce job and vacating
> > resources as soon as possible is very desirable for our high
> > throughput community.
>
> Take a look at hod. This is essentially what it does, by letting PBS
> (torque) schedule nodes to run MR and/or HDFS frameworks temporarily.
>
>
Re: Task tracker shutdown given inactivity
Posted by Allen Wittenauer <aw...@yahoo-inc.com>.
On 7/1/09 12:51 PM, "Faisal Khan" <fa...@gmail.com> wrote:
In other words, the ability to schedule and run a single
> MapReduce job on a computing cluster that is also handling
> non-MapReduce job and vacating
> resources as soon as possible is very desirable for our high
> throughput community.
Take a look at hod. This is essentially what it does, by letting PBS
(torque) schedule nodes to run MR and/or HDFS frameworks temporarily.
Task tracker shutdown given inactivity
Posted by Faisal Khan <fa...@gmail.com>.
Hi everyone,
Is it possible to configure Hadoop's TaskTracker in a way that it
shutdown itself once being idle (no task to run) for a configurable
amount of time? I guess it
is not possible in current implementation. This feature is very useful
in situations (like of ours) where a MapReduce jobs can be scheduled
by a batch processing
system. In other words, the ability to schedule and run a single
MapReduce job on a computing cluster that is also handling
non-MapReduce job and vacating
resources as soon as possible is very desirable for our high
throughput community.
Please let me know If you think this is already possible in Hadoop.
Otherwise, we would be happy to work on contributing a patch to
be included in Hadoop to server this purpose.
Thanks for your help.
--
Faisal
Condor Project