Temperature-aware yarn scheduler

[smit <sm...@gmail.com>]

Hi

We are planning to modify YARN to incorporate the idea of temperature-aware
scheduling of jobs. Basically, when a node is heated up, we would like our
scheduler to reduce the workload assigned to that particular node. So the
way we look at this problem, we need help in figuring out the following
things :
- Pathway for passing node's information (mainly temperature) to the
scheduler.
- Mechanism through which scheduler chooses the nodes. (to prevent a heated
node from being selected)
- General overview of the workflow of the yarn code.

We tried asking for help at IRC, and acmurthy suggested us to mail it
directly to the mailing list. It would be very grateful of you if you can
help us in this regard.

Thanking you.

Smit Mehta
Final year, B.Tech
Computer Science and Engineering
IIT Madras
+91-9884996531 || http://www.smitmehta.com

Re: Temperature-aware yarn scheduler

[Sandy Ryza <sa...@cloudera.com>]

Hi Smit,

If by temperature you mean physical temperature, YARN currently does not
collect this information.  Check out NodeStatusUpdaterImpl.java for where
the NodeManagers pass information to the ResourceManager / scheduler.

If by temperature you mean resource (memory/CPU) usage, we recently
implemented something like this in the Fair Scheduler.  Either way,
https://issues.apache.org/jira/browse/YARN-1290 might be a good starting
point for learning the relevant code on the scheduler side.

cheers
-Sandy


On Wed, Dec 4, 2013 at 2:16 AM, smit <sm...@gmail.com> wrote:

> Hi
>
> We are planning to modify YARN to incorporate the idea of temperature-aware
> scheduling of jobs. Basically, when a node is heated up, we would like our
> scheduler to reduce the workload assigned to that particular node. So the
> way we look at this problem, we need help in figuring out the following
> things :
> - Pathway for passing node's information (mainly temperature) to the
> scheduler.
> - Mechanism through which scheduler chooses the nodes. (to prevent a heated
> node from being selected)
> - General overview of the workflow of the yarn code.
>
> We tried asking for help at IRC, and acmurthy suggested us to mail it
> directly to the mailing list. It would be very grateful of you if you can
> help us in this regard.
>
> Thanking you.
>
> Smit Mehta
> Final year, B.Tech
> Computer Science and Engineering
> IIT Madras
> +91-9884996531 || http://www.smitmehta.com
>