Yet Another Scheduling Idea

[santosh kumar <sa...@gmail.com>]

Hi,

I am trying to develop an auction based scheduling algorithm for YARN. I
have few questions about the same.

Idea is to have an "Auction Manager" inside every "Node Manager" who
receive auction requests from "Resource Manager" for every resource request
from "Application Master". The cluster nodes are grouped such that each
rack has one seller and all other as participants. Based on few
parameters/metrics the seller selects a winning participant to serve the
resource request.

Question 1:
--------------
I added a new module for auctions in my local code repository and pretty
much covered most of the use cases. But I am stuck at understanding how
communication actually takes place between RM and NM or among NMs. I need
to include something on lines of send/recv commands. Like how the
map/reduce commands are given using some job.xml.

Question 2:
--------------
All my code additions are in:
hadoop-yarn-api; hadoop-yarn-common; hadoop-yarn-server-common;
hadoop-yarn-server-nodemanager; hadoop-yarn-server-resourcemanager

Would I need other modules like hadoop-yarn-client etc? My guess is I don't
think so.

Question 3:
--------------
Did not really understand the Yarn-Protos. Any place where I can get some
decent documentation about this?


Any response/help is greatly appreciated.

Thanks,
Santosh

Re: Yet Another Scheduling Idea

[Steve Loughran <st...@hortonworks.com>]

There's an auction/bid scheduler in the contrib library for Hadoop 2.20, I
think you'd be better off working on getting that working with Hadoop 2.x,
including the pre-emption feature, where someone bidding more can get
lower-priced work killed.

https://github.com/apache/hadoop-common/tree/branch-0.21/mapreduce/src/contrib/dynamic-scheduler

Trying to do fundamental changes to how the NM/RM system works is a major
undertaking that I would not encourage you to attempt until you've spent a
lot of time in the YARN codebase understanding how things glue together. At
the very least, try writing your own YARN application with protobuf-based
RPCs & get it working on a secure cluster, which I can assure you gets you
into some of the issues. You seem to have got pretty far already, but
there's still a lot left to go

The scheduler plugin point is designed for third party schedulers, and you
should focus on there: if there is information you need from the nodes,
such as recent CPU, IO and RAM load, then those are enhancements that could
be added to NM heartbeats -for the benefit of all schedulers.



On 9 December 2013 20:43, santosh kumar <sa...@gmail.com> wrote:

> Hi,
>
> I am trying to develop an auction based scheduling algorithm for YARN. I
> have few questions about the same.
>
> Idea is to have an "Auction Manager" inside every "Node Manager" who
> receive auction requests from "Resource Manager" for every resource request
> from "Application Master". The cluster nodes are grouped such that each
> rack has one seller and all other as participants. Based on few
> parameters/metrics the seller selects a winning participant to serve the
> resource request.
>
> Question 1:
> --------------
> I added a new module for auctions in my local code repository and pretty
> much covered most of the use cases. But I am stuck at understanding how
> communication actually takes place between RM and NM or among NMs. I need
> to include something on lines of send/recv commands. Like how the
> map/reduce commands are given using some job.xml.
>
> Question 2:
> --------------
> All my code additions are in:
> hadoop-yarn-api; hadoop-yarn-common; hadoop-yarn-server-common;
> hadoop-yarn-server-nodemanager; hadoop-yarn-server-resourcemanager
>
> Would I need other modules like hadoop-yarn-client etc? My guess is I don't
> think so.
>
> Question 3:
> --------------
> Did not really understand the Yarn-Protos. Any place where I can get some
> decent documentation about this?
>
>
> Any response/help is greatly appreciated.
>
> Thanks,
> Santosh
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.