How to implement a REEF extension

[Youngeun Cho <ye...@rubis.snu.ac.kr>]

To whom it may concern,

We write this email to ask for your guide to our REEF extension.

We are trying to implement an interference-aware multiple job scheduler on
REEF.

To achieve this, we composed our architecture as two parts.
First, we profile each job's execution, and according to it, we allocate
the job to the best available resource.

We are trying to modify the REEF source code to attach our profiler
implementation.
However we are struggling to find the good starting place to insert our
code.

We think the best place to insert our code, may be where REEF calls lower
resource manager to allocate resources for the job.

Could you kindly let us know where(name and location of the file) to modify
the source code?

Thank you.

Youngeun Cho,Youngsu Cho, and Joongsuk Park

Department of Computer Science,
Seoul National University

Re: How to implement a REEF extension

[Markus Weimer <ma...@weimo.de>]

Welcome to the dev list! Please consider subscribing, as we otherwise have
to moderate each of your messages, which will delay their posting :)

On Thu, Nov 2, 2017 at 4:13 AM, Youngeun Cho <ye...@rubis.snu.ac.kr> wrote:

> We think the best place to insert our code, may be where REEF calls lower
> resource manager to allocate resources for the job.
>
> Could you kindly let us know where(name and location of the file) to modify
> the source code?
>

This code is different for each resource manager REEF supports. It is
captured in the Maven modules starting with `reef-runtime`, e.g.
`reef-runtime-yarn`. Each of the runtimes implements the following
interfaces:

   - `o.a.r.runtime.common.driver.api.ResourceLaunchHandler`: For launching
   containers
   - `o.a.r.runtime.common.driver.api.ResourceReleaseHandler`: For
   returning containers to the underlying resource manager
   - `o.a.r.runtime.common.driver.api.ResourceRequestHandler`: For
   requesting containers from the resource manager

In general, the package `org.apache.reef.runtime.common.driver.api` is a
worthwhile starting point to learn more about the interface between REEF
and a particular RM on the Driver.

Hope this helps,

Markus