Mechanism to Launch reef evaluators inside the spark environment

[Saikat Kanjilal <sx...@gmail.com>]

Hello Reef folks,
I've been researching the most efficient method to launch reef evaluators
on top of spark and I'm currently leaning towards using the following
method:

try (final REEFEnvironment reef =
    REEFEnvironment.fromConfiguration(client.getUser(), yarnAmConfig,
DRIVER_CONFIG)) {

  reef.run();

  final ReefServiceProtos.JobStatusProto status = reef.getLastStatus();

  LOG.log(Level.INFO, "REEF-on-REEF inner job {0} completed: state {1}",
      new Object[] {innerApplicationId, status.getState()});
}


However I am wondering if this might have some unintended consequences
when running inside the yarn container that is spawned by the
appication master, any thoughts on this .

Are there other methods I should consider as well?


Would really appreciate some input as this is blocking me from running
this end to end.

Thanks in advance for your help.

Re: Mechanism to Launch reef evaluators inside the spark environment

[Saikat Kanjilal <sx...@gmail.com>]

Hi Markus,
Actually I researched this some more and am moving forward using the
following code:

try (final REEFEnvironment reef =
REEFEnvironment.fromConfiguration(dataLoadConfiguration)) {
  reef.run();
} catch (final Exception ex) {
  //throw fatal("Unable to configure and start REEFEnvironment.", ex);
  LOG.log(Level.FINEST, "SparkRunner::caught an exception with trace="
+ ex.getMessage());
}

The dataLoadConfiguration above instantiates a group of evaluators.
Also when I look at the comments associated with the run method for
the REEFEnvironment it says that the run method is used to launch a
reef component (evaluator or Driver)

I have refactored the workflow to execute the above code from the
SparkRunner and am getting ready to test this end to end on a local
spark on yarn setup.

Thoughts?



On Mon, Dec 4, 2017 at 10:53 AM, Markus Weimer <ma...@weimo.de> wrote:

> Hi,
>
>
> On Fri, Dec 1, 2017 at 4:51 PM, Saikat Kanjilal <sx...@gmail.com> wrote:
>
> >
> > try (final REEFEnvironment reef =
> >     REEFEnvironment.fromConfiguration(client.getUser(), yarnAmConfig,
> > DRIVER_CONFIG)) {
> >
>
> Wouldn't this launch a Driver?
>
> Markus
>

Re: Mechanism to Launch reef evaluators inside the spark environment

[Markus Weimer <ma...@weimo.de>]

Hi,


On Fri, Dec 1, 2017 at 4:51 PM, Saikat Kanjilal <sx...@gmail.com> wrote:

>
> try (final REEFEnvironment reef =
>     REEFEnvironment.fromConfiguration(client.getUser(), yarnAmConfig,
> DRIVER_CONFIG)) {
>

Wouldn't this launch a Driver?

Markus