[jira] [Created] (REEF-1383) Improve MultiRuntimeConfigurationBuilder API

["Markus Weimer (JIRA)" <ji...@apache.org>]

Markus Weimer created REEF-1383:
-----------------------------------

             Summary: Improve MultiRuntimeConfigurationBuilder API
                 Key: REEF-1383
                 URL: https://issues.apache.org/jira/browse/REEF-1383
             Project: REEF
          Issue Type: Sub-task
            Reporter: Markus Weimer


I had another look at {{MultiRuntimeConfigurationBuilder}}. It now seems very counter-intuitive to me. Look at the code in {{HelloREEFMultiYarn.getHybridYarnSubmissionRuntimeConfiguration}}:

{code}
return new MultiRuntimeConfigurationBuilder()
            .setDefaultRuntime(org.apache.reef.runtime.yarn.driver.RuntimeIdentifier.RUNTIME_NAME)
            .setSubmissionRuntime(org.apache.reef.runtime.yarn.driver.RuntimeIdentifier.RUNTIME_NAME)
            .addRuntime(org.apache.reef.runtime.local.driver.RuntimeIdentifier.RUNTIME_NAME)
            .addRuntime(org.apache.reef.runtime.yarn.driver.RuntimeIdentifier.RUNTIME_NAME)
            .setMaxEvaluatorsNumberForLocalRuntime(1)
            .build();
{code}

At no point does the user actually configure the runtimes in question. For instance, what if they want to submit to a YARN instance which needs a correct user name to work? There doesn't seem to be a way to specify that. What if I want to support Mesos next? This tight coupling of the multi-runtime with the (small) set of supported participants creates an explosion in the complexity of all the code associated with this change. What prevents us from creating an API that makes the client code look more like this:

{code}
return new MultiRuntimeConfigurationBuilder()
  .addSubmissionRuntime(YARN, YarnClientConfiguration.CONF.build())
  .addDefaultRuntime(YARN, YARNDriverConfiguration.CONF.build())
  .addRuntime(LOCAL, LocalDriverConfiguration.CONF.set(NUMBER_OF_EVALUATORS, 1).build())
  .build();
{code}

This would allow an arbitrary set of runtimes to take part in the multi-runtime. Also, it would remove all special-casing in the multi-runtime for specific (combinations of) runtimes. This should simplify the code in the multi-runtime tremendously.

On the flip side, runtimes now need to implement those APIs, namely {{ConfigurationBuilder}}s for the Driver side and the Client side separately. That should be doable and has been introduced via the {{Extensible*}} builders done as part of this work.

WDYT?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)