using NamedParameter to implement a no-arg boolean command-line option

[Tobin Baker <td...@cs.washington.edu>]

Hi, I'm trying to use CommandLine.parseToConfiguration() to implement an
implicit boolean command-line option, where the boolean value is understood
to be true if the option is present, and false otherwise. I.e., I want
something that looks like this:

java -cp '/Users/tdbaker/myria/build/libs/*'
edu.washington.escience.myria.daemon.MyriaDriverLauncher -yarn

where I can set a boolean variable useYarn based on whether -yarn is
present or not. Obviously I could just do

java -cp '/Users/tdbaker/myria/build/libs/*'
edu.washington.escience.myria.daemon.MyriaDriverLauncher -yarn true

using Name<Boolean> and getNamedInstance(), but I'd rather not for
ergonomics' sake.

I can't seem to find the right combination of Name<Boolean>, Name<Void>,
default_value, or isParameterSet() to achieve this behavior, and I can't
find examples that implement this command-line behavior either. Any
suggestions?

Thanks,

Tobin

Re: using NamedParameter to implement a no-arg boolean command-line option

[Markus Weimer <ma...@weimo.de>]

On 2015-12-04 14:30, Tobin Baker wrote:
> Are you suggesting that if we had something like standardized
> "runtime strings", those could be specified on the command line, and
> the implementer of the main class wouldn't need to know which
> runtimes were supported?

I wasn't thinking of that, but: If your code has a `static void
runMyria(REEF reef)`, then anyone can instantiate an instance of `REEF`
and call into it, e.g. on mesos.

The goal of separating out the runtime configuration from the actual 
application as part of the REEF abstractions seems like a worthy one, 
though.

We faced that issue when preparing examples for the Machine Learning 
Summer School 2014. The solution we came up was as follows: There is one 
static method `final static REEF getREEF()` which uses environment 
variables to assemble the configuration for the runtime and instantiate 
it. In that case, we had an environment variable which pointed to a REEF 
runtime configuration stored in a JSON file.

Maybe it is time we added that to REEF proper?

Markus

Re: using NamedParameter to implement a no-arg boolean command-line option

[Tobin Baker <td...@cs.washington.edu>]

Yes, I believe that's exactly what I'm doing now (inspired by the
examples). My main() method calls a static run() method which instantiates
the main class using a particular runtime configuration, which determines
the REEF instance that is injected in the main class's constructor. Then I
call an instance run() method which submits the driver to the injected REEF
instance. The static run() method just needs to know which runtime to use
for injecting the REEF instance, which it currently learns from
command-line configuration.

Are you suggesting that if we had something like standardized "runtime
strings", those could be specified on the command line, and the implementer
of the main class wouldn't need to know which runtimes were supported? I
suppose I could just take a fully qualified class name on the command line
for now (e.g.,
org.apache.reef.runtime.yarn.client.YarnClientConfiguration,
org.apache.reef.runtime.local.client.LocalRuntimeConfiguration)?
At least that would be less ad-hoc than my current approach, even if the
class names are a bit verbose and arbitrary.

On Fri, Dec 4, 2015 at 1:48 PM, Markus Weimer <ma...@weimo.de> wrote:

> On 2015-12-04 13:23, Tobin Baker wrote:
> > Your workaround of using different main classes for each runtime is
> > probably better than introducing a command-line option to choose the
> > runtime.
>
> If you do it properly, it also means that *someone else* can port your
> app to other resource managers:
>
> You could have a main class, e.g. `MyriaREEFMain` that expects a `REEF`
> instance to be injected. In its `run()` method, it submits the Driver to
> REEF. Now, instantiate `MyriaREEFMain` from the various entry points
> using different runtime configurations.
>
> Does this make sense?
>
> Markus
>

Re: using NamedParameter to implement a no-arg boolean command-line option

[Markus Weimer <ma...@weimo.de>]

On 2015-12-04 13:23, Tobin Baker wrote:
> Your workaround of using different main classes for each runtime is
> probably better than introducing a command-line option to choose the
> runtime.

If you do it properly, it also means that *someone else* can port your
app to other resource managers:

You could have a main class, e.g. `MyriaREEFMain` that expects a `REEF`
instance to be injected. In its `run()` method, it submits the Driver to
REEF. Now, instantiate `MyriaREEFMain` from the various entry points
using different runtime configurations.

Does this make sense?

Markus

Re: using NamedParameter to implement a no-arg boolean command-line option

[Tobin Baker <td...@cs.washington.edu>]

That all sounds great, I just don't really have a motivating scenario at
the moment that requires detailed information on the runtime/environment,
and I assume much of the environment information can be gathered through
Java APIs. If someone on the list has such a scenario, it would be great to
inform this design with some real-world requirements.

Your workaround of using different main classes for each runtime is
probably better than introducing a command-line option to choose the
runtime.

On Fri, Dec 4, 2015 at 10:48 AM, Markus Weimer <ma...@weimo.de> wrote:

> On 2015-12-04 09:53, Tobin Baker wrote:
> > The need is simple: if my launcher is invoked by "yarn jar", I should
> > use the YARN runtime, and the local runtime otherwise.
>
> Ah, yes, that makes sense. We've so far "solved" this with two different
> `main()` methods and classes: One for local runs, one for YARN runs.
>
> > I like your idea of a "runtime string" exposed by appropriate
> > interfaces in the client/driver/evaluators.
>
> Maybe we can do even better than that by providing structured
> information via those interfaces, including:
>
>   * Runtime name
>   * Runtime version
>   * A string describing the environment (e.g. Linux, Windows, ...))
>   * A string that captures the main configuration parameters, for
>     logs etc.
>   * A runtime string that describes all of the above.
>
> How does this sound?
>
> Markus
>

Re: using NamedParameter to implement a no-arg boolean command-line option

[Markus Weimer <ma...@weimo.de>]

On 2015-12-04 09:53, Tobin Baker wrote:
> The need is simple: if my launcher is invoked by "yarn jar", I should
> use the YARN runtime, and the local runtime otherwise.

Ah, yes, that makes sense. We've so far "solved" this with two different
`main()` methods and classes: One for local runs, one for YARN runs.

> I like your idea of a "runtime string" exposed by appropriate
> interfaces in the client/driver/evaluators.

Maybe we can do even better than that by providing structured
information via those interfaces, including:

  * Runtime name
  * Runtime version
  * A string describing the environment (e.g. Linux, Windows, ...))
  * A string that captures the main configuration parameters, for
    logs etc.
  * A runtime string that describes all of the above.

How does this sound?

Markus

Re: using NamedParameter to implement a no-arg boolean command-line option

[Tobin Baker <td...@cs.washington.edu>]

The need is simple: if my launcher is invoked by "yarn jar", I should use
the YARN runtime, and the local runtime otherwise. I was thinking of
testing for YARN configuration using the YarnConfiguration default
constructor, but that seems like a fragile hack. Currently my driver- or
evaluator-based code doesn't need to know which runtime is already in use,
but I could see that being important for cases where REEF doesn't expose
some YARN functionality (such as, until recently, node/rack locality). I
imagine Mesos users might have similar needs. I like your idea of a
"runtime string" exposed by appropriate interfaces in the
client/driver/evaluators.


On Fri, Dec 4, 2015 at 9:26 AM, Markus Weimer <ma...@weimo.de> wrote:

> On 2015-12-03 18:54, Tobin Baker wrote:
>
>> I'm looking for a way to detect my code is running under YARN
>>
>
> Can you elaborate on where that need is coming from? In an ideal world,
> your app wouldn't have to know. Of course, no one lives in that world :-)
>
> I can imagine a few hacky ways to do this, e.g. by using multiple
> constructors, some of which can only be invoked by Tang if YARN is
> configured. I would advise not to do that :-)
>
> If we wanted to make this a first class thing in REEF, we could have the
> concept of a "Runtime String" in much the same way of a user agent in
> browsers. That would allow us to keep the set of runtimes open while
> also giving users access to it.
>
> On the client, we could expose a new method in
> `org.apache.reef.client.REEF` to get access to it. On the Driver, we'd
> probably want to create a new interface `RuntimeInformation` or such.
>
> Markus
>
>

Re: using NamedParameter to implement a no-arg boolean command-line option

[Markus Weimer <ma...@weimo.de>]

On 2015-12-03 18:54, Tobin Baker wrote:
> I'm looking for a way to detect my code is running under YARN

Can you elaborate on where that need is coming from? In an ideal world,
your app wouldn't have to know. Of course, no one lives in that world :-)

I can imagine a few hacky ways to do this, e.g. by using multiple
constructors, some of which can only be invoked by Tang if YARN is
configured. I would advise not to do that :-)

If we wanted to make this a first class thing in REEF, we could have the
concept of a "Runtime String" in much the same way of a user agent in
browsers. That would allow us to keep the set of runtimes open while
also giving users access to it.

On the client, we could expose a new method in
`org.apache.reef.client.REEF` to get access to it. On the Driver, we'd 
probably want to create a new interface `RuntimeInformation` or such.

Markus

Re: using NamedParameter to implement a no-arg boolean command-line option

[Tobin Baker <td...@cs.washington.edu>]

Yeah, that's what I'm doing now. Off-topic (but related to my example),
since I'm launching my app under YARN with "yarn jar", I'm looking for a
way to detect my code is running under YARN (which would let me avoid
specifying this option). A horribly hacky way to do this might be to
examine the call stack in the main thread for "runJar" (which calls my main
method via reflection). Any better ideas?

On Thu, Dec 3, 2015 at 6:02 PM, Brian Cho <ch...@gmail.com> wrote:

> Hi Tobin,
>
> I don't really have a suggestion. Just wanted to add that in my experience,
> I've only seen the `-option true` route.
>
> -Brian
>
> On Fri, Dec 4, 2015 at 6:08 AM, Tobin Baker <td...@cs.washington.edu>
> wrote:
>
> > Hi, I'm trying to use CommandLine.parseToConfiguration() to implement an
> > implicit boolean command-line option, where the boolean value is
> understood
> > to be true if the option is present, and false otherwise. I.e., I want
> > something that looks like this:
> >
> > java -cp '/Users/tdbaker/myria/build/libs/*'
> > edu.washington.escience.myria.daemon.MyriaDriverLauncher -yarn
> >
> > where I can set a boolean variable useYarn based on whether -yarn is
> > present or not. Obviously I could just do
> >
> > java -cp '/Users/tdbaker/myria/build/libs/*'
> > edu.washington.escience.myria.daemon.MyriaDriverLauncher -yarn true
> >
> > using Name<Boolean> and getNamedInstance(), but I'd rather not for
> > ergonomics' sake.
> >
> > I can't seem to find the right combination of Name<Boolean>, Name<Void>,
> > default_value, or isParameterSet() to achieve this behavior, and I can't
> > find examples that implement this command-line behavior either. Any
> > suggestions?
> >
> > Thanks,
> >
> > Tobin
> >
>

Re: using NamedParameter to implement a no-arg boolean command-line option

[Brian Cho <ch...@gmail.com>]

Hi Tobin,

I don't really have a suggestion. Just wanted to add that in my experience,
I've only seen the `-option true` route.

-Brian

On Fri, Dec 4, 2015 at 6:08 AM, Tobin Baker <td...@cs.washington.edu>
wrote:

> Hi, I'm trying to use CommandLine.parseToConfiguration() to implement an
> implicit boolean command-line option, where the boolean value is understood
> to be true if the option is present, and false otherwise. I.e., I want
> something that looks like this:
>
> java -cp '/Users/tdbaker/myria/build/libs/*'
> edu.washington.escience.myria.daemon.MyriaDriverLauncher -yarn
>
> where I can set a boolean variable useYarn based on whether -yarn is
> present or not. Obviously I could just do
>
> java -cp '/Users/tdbaker/myria/build/libs/*'
> edu.washington.escience.myria.daemon.MyriaDriverLauncher -yarn true
>
> using Name<Boolean> and getNamedInstance(), but I'd rather not for
> ergonomics' sake.
>
> I can't seem to find the right combination of Name<Boolean>, Name<Void>,
> default_value, or isParameterSet() to achieve this behavior, and I can't
> find examples that implement this command-line behavior either. Any
> suggestions?
>
> Thanks,
>
> Tobin
>