Stack overflow solution

[Jing Gao <ke...@gmail.com>]

Hi developers,

Is there a way to configure REEF Evaluator stack size? When running some highly recursive code in REEF Evaluator, I got stack overflow error.

Something like -Xss256m will be helpful, but I don’t know where to specify that:
Option A. configure Yarn stack size: There’s no stack-specific config in yarn-site.xml.
Option B: add -Xss256m when running Reef app: can I add it somewhere in "$HADOOP_HOME/bin/yarn jar hello.jar hello.Main” ?
Option C. configure Reef stack size: Please let me know how.

Dependencies are packed in hello.jar
"$HADOOP_HOME/bin/yarn jar hello.jar hello.Main”    works
“java -Xss256m -cp hello.jar hello.Main”    fails
Can I avoid this by setting dependencies in Yarn, so that I can do the latter? How?

Thanks,

Jing

Re: configurable stack size for REEF evaluator

[Markus Weimer <ma...@weimo.de>]

Hi Jing,

(adding the mailing list)

> Thanks for the great work which makes YARN apps easier to develop.

Thanks! Please don't hesitate sharing your experiences, good or bad,
with the mailing list. It is always good to hear fresh perspectives on
our work.

> In future REEF, do you have plan to add stack size as a configurable parameter for evaluator JVM?

Yes, this is most definitely on our list, if not picked up by anyone as
of now:

  https://issues.apache.org/jira/browse/REEF-69

Eventually, we should open the whole API up for users to override the
CommandLineBuilder interface, which would allow the most flexible behavior.

As a remedy for the time being, you could just override the behavior in
JavaCommandLineBuilder to suit your needs. It is a bit of a hack, but
would unblock you.

> CMU student working with Professor Andy Pavlo, on incremental computation

Is it possible to share more of your use case with the list? It is
always interesting to hear what people do with REEF :-)

Markus

Re: Stack overflow solution

[ma...@weimo.de]

Hi,


we currently don’t have an easy way to set it. The command line used to launch the JVM for an Evaluator is built in AllocatedEvaluatorImpl.launch(). There, we use the JavaCommandLineBuilder to assemble the command line. If you just want to see whether changing the stack size helps, that class is the one you’d need to change.


Of course, if that deems useful it would be great to add this as a configurable parameter on a per-Evaluator basis.


Hope this helps,


Markus





From: Jing Gao
Sent: ‎Tuesday‎, ‎November‎ ‎4‎, ‎2014 ‎13‎:‎19
To: REEF Developers Mailinglist





Hi developers,

Is there a way to configure REEF Evaluator stack size? When running some highly recursive code in REEF Evaluator, I got stack overflow error.

Something like -Xss256m will be helpful, but I don’t know where to specify that:
Option A. configure Yarn stack size: There’s no stack-specific config in yarn-site.xml.
Option B: add -Xss256m when running Reef app: can I add it somewhere in "$HADOOP_HOME/bin/yarn jar hello.jar hello.Main” ?
Option C. configure Reef stack size: Please let me know how.

Dependencies are packed in hello.jar
"$HADOOP_HOME/bin/yarn jar hello.jar hello.Main”    works
“java -Xss256m -cp hello.jar hello.Main”    fails
Can I avoid this by setting dependencies in Yarn, so that I can do the latter? How?

Thanks,

Jing