[jira] [Created] (SAMZA-37) Allow run-am.sh override for YARN AM or disable run-task.sh override

["Chris Riccomini (JIRA)" <ji...@apache.org>]

Chris Riccomini created SAMZA-37:
------------------------------------

             Summary: Allow run-am.sh override for YARN AM or disable run-task.sh override
                 Key: SAMZA-37
                 URL: https://issues.apache.org/jira/browse/SAMZA-37
             Project: Samza
          Issue Type: Bug
          Components: container, yarn
    Affects Versions: 0.6.0
            Reporter: Chris Riccomini


Samza allows developers to over-ride the path for the run-task.sh script, but does not allow developers to override the path for the run-am.sh script. The run-am.sh script is hard coded to __package/bin/run-am.sh.

The two relevant configurations that allow run-task.sh override are:

* task.command.class=org.apache.samza.job.ShellCommandBuilder
* task.execute=bin/run-task.sh

The motivation for allowing this override is to support Samza job packages with different package layouts, and also to allow for supporting other languages (e.g. running a python virtual environment as a Samza job).

If we believe that the run-task.sh override feature is a good thing (one might argue it's kind of complex, and we should just force a convention), then we should allow run-am.sh override using the same style. If we go this route, we should add the following configuration to YarnConfig:

* yarn.am.command.class=org.apache.samza.job.YarnAmShellCommandBuilder
* yarm.am.execute=bin/run-task.sh

The YarnAmShellCommandBuilder should just extend ShellCommandBuilder, but override buildCommand to call config.getAmCommand (yarn.am.execute), which will be defined in YarnConfig.

I think we should just force the bin/run-task.sh convention, and disable run-task.sh override. We can still keep the ShellCommand* classes around, since they help build the environment variables for the SamzaContainer.

I also think we should rename run-task.sh to run-container.sh, but I'll open a separate ticket for that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira