[jira] [Created] (GOBBLIN-739) Add a way to propagate the Azkaban config to Gobblin on YARN

["Hung Tran (JIRA)" <ji...@apache.org>]

Hung Tran created GOBBLIN-739:
---------------------------------

             Summary: Add a way to propagate the Azkaban config to Gobblin on YARN
                 Key: GOBBLIN-739
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-739
             Project: Apache Gobblin
          Issue Type: Task
            Reporter: Hung Tran
            Assignee: Hung Tran


The AzkabanGobblinYarnAppLauncher can be used to launch a Gobblin application master on YARN, which then loads configuration from an application.conf file. Currently, the application.conf is pre-generated and packaged with the Azkaban job zip. This results in duplication of config between the Azkaban job properties and the application.conf file. It also doesn't allow user overrides in the Azkaban UI to be propagated to the app master and containers.

A config should be added to specify an output path to write the Azkaban job config to in HOCON format. The gobblin yarn config such as gobblin.yarn.app.master.files.local and gobblin.yarn.container.files.local can be set to point to the output file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)