[jira] [Created] (GOBBLIN-227) JobLauncherUtils.cleanTaskStagingData fails for jobs with forks

["Joel Baranick (JIRA)" <ji...@apache.org>]

Joel Baranick created GOBBLIN-227:
-------------------------------------

             Summary: JobLauncherUtils.cleanTaskStagingData fails for jobs with forks
                 Key: GOBBLIN-227
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-227
             Project: Apache Gobblin
          Issue Type: Bug
            Reporter: Joel Baranick


*Precondition:* 
Using Hocon configuration and have two forks configured.

*Summary:* 
When {{AbstractJobLauncher}} calls {{JobLauncherUtils.cleanTaskStagingData}} it tries to lookup {{writer.staging.dir}} in the configuration and fails.

*Details:*
Hocon configuration doesn't allow the following config:
{code:none}
writer.staging.dir=/foo
writer.staging.dir.0=/foo
writer.staging.dir.1=/foo
{code}
Initially {{writer.staging.dir}} is of type String, but when the Hocon parser encounters {{writer.staging.dir.0}}, it decides that {{writer.staging.dir}} is now of type Object, overwriting the prior value with {{_\{"0": "/foo"\}_}}.
The effective Hocon configuration is:

{code:javascript}
{
  "writer": {
    "staging": {
      "dir": {
        "0": "/foo",
        "1": "/foo"
      }
    }
  }
}
{code}

Fork specific configuration uses the same config keys as regular configuration except the fork number is appended like: {{.1}}.  The code that looks up fork specific configuration doesn't automatically fallback to regular configuration.  For example, if the code is trying to find {{writer.staging.dir.0}} and it isn't configured, the job will fail.  Then means that all forks must configure fork specific versions of {{writer.staging.dir}}.

When {{AbstractJobLauncher}} calls {{JobLauncherUtils.cleanTaskStagingData}} it cleans up the based on the current job's configuration.  Because of this, {{fork.branches}} is always set to {{1}}. The call to {{WriterUtils.getWriterStagingDir(state, numBranches, branchId)}} is make with {{numBranches=1}} and {{branchId=0}}.  This results in the method looking for {{writer.staging.dir}}. Unfortunately, when using Hocon configuration the value {{writer.staging.dir}} doesn't exist and the job fails.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)