You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2020/01/17 21:57:21 UTC

[GitHub] [incubator-gobblin] autumnust opened a new pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

autumnust opened a new pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
   
   
   ### JIRA
   - [ ] https://issues.apache.org/jira/browse/GOBBLIN-1031
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if applicable):
   - The major part of this PR is `AzkabanJobRunner.java` which could power gobblin job to be running using the same Azkaban setting that Linkedin is using internally for job-scheduling, locally. 
   - On top of that, is an Embedded Gobblin JobLauncher that starts a Gobblin-Streaming job and submit ApplictionMaster to `MiniYARNCluster` spun up in the same process. 
   - Also provide a skeleton of configurations that can be used for running the whole job locally. The skeleton has many class-definition that have not been open-sourced or only serve Linkedin-specific purpose, the general-purpose job configuration is still under development.
   
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r368764748
 
 

 ##########
 File path: gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobLauncher.java
 ##########
 @@ -93,8 +93,8 @@
  * </p>
  *
  * <p>
- *   This class runs in the {@link GobblinClusterManager}. The actual task execution happens in the in the
- *   {@link GobblinTaskRunner}.
+ *   The instance of this class is instantiated and runing along with the {@link GobblinHelixJobScheduler}.
 
 Review comment:
   Rewording: This class is instantiated by the {@link GobblinHelixJobScheduler} on every job submission to launch the Gobblin job.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r368766370
 
 

 ##########
 File path: gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/EmbeddedGobblinYarnAppLauncher.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.azkaban;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.lang.reflect.Field;
+import java.util.Map;
+
+import org.apache.gobblin.testing.AssertWithBackoff;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.testng.collections.Lists;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.io.Closer;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+/**
+ * Given a set up Azkaban job configuration, launch the Gobblin-on-Yarn job in a semi-embedded mode:
+ * - Uses external Kafka cluster and requires external Zookeeper(Non-embedded TestingServer) to be set up.
+ * - Uses MiniYARNCluster so YARN components don't have to be installed.
+ */
+@Slf4j
+public class EmbeddedGobblinYarnAppLauncher extends AzkabanJobRunner {
+  public static final String DYNAMIC_CONF_PATH = "dynamic.conf";
+  public static final String YARN_SITE_XML_PATH = "yarn-site.xml";
+  private static String zkString = "zk-ltx1-gobblin.stg.linkedin.com:6312";
 
 Review comment:
   Change this line. Shouldn't expose LI internal stuff outside.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r368764148
 
 

 ##########
 File path: gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobLauncher.java
 ##########
 @@ -93,8 +93,8 @@
  * </p>
  *
  * <p>
- *   This class runs in the {@link GobblinClusterManager}. The actual task execution happens in the in the
- *   {@link GobblinTaskRunner}.
+ *   The instance of this class is instantiated and runing along with the {@link GobblinHelixJobScheduler}.
+ *   The actual task execution happens in the in the {@link GobblinTaskRunner}, usually in a different process.
 
 Review comment:
   typo: "in the in the" -> "in the".

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r368766539
 
 

 ##########
 File path: gobblin-modules/gobblin-azkaban/src/main/resources/conf/properties/local.properties
 ##########
 @@ -0,0 +1,23 @@
+# Misc. deployment/cluster specific variables
+user.name=kafkaetl
+logs.user.name=kafkaetlstream
+root.project.name=gobblin-kafka-streaming-local
+grid.name=ltx1-holdem
 
 Review comment:
   Change this config.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#issuecomment-575821726
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=h1) Report
   > Merging [#2873](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/ef3003c85531aaeacb760a4e30fca2d3069001aa?src=pr&el=desc) will **decrease** coverage by `0.04%`.
   > The diff coverage is `3.5%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #2873      +/-   ##
   ============================================
   - Coverage     45.73%   45.68%   -0.05%     
     Complexity     9112     9112              
   ============================================
     Files          1921     1924       +3     
     Lines         72391    72495     +104     
     Branches       7967     7977      +10     
   ============================================
   + Hits          33106    33120      +14     
   - Misses        36262    36353      +91     
   + Partials       3023     3022       -1
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=) | `54.75% <ø> (ø)` | `28 <0> (ø)` | :arrow_down: |
   | [...n/java/org/apache/gobblin/util/logs/LogCopier.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbG9ncy9Mb2dDb3BpZXIuamF2YQ==) | `69.56% <ø> (ø)` | `18 <0> (ø)` | :arrow_down: |
   | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `66.66% <ø> (ø)` | `1 <0> (ø)` | :arrow_down: |
   | [.../org/apache/gobblin/yarn/GobblinYarnLogSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Mb2dTb3VyY2UuamF2YQ==) | `23.07% <ø> (ø)` | `3 <0> (ø)` | :arrow_down: |
   | [...e/gobblin/yarn/AbstractYarnAppSecurityManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vQWJzdHJhY3RZYXJuQXBwU2VjdXJpdHlNYW5hZ2VyLmphdmE=) | `48.27% <ø> (ø)` | `6 <0> (ø)` | :arrow_down: |
   | [...gobblin/azkaban/AzkabanGobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluWWFybkFwcExhdW5jaGVyLmphdmE=) | `30.55% <0%> (-2.78%)` | `2 <0> (ø)` | |
   | [...obblin/azkaban/EmbeddedGobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0VtYmVkZGVkR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...in/azkaban/AzkabanGobblinLocalYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluTG9jYWxZYXJuQXBwTGF1bmNoZXIuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `21.18% <0%> (-0.22%)` | `8 <0> (ø)` | |
   | ... and [10 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=footer). Last update [ef3003c...178065c](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] asfgit closed pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
autumnust commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r374996548
 
 

 ##########
 File path: gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/EmbeddedGobblinYarnAppLauncher.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.azkaban;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.lang.reflect.Field;
+import java.util.Map;
+
+import org.apache.gobblin.testing.AssertWithBackoff;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.testng.collections.Lists;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.io.Closer;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+/**
+ * Given a set up Azkaban job configuration, launch the Gobblin-on-Yarn job in a semi-embedded mode:
+ * - Uses external Kafka cluster and requires external Zookeeper(Non-embedded TestingServer) to be set up.
 
 Review comment:
   Yes, Kafka was intentional from very beginning, as we don't have quick set-up script for kafka and a good way to create a golden datasets there. 
   
   ZK was kind of intentional, but briefly the Helix behaves unexpectedly unstable on embedded ZK server so I abandoned that eventually. Will add these into comments.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r376171137
 
 

 ##########
 File path: gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java
 ##########
 @@ -687,9 +691,13 @@ private void addAppLocalFiles(String localFilePathList, Optional<Map<String, Loc
     for (String localFilePath : SPLITTER.split(localFilePathList)) {
       Path srcFilePath = new Path(localFilePath);
       Path destFilePath = new Path(destDir, srcFilePath.getName());
-      this.fs.copyFromLocalFile(srcFilePath, destFilePath);
-      if (resourceMap.isPresent()) {
-        YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+      if (fs.exists(srcFilePath)) {
+        this.fs.copyFromLocalFile(srcFilePath, destFilePath);
+        if (resourceMap.isPresent()) {
+          YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+        }
+      } else {
+        throw new IllegalStateException(String.format("The request file %s doesn't exist", srcFilePath));
 
 Review comment:
   We probably should not throw an exception here since the resourceMap is expected to be absent in the call from addContainerLocalResources().

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r376166058
 
 

 ##########
 File path: gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/EmbeddedGobblinYarnAppLauncher.java
 ##########
 @@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.azkaban;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.lang.reflect.Field;
+import java.util.Map;
+
+import org.apache.gobblin.testing.AssertWithBackoff;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.testng.collections.Lists;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.io.Closer;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+/**
+ * Given a set up Azkaban job configuration, launch the Gobblin-on-Yarn job in a semi-embedded mode:
+ * - Uses external Kafka cluster and requires external Zookeeper(Non-embedded TestingServer) to be set up.
+ * The Kafka Cluster was intentionally set to be external due to the data availability. External ZK was unintentional
+ * as the helix version (0.9) being used cannot finish state transition in the Embedded ZK.
+ * TODO: Adding embedded Kafka cluster and set golden datasets for data-validation.
+ * - Uses MiniYARNCluster so YARN components don't have to be installed.
+ */
+@Slf4j
+public class EmbeddedGobblinYarnAppLauncher extends AzkabanJobRunner {
+  public static final String DYNAMIC_CONF_PATH = "dynamic.conf";
+  public static final String YARN_SITE_XML_PATH = "yarn-site.xml";
+  private static String zkString = "";
+  private static String fileAddress = "";
+
+  private static void setup(String[] args)
+      throws Exception {
+    // Parsing zk-string
+    Preconditions.checkArgument(args.length == 1);
+    zkString = args[0];
+
+    // Initialize necessary external components: Yarn and Helix
+    Closer closer = Closer.create();
+
+    // Set java home in environment since it isn't set on some systems
+    String javaHome = System.getProperty("java.home");
+    setEnv("JAVA_HOME", javaHome);
+
+    final YarnConfiguration clusterConf = new YarnConfiguration();
+    clusterConf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
+    clusterConf.set("yarn.nodemanager.resource.memory-mb", "512");
+    clusterConf.set("yarn.scheduler.maximum-allocation-mb", "1024");
+
+    MiniYARNCluster miniYARNCluster = closer.register(new MiniYARNCluster("TestCluster", 1, 1, 1));
+    miniYARNCluster.init(clusterConf);
+    miniYARNCluster.start();
+
+    // YARN client should not be started before the Resource Manager is up
+    AssertWithBackoff.create().logger(log).timeoutMs(10000).assertTrue(new Predicate<Void>() {
+      @Override
+      public boolean apply(Void input) {
+        return !clusterConf.get(YarnConfiguration.RM_ADDRESS).contains(":0");
+      }
+    }, "Waiting for RM");
+
+    try (PrintWriter pw = new PrintWriter(DYNAMIC_CONF_PATH, "UTF-8")) {
+      File dir = new File("target/dummydir");
+
+      // dummy directory specified in configuration
+      if (!dir.mkdir()) {
+        log.error("The dummy folder's creation is not successful");
+      }
+      dir.deleteOnExit();
+
+      pw.println("gobblin.cluster.zk.connection.string=\"" + zkString + "\"");
+      pw.println("jobconf.fullyQualifiedPath=\"" + dir.getAbsolutePath() + "\"");
+    }
+
+    // YARN config is dynamic and needs to be passed to other processes
+    try (OutputStream os = new FileOutputStream(new File(YARN_SITE_XML_PATH))) {
+      clusterConf.writeXml(os);
+    }
+
+    /** Have to pass the same yarn-site.xml to the GobblinYarnAppLauncher to initialize Yarn Client. */
+    fileAddress = new File(YARN_SITE_XML_PATH).getAbsolutePath();
+  }
+
+  public static void setEnv(String key, String value) {
 
 Review comment:
   Does this need to be public?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#issuecomment-575821726
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=h1) Report
   > Merging [#2873](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/adb0efcc20a306d16b2f59d89c189539bd20dc2d?src=pr&el=desc) will **decrease** coverage by `0.05%`.
   > The diff coverage is `3.5%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #2873      +/-   ##
   ============================================
   - Coverage      45.8%   45.74%   -0.06%     
   - Complexity     9108     9110       +2     
   ============================================
     Files          1915     1918       +3     
     Lines         72266    72389     +123     
     Branches       7969     7982      +13     
   ============================================
   + Hits          33098    33112      +14     
   - Misses        36143    36250     +107     
   - Partials       3025     3027       +2
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=) | `53.91% <ø> (ø)` | `27 <0> (ø)` | :arrow_down: |
   | [...n/java/org/apache/gobblin/util/logs/LogCopier.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbG9ncy9Mb2dDb3BpZXIuamF2YQ==) | `69.56% <ø> (ø)` | `18 <0> (ø)` | :arrow_down: |
   | [...main/java/org/apache/gobblin/yarn/YarnService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFyblNlcnZpY2UuamF2YQ==) | `15.16% <ø> (ø)` | `4 <0> (ø)` | :arrow_down: |
   | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `66.66% <ø> (ø)` | `1 <0> (ø)` | :arrow_down: |
   | [.../org/apache/gobblin/yarn/GobblinYarnLogSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Mb2dTb3VyY2UuamF2YQ==) | `23.07% <ø> (ø)` | `3 <0> (ø)` | :arrow_down: |
   | [...e/gobblin/yarn/AbstractYarnAppSecurityManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vQWJzdHJhY3RZYXJuQXBwU2VjdXJpdHlNYW5hZ2VyLmphdmE=) | `48.27% <ø> (ø)` | `6 <0> (ø)` | :arrow_down: |
   | [...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...a/org/apache/gobblin/azkaban/AzkabanJobRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Kb2JSdW5uZXIuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...obblin/azkaban/EmbeddedGobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0VtYmVkZGVkR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | ... and [16 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=footer). Last update [adb0efc...cce9de8](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r368765618
 
 

 ##########
 File path: gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/EmbeddedGobblinYarnAppLauncher.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.azkaban;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.lang.reflect.Field;
+import java.util.Map;
+
+import org.apache.gobblin.testing.AssertWithBackoff;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.testng.collections.Lists;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.io.Closer;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+/**
+ * Given a set up Azkaban job configuration, launch the Gobblin-on-Yarn job in a semi-embedded mode:
+ * - Uses external Kafka cluster and requires external Zookeeper(Non-embedded TestingServer) to be set up.
 
 Review comment:
   Can you add why we are using an external ZK/Kafka server? And a TODO to set these up in embedded mode if that is the intention?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r368766551
 
 

 ##########
 File path: gobblin-modules/gobblin-azkaban/src/main/resources/conf/properties/local.properties
 ##########
 @@ -0,0 +1,23 @@
+# Misc. deployment/cluster specific variables
+user.name=kafkaetl
 
 Review comment:
   Change this config.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r368764662
 
 

 ##########
 File path: gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterManager.java
 ##########
 @@ -228,6 +228,8 @@ private void stopAppLauncherAndServices() {
 
   /**
    * Configure Helix quota-based task scheduling
+   * This Task-quota indicates how many helix-tasks can be running concurrently within a Helix Job.
 
 Review comment:
   More precisely: this config controls the number of tasks that are concurrently assigned to a single Helix instance.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r368765996
 
 

 ##########
 File path: gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnService.java
 ##########
 @@ -533,6 +533,7 @@ private void addRemoteAppFiles(String hdfsFileList, Map<String, LocalResource> r
     }
   }
 
+  // TODO: Not fully understand the purpose of this method.
 
 Review comment:
   What is the TODO here? Drop this comment if unintentional.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r368766504
 
 

 ##########
 File path: gobblin-modules/gobblin-azkaban/src/main/resources/conf/properties/local.properties
 ##########
 @@ -0,0 +1,23 @@
+# Misc. deployment/cluster specific variables
+user.name=kafkaetl
+logs.user.name=kafkaetlstream
+root.project.name=gobblin-kafka-streaming-local
+grid.name=ltx1-holdem
+
+
+# Cluster specific directory configurations
+# home.dir and root.data.location should be unique per cluster deployment
+
+home.dir=/jobs/${user.name}
+logs.dir=/jobs/${logs.user.name}
+root.data.location=/data
+
+fs.uri=file:///
+
+# Yarn Configuration
+gobblin.yarn.app.queue=default
+
+# Gobblin Cluster
+#Add a suffix "-1" to the app name becuase of a corrupted Helix cluster znode (LIZK-619). This made the helix cluster
 
 Review comment:
   Drop this comment.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
autumnust commented on a change in pull request #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#discussion_r378448071
 
 

 ##########
 File path: gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/EmbeddedGobblinYarnAppLauncher.java
 ##########
 @@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.azkaban;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.lang.reflect.Field;
+import java.util.Map;
+
+import org.apache.gobblin.testing.AssertWithBackoff;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.testng.collections.Lists;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.io.Closer;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+/**
+ * Given a set up Azkaban job configuration, launch the Gobblin-on-Yarn job in a semi-embedded mode:
+ * - Uses external Kafka cluster and requires external Zookeeper(Non-embedded TestingServer) to be set up.
+ * The Kafka Cluster was intentionally set to be external due to the data availability. External ZK was unintentional
+ * as the helix version (0.9) being used cannot finish state transition in the Embedded ZK.
+ * TODO: Adding embedded Kafka cluster and set golden datasets for data-validation.
+ * - Uses MiniYARNCluster so YARN components don't have to be installed.
+ */
+@Slf4j
+public class EmbeddedGobblinYarnAppLauncher extends AzkabanJobRunner {
+  public static final String DYNAMIC_CONF_PATH = "dynamic.conf";
+  public static final String YARN_SITE_XML_PATH = "yarn-site.xml";
+  private static String zkString = "";
+  private static String fileAddress = "";
+
+  private static void setup(String[] args)
+      throws Exception {
+    // Parsing zk-string
+    Preconditions.checkArgument(args.length == 1);
+    zkString = args[0];
+
+    // Initialize necessary external components: Yarn and Helix
+    Closer closer = Closer.create();
+
+    // Set java home in environment since it isn't set on some systems
+    String javaHome = System.getProperty("java.home");
+    setEnv("JAVA_HOME", javaHome);
+
+    final YarnConfiguration clusterConf = new YarnConfiguration();
+    clusterConf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
+    clusterConf.set("yarn.nodemanager.resource.memory-mb", "512");
+    clusterConf.set("yarn.scheduler.maximum-allocation-mb", "1024");
+
+    MiniYARNCluster miniYARNCluster = closer.register(new MiniYARNCluster("TestCluster", 1, 1, 1));
+    miniYARNCluster.init(clusterConf);
+    miniYARNCluster.start();
+
+    // YARN client should not be started before the Resource Manager is up
+    AssertWithBackoff.create().logger(log).timeoutMs(10000).assertTrue(new Predicate<Void>() {
+      @Override
+      public boolean apply(Void input) {
+        return !clusterConf.get(YarnConfiguration.RM_ADDRESS).contains(":0");
+      }
+    }, "Waiting for RM");
+
+    try (PrintWriter pw = new PrintWriter(DYNAMIC_CONF_PATH, "UTF-8")) {
+      File dir = new File("target/dummydir");
+
+      // dummy directory specified in configuration
+      if (!dir.mkdir()) {
+        log.error("The dummy folder's creation is not successful");
+      }
+      dir.deleteOnExit();
+
+      pw.println("gobblin.cluster.zk.connection.string=\"" + zkString + "\"");
+      pw.println("jobconf.fullyQualifiedPath=\"" + dir.getAbsolutePath() + "\"");
+    }
+
+    // YARN config is dynamic and needs to be passed to other processes
+    try (OutputStream os = new FileOutputStream(new File(YARN_SITE_XML_PATH))) {
+      clusterConf.writeXml(os);
+    }
+
+    /** Have to pass the same yarn-site.xml to the GobblinYarnAppLauncher to initialize Yarn Client. */
+    fileAddress = new File(YARN_SITE_XML_PATH).getAbsolutePath();
+  }
+
+  public static void setEnv(String key, String value) {
 
 Review comment:
   Addressed

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] codecov-io commented on issue #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #2873: [GOBBLIN-1031]Gobblin-on-Yarn locally running Azkaban job skeleton
URL: https://github.com/apache/incubator-gobblin/pull/2873#issuecomment-575821726
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=h1) Report
   > Merging [#2873](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/adb0efcc20a306d16b2f59d89c189539bd20dc2d?src=pr&el=desc) will **decrease** coverage by `0.05%`.
   > The diff coverage is `3.5%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #2873      +/-   ##
   ============================================
   - Coverage      45.8%   45.74%   -0.06%     
   - Complexity     9108     9110       +2     
   ============================================
     Files          1915     1918       +3     
     Lines         72266    72389     +123     
     Branches       7969     7982      +13     
   ============================================
   + Hits          33098    33112      +14     
   - Misses        36143    36250     +107     
   - Partials       3025     3027       +2
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=) | `53.91% <ø> (ø)` | `27 <0> (ø)` | :arrow_down: |
   | [...n/java/org/apache/gobblin/util/logs/LogCopier.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbG9ncy9Mb2dDb3BpZXIuamF2YQ==) | `69.56% <ø> (ø)` | `18 <0> (ø)` | :arrow_down: |
   | [...main/java/org/apache/gobblin/yarn/YarnService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFyblNlcnZpY2UuamF2YQ==) | `15.16% <ø> (ø)` | `4 <0> (ø)` | :arrow_down: |
   | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `66.66% <ø> (ø)` | `1 <0> (ø)` | :arrow_down: |
   | [.../org/apache/gobblin/yarn/GobblinYarnLogSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Mb2dTb3VyY2UuamF2YQ==) | `23.07% <ø> (ø)` | `3 <0> (ø)` | :arrow_down: |
   | [...e/gobblin/yarn/AbstractYarnAppSecurityManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vQWJzdHJhY3RZYXJuQXBwU2VjdXJpdHlNYW5hZ2VyLmphdmE=) | `48.27% <ø> (ø)` | `6 <0> (ø)` | :arrow_down: |
   | [...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...a/org/apache/gobblin/azkaban/AzkabanJobRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Kb2JSdW5uZXIuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...obblin/azkaban/EmbeddedGobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0VtYmVkZGVkR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | ... and [16 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2873/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=footer). Last update [adb0efc...cce9de8](https://codecov.io/gh/apache/incubator-gobblin/pull/2873?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services