You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Will-Lo (via GitHub)" <gi...@apache.org> on 2024/08/16 02:45:23 UTC

[PR] Cache temporal jars [gobblin]

Will-Lo opened a new pull request, #4030:
URL: https://github.com/apache/gobblin/pull/4030

   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
   
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
       - https://issues.apache.org/jira/browse/GOBBLIN-XXX
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if applicable):
   
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "Will-Lo (via GitHub)" <gi...@apache.org>.
Will-Lo commented on code in PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#discussion_r1755496938


##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java:
##########
@@ -572,56 +573,22 @@ private void addJars(Path jarFileDir, String jarFileList, Configuration conf) th
     for (String jarFile : SPLITTER.split(jarFileList)) {
       Path srcJarFile = new Path(jarFile);
       FileStatus[] fileStatusList = lfs.globStatus(srcJarFile);
-
       for (FileStatus status : fileStatusList) {
+        Path destJarFile = HdfsJarUploadUtils.calculateDestJarFile(fs, status, this.unsharedJarsDir, jarFileDir);
         // For each FileStatus there are chances it could fail in copying at the first attempt, due to file-existence
         // or file-copy is ongoing by other job instance since all Gobblin jobs share the same jar file directory.
         // the retryCount is to avoid cases (if any) where retry is going too far and causes job hanging.
-        int retryCount = 0;
-        boolean shouldFileBeAddedIntoDC = true;
-        Path destJarFile = calculateDestJarFile(status, jarFileDir);
-        // Adding destJarFile into HDFS until it exists and the size of file on targetPath matches the one on local path.
-        while (!this.fs.exists(destJarFile) || fs.getFileStatus(destJarFile).getLen() != status.getLen()) {
-          try {
-            if (this.fs.exists(destJarFile) && fs.getFileStatus(destJarFile).getLen() != status.getLen()) {
-              Thread.sleep(WAITING_TIME_ON_IMCOMPLETE_UPLOAD);
-              throw new IOException("Waiting for file to complete on uploading ... ");
-            }
-            // Set the first parameter as false for not deleting sourceFile
-            // Set the second parameter as false for not overwriting existing file on the target, by default it is true.
-            // If the file is preExisted but overwrite flag set to false, then an IOException if thrown.
-            this.fs.copyFromLocalFile(false, false, status.getPath(), destJarFile);
-          } catch (IOException | InterruptedException e) {
-            LOG.warn("Path:" + destJarFile + " is not copied successfully. Will require retry.");
-            retryCount += 1;
-            if (retryCount >= this.jarFileMaximumRetry) {
-              LOG.error("The jar file:" + destJarFile + "failed in being copied into hdfs", e);
-              // If retry reaches upper limit, skip copying this file.
-              shouldFileBeAddedIntoDC = false;
-              break;
-            }
-          }
-        }
-        if (shouldFileBeAddedIntoDC) {
+        if (HdfsJarUploadUtils.uploadJarToHdfs(this.fs, status, this.jarFileMaximumRetry, destJarFile)) {
           // Then add the jar file on HDFS to the classpath
           LOG.info(String.format("Adding %s to classpath", destJarFile));
           DistributedCache.addFileToClassPath(destJarFile, conf, this.fs);
+        } else {
+          LOG.error("Failed to upload jar file: " + status.getPath());

Review Comment:
   Possibly we should throw an error, but I think given that another job could be uploading the same jars though it might be better to let the job attempt to try and run, if that job fails it should be emitting the failed event anyways.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "Will-Lo (via GitHub)" <gi...@apache.org>.
Will-Lo merged PR #4030:
URL: https://github.com/apache/gobblin/pull/4030


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "Will-Lo (via GitHub)" <gi...@apache.org>.
Will-Lo commented on code in PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#discussion_r1755526864


##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -720,26 +731,26 @@ private void addLibJars(Path srcLibJarDir, Optional<Map<String, LocalResource>>
     }
 
     for (FileStatus libJarFile : libJarFiles) {
-      Path destFilePath = new Path(destDir, libJarFile.getPath().getName());
-      this.fs.copyFromLocalFile(libJarFile.getPath(), destFilePath);
-      if (resourceMap.isPresent()) {
+      Path destFilePath = HdfsJarUploadUtils.calculateDestJarFile(this.fs, libJarFile, unsharedDir, destCacheDir);
+      if (HdfsJarUploadUtils.uploadJarToHdfs(fs, libJarFile, MAXIMUM_JAR_COPY_RETRY_TIMES_DEFAULT, destFilePath) && resourceMap.isPresent()) {
         YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+      } else {
+        LOGGER.warn("Failed to upload jar file {} to HDFS", libJarFile.getPath());
       }
     }
   }
-
-  private void addAppJars(String jarFilePathList, Optional<Map<String, LocalResource>> resourceMap,
-      Path destDir, FileSystem localFs) throws IOException {
+  private void addAppJars(String jarFilePathList, Optional<Map<String, LocalResource>> resourceMap, Path destCacheDir, Path unsharedDir,
+      FileSystem localFs) throws IOException {
     for (String jarFilePath : SPLITTER.split(jarFilePathList)) {
       Path srcFilePath = new Path(jarFilePath);
-      Path destFilePath = new Path(destDir, srcFilePath.getName());
-      if (localFs.exists(srcFilePath)) {
-        this.fs.copyFromLocalFile(srcFilePath, destFilePath);
+      FileStatus localJar = localFs.getFileStatus(srcFilePath);
+      Path destFilePath = HdfsJarUploadUtils.calculateDestJarFile(this.fs, localJar, unsharedDir, destCacheDir);
+      if (HdfsJarUploadUtils.uploadJarToHdfs(fs, localJar, MAXIMUM_JAR_COPY_RETRY_TIMES_DEFAULT, destFilePath)) {
+        if (resourceMap.isPresent()) {
+          YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+        }
       } else {
-        LOGGER.warn("The src destination " + srcFilePath + " doesn't exists");
-      }
-      if (resourceMap.isPresent()) {
-        YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+        LOGGER.warn("Failed to upload jar file {} to HDFS", srcFilePath);

Review Comment:
   I think the concern is valid but currently trying to get parity with MR job launcher, I think since there could always be concurrent executions adding the jars instead so it can be worthwhile to just attempt the job, it will fail loudly if the jars weren't uploaded properly anyways.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "phet (via GitHub)" <gi...@apache.org>.
phet commented on code in PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#discussion_r1735415804


##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java:
##########
@@ -572,56 +573,22 @@ private void addJars(Path jarFileDir, String jarFileList, Configuration conf) th
     for (String jarFile : SPLITTER.split(jarFileList)) {
       Path srcJarFile = new Path(jarFile);
       FileStatus[] fileStatusList = lfs.globStatus(srcJarFile);
-
       for (FileStatus status : fileStatusList) {
+        Path destJarFile = HdfsJarUploadUtils.calculateDestJarFile(fs, status, this.unsharedJarsDir, jarFileDir);
         // For each FileStatus there are chances it could fail in copying at the first attempt, due to file-existence
         // or file-copy is ongoing by other job instance since all Gobblin jobs share the same jar file directory.
         // the retryCount is to avoid cases (if any) where retry is going too far and causes job hanging.
-        int retryCount = 0;
-        boolean shouldFileBeAddedIntoDC = true;
-        Path destJarFile = calculateDestJarFile(status, jarFileDir);
-        // Adding destJarFile into HDFS until it exists and the size of file on targetPath matches the one on local path.
-        while (!this.fs.exists(destJarFile) || fs.getFileStatus(destJarFile).getLen() != status.getLen()) {
-          try {
-            if (this.fs.exists(destJarFile) && fs.getFileStatus(destJarFile).getLen() != status.getLen()) {
-              Thread.sleep(WAITING_TIME_ON_IMCOMPLETE_UPLOAD);
-              throw new IOException("Waiting for file to complete on uploading ... ");
-            }
-            // Set the first parameter as false for not deleting sourceFile
-            // Set the second parameter as false for not overwriting existing file on the target, by default it is true.
-            // If the file is preExisted but overwrite flag set to false, then an IOException if thrown.
-            this.fs.copyFromLocalFile(false, false, status.getPath(), destJarFile);
-          } catch (IOException | InterruptedException e) {
-            LOG.warn("Path:" + destJarFile + " is not copied successfully. Will require retry.");
-            retryCount += 1;
-            if (retryCount >= this.jarFileMaximumRetry) {
-              LOG.error("The jar file:" + destJarFile + "failed in being copied into hdfs", e);
-              // If retry reaches upper limit, skip copying this file.
-              shouldFileBeAddedIntoDC = false;
-              break;
-            }
-          }
-        }
-        if (shouldFileBeAddedIntoDC) {
+        if (HdfsJarUploadUtils.uploadJarToHdfs(this.fs, status, this.jarFileMaximumRetry, destJarFile)) {
           // Then add the jar file on HDFS to the classpath
           LOG.info(String.format("Adding %s to classpath", destJarFile));
           DistributedCache.addFileToClassPath(destJarFile, conf, this.fs);
+        } else {
+          LOG.error("Failed to upload jar file: " + status.getPath());

Review Comment:
   I don't find the prior code throwing an error...
   
   nonetheless, should everything continue on w/ just some error logs?
   
   shouldn't we instead fail the overall job because presumably necessary jars won't be there?



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java:
##########
@@ -484,12 +487,29 @@ private void requestContainer(Optional<String> preferredNode, Resource resource)
   protected ContainerLaunchContext newContainerLaunchContext(ContainerInfo containerInfo)
       throws IOException {
     Path appWorkDir = GobblinClusterUtils.getAppWorkDirPathFromConfig(this.config, this.fs, this.applicationName, this.applicationId);
+    Path containerJarsUnsharedDir = new Path(appWorkDir, GobblinYarnConfigurationKeys.CONTAINER_WORK_DIR_NAME);

Review Comment:
   though you said in the PR desc, suggest a comment here about "unshared" dir being for "-SNAPSHOT" versions



##########
gobblin-utility/src/main/java/org/apache/gobblin/util/filesystem/HdfsJarUploadUtils.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.util.filesystem;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+/**
+ * Utility class for uploading jar files to HDFS with retries to handle concurrency
+ */
+@Slf4j
+public class HdfsJarUploadUtils {
+
+  private static final long WAITING_TIME_ON_IMCOMPLETE_UPLOAD = 3000;
+
+  /**
+   * Calculate the target filePath of the jar file to be copied on HDFS,
+   * given the {@link FileStatus} of a jarFile and the path of directory that contains jar.
+   * Snapshot dirs should not be shared, as different jobs may be using different versions of it.
+   * @param fs
+   * @param localJar
+   * @param unsharedJarsDir
+   * @param jarCacheDir
+   * @return
+   * @throws IOException
+   */
+  public static Path calculateDestJarFile(FileSystem fs, FileStatus localJar, Path unsharedJarsDir, Path jarCacheDir) throws IOException {

Review Comment:
   suggest to name `calculateDestJarFilePath`.
   
   and, since the only use is `localJar.getPath().getName()` suggest to make the param `String jarName`



##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -300,10 +304,10 @@ public GobblinYarnAppLauncher(Config config, YarnConfiguration yarnConfiguration
         GobblinYarnConfigurationKeys.DEFAULT_GOBBLIN_YARN_CONTAINER_TIMEZONE);
 
     this.detachOnExitEnabled = ConfigUtils
-        .getBoolean(config, GobblinYarnConfigurationKeys.GOBBLIN_YARN_DETACH_ON_EXIT_ENABLED,
+        .getBoolean(this.config, GobblinYarnConfigurationKeys.GOBBLIN_YARN_DETACH_ON_EXIT_ENABLED,
             GobblinYarnConfigurationKeys.DEFAULT_GOBBLIN_YARN_DETACH_ON_EXIT);
-    this.appLauncherMode = ConfigUtils.getString(config, GOBBLIN_YARN_APP_LAUNCHER_MODE, DEFAULT_GOBBLIN_YARN_APP_LAUNCHER_MODE);
-
+    this.appLauncherMode = ConfigUtils.getString(this.config, GOBBLIN_YARN_APP_LAUNCHER_MODE, DEFAULT_GOBBLIN_YARN_APP_LAUNCHER_MODE);
+    this.jarCacheEnabled = ConfigUtils.getBoolean(config, GobblinYarnConfigurationKeys.JAR_CACHE_ENABLED, GobblinYarnConfigurationKeys.JAR_CACHE_ENABLED_DEFAULT);

Review Comment:
   NBD, but you just updated the two above to be `this.config`, but only use `config` here :)



##########
gobblin-utility/src/main/java/org/apache/gobblin/util/filesystem/HdfsJarUploadUtils.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.util.filesystem;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+/**
+ * Utility class for uploading jar files to HDFS with retries to handle concurrency
+ */
+@Slf4j
+public class HdfsJarUploadUtils {
+
+  private static final long WAITING_TIME_ON_IMCOMPLETE_UPLOAD = 3000;

Review Comment:
   are these seconds?  suggest a suffix to clarify



##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java:
##########
@@ -199,6 +203,18 @@ public static void setYarnClassPath(Config config, Configuration yarnConfigurati
     }
   }
 
+  public static Path getJarPathCacheAndCleanIfNeeded(Config config, FileSystem fs) throws IOException {

Review Comment:
   this feels like two separate operations:
   ```
   Path calcJarCacheCurrentPath(Config, FileSystem);
   boolean retainKLatestCachePaths(Path parentCachePath, int k);  /// true iff any deletion
   ```



##########
gobblin-utility/src/main/java/org/apache/gobblin/util/filesystem/HdfsJarUploadUtils.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.util.filesystem;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+/**
+ * Utility class for uploading jar files to HDFS with retries to handle concurrency
+ */
+@Slf4j
+public class HdfsJarUploadUtils {
+
+  private static final long WAITING_TIME_ON_IMCOMPLETE_UPLOAD = 3000;
+
+  /**
+   * Calculate the target filePath of the jar file to be copied on HDFS,
+   * given the {@link FileStatus} of a jarFile and the path of directory that contains jar.
+   * Snapshot dirs should not be shared, as different jobs may be using different versions of it.
+   * @param fs
+   * @param localJar
+   * @param unsharedJarsDir
+   * @param jarCacheDir
+   * @return
+   * @throws IOException
+   */
+  public static Path calculateDestJarFile(FileSystem fs, FileStatus localJar, Path unsharedJarsDir, Path jarCacheDir) throws IOException {
+    Path uploadDir = localJar.getPath().getName().contains("SNAPSHOT") ? unsharedJarsDir : jarCacheDir;
+    Path destJarFile = new Path(fs.makeQualified(uploadDir), localJar.getPath().getName());
+    return destJarFile;
+  }
+  /**
+   * Upload a jar file to HDFS with retries to handle already existing jars
+   * @param fs
+   * @param localJar
+   * @param destJarFile
+   * @param jarFileMaximumRetry
+   * @return
+   * @throws IOException
+   */
+  public static boolean uploadJarToHdfs(FileSystem fs, FileStatus localJar, int jarFileMaximumRetry, Path destJarFile) throws IOException {

Review Comment:
   `jarFileMaximumRetry`  => simply `maxAttempts`?



##########
gobblin-utility/src/main/java/org/apache/gobblin/util/filesystem/HdfsJarUploadUtils.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.util.filesystem;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+/**
+ * Utility class for uploading jar files to HDFS with retries to handle concurrency
+ */
+@Slf4j
+public class HdfsJarUploadUtils {
+
+  private static final long WAITING_TIME_ON_IMCOMPLETE_UPLOAD = 3000;
+
+  /**
+   * Calculate the target filePath of the jar file to be copied on HDFS,
+   * given the {@link FileStatus} of a jarFile and the path of directory that contains jar.
+   * Snapshot dirs should not be shared, as different jobs may be using different versions of it.
+   * @param fs
+   * @param localJar
+   * @param unsharedJarsDir
+   * @param jarCacheDir
+   * @return
+   * @throws IOException
+   */
+  public static Path calculateDestJarFile(FileSystem fs, FileStatus localJar, Path unsharedJarsDir, Path jarCacheDir) throws IOException {
+    Path uploadDir = localJar.getPath().getName().contains("SNAPSHOT") ? unsharedJarsDir : jarCacheDir;
+    Path destJarFile = new Path(fs.makeQualified(uploadDir), localJar.getPath().getName());
+    return destJarFile;
+  }
+  /**
+   * Upload a jar file to HDFS with retries to handle already existing jars
+   * @param fs
+   * @param localJar
+   * @param destJarFile
+   * @param jarFileMaximumRetry
+   * @return
+   * @throws IOException
+   */
+  public static boolean uploadJarToHdfs(FileSystem fs, FileStatus localJar, int jarFileMaximumRetry, Path destJarFile) throws IOException {
+    int retryCount = 0;
+    while (!fs.exists(destJarFile) || fs.getFileStatus(destJarFile).getLen() != localJar.getLen()) {
+      try {
+        if (fs.exists(destJarFile) && fs.getFileStatus(destJarFile).getLen() != localJar.getLen()) {
+          Thread.sleep(WAITING_TIME_ON_IMCOMPLETE_UPLOAD);

Review Comment:
   what are we waiting on here?  is `copyFromLocalFile` asynchronous?  or are we thinking a different process may be doing upload.
   
   with `overwriteAnyExistingDestFile == false`, we don't seem to undertake any "repair", do we?  instead we just wait `(waitTime * jarFileMaximumRetry)`



##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -640,31 +643,36 @@ private Resource prepareContainerResource(GetNewApplicationResponse newApplicati
 
   private Map<String, LocalResource> addAppMasterLocalResources(ApplicationId applicationId) throws IOException {
     Path appWorkDir = GobblinClusterUtils.getAppWorkDirPathFromConfig(this.config, this.fs, this.applicationName, applicationId.toString());
+    Path jarsRootDir = this.jarCacheEnabled ? YarnHelixUtils.getJarPathCacheAndCleanIfNeeded(this.config, this.fs) : appWorkDir;
 
     Path appMasterWorkDir = new Path(appWorkDir, GobblinYarnConfigurationKeys.APP_MASTER_WORK_DIR_NAME);
-    LOGGER.info("Configured GobblinApplicationMaster work directory to: {}", appMasterWorkDir.toString());
+    Path appMasterJarsCacheDir = new Path(jarsRootDir, GobblinYarnConfigurationKeys.APP_MASTER_WORK_DIR_NAME);
+    LOGGER.info("Configured GobblinApplicationMaster work directory to: {}", appMasterWorkDir);
+    LOGGER.info("Configured GobblinApplicationMaster jars directory to: {}", appMasterJarsCacheDir);
 
     Map<String, LocalResource> appMasterResources = Maps.newHashMap();
     FileSystem localFs = FileSystem.getLocal(new Configuration());
 
-    // NOTE: log after each step below for insight into what takes bulk of time
     if (this.config.hasPath(GobblinYarnConfigurationKeys.LIB_JARS_DIR_KEY)) {
-      Path libJarsDestDir = new Path(appWorkDir, GobblinYarnConfigurationKeys.LIB_JARS_DIR_NAME);
+      // Lib jars are shared between all containers, store at the root level
+      Path libJarsDestDir = new Path(jarsRootDir, GobblinYarnConfigurationKeys.LIB_JARS_DIR_NAME);
+      Path unsharedJarsDestDir = new Path(appWorkDir, GobblinYarnConfigurationKeys.LIB_JARS_DIR_NAME);
       addLibJars(new Path(this.config.getString(GobblinYarnConfigurationKeys.LIB_JARS_DIR_KEY)),
-          Optional.of(appMasterResources), libJarsDestDir, localFs);
-      LOGGER.info("Added lib jars to directory: {}", libJarsDestDir.toString());
+          Optional.of(appMasterResources), libJarsDestDir, unsharedJarsDestDir, localFs);
+      LOGGER.info("Added lib jars to directory: {}", libJarsDestDir);

Review Comment:
   it seems `addLibJars` could also add some to `unsharedJarsDestDir`, so good to mention that while logging



##########
gobblin-utility/src/main/java/org/apache/gobblin/util/filesystem/HdfsJarUploadUtils.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.util.filesystem;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+/**
+ * Utility class for uploading jar files to HDFS with retries to handle concurrency
+ */
+@Slf4j
+public class HdfsJarUploadUtils {
+
+  private static final long WAITING_TIME_ON_IMCOMPLETE_UPLOAD = 3000;
+
+  /**
+   * Calculate the target filePath of the jar file to be copied on HDFS,
+   * given the {@link FileStatus} of a jarFile and the path of directory that contains jar.
+   * Snapshot dirs should not be shared, as different jobs may be using different versions of it.
+   * @param fs
+   * @param localJar
+   * @param unsharedJarsDir
+   * @param jarCacheDir
+   * @return
+   * @throws IOException
+   */
+  public static Path calculateDestJarFile(FileSystem fs, FileStatus localJar, Path unsharedJarsDir, Path jarCacheDir) throws IOException {
+    Path uploadDir = localJar.getPath().getName().contains("SNAPSHOT") ? unsharedJarsDir : jarCacheDir;
+    Path destJarFile = new Path(fs.makeQualified(uploadDir), localJar.getPath().getName());
+    return destJarFile;
+  }
+  /**
+   * Upload a jar file to HDFS with retries to handle already existing jars
+   * @param fs
+   * @param localJar
+   * @param destJarFile
+   * @param jarFileMaximumRetry
+   * @return
+   * @throws IOException
+   */
+  public static boolean uploadJarToHdfs(FileSystem fs, FileStatus localJar, int jarFileMaximumRetry, Path destJarFile) throws IOException {
+    int retryCount = 0;
+    while (!fs.exists(destJarFile) || fs.getFileStatus(destJarFile).getLen() != localJar.getLen()) {
+      try {
+        if (fs.exists(destJarFile) && fs.getFileStatus(destJarFile).getLen() != localJar.getLen()) {
+          Thread.sleep(WAITING_TIME_ON_IMCOMPLETE_UPLOAD);
+          throw new IOException("Waiting for file to complete on uploading ... ");
+        }
+        // Set the first parameter as false for not deleting sourceFile
+        // Set the second parameter as false for not overwriting existing file on the target, by default it is true.
+        // If the file is preExisted but overwrite flag set to false, then an IOException if thrown.
+        fs.copyFromLocalFile(false, false, localJar.getPath(), destJarFile);

Review Comment:
   ```
   boolean deleteSourceFile = false;
   boolean overwriteAnyExistingDestFile = false; // IOException will be thrown if does already exist
   fs.copyFromLocalFile(deleteSourceFile, overwriteAnyExistingDestFile, localJar.getPath(), destJarFile)
   ```
   



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java:
##########
@@ -484,12 +487,29 @@ private void requestContainer(Optional<String> preferredNode, Resource resource)
   protected ContainerLaunchContext newContainerLaunchContext(ContainerInfo containerInfo)
       throws IOException {
     Path appWorkDir = GobblinClusterUtils.getAppWorkDirPathFromConfig(this.config, this.fs, this.applicationName, this.applicationId);
+    Path containerJarsUnsharedDir = new Path(appWorkDir, GobblinYarnConfigurationKeys.CONTAINER_WORK_DIR_NAME);
+    Path jarCacheDir = this.jarCacheEnabled ? YarnHelixUtils.getJarPathCacheAndCleanIfNeeded(this.config, this.fs) : appWorkDir;
+    Path containerJarsCachedDir = new Path(jarCacheDir, GobblinYarnConfigurationKeys.CONTAINER_WORK_DIR_NAME);
+    LOGGER.info("Container cached jars root dir: " + containerJarsCachedDir);
+    LOGGER.info("Container uncached jars root dir: " + containerJarsUnsharedDir);

Review Comment:
   I'd find something like "execution-private" or "unshared" more explicit than uncached.
   



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java:
##########
@@ -484,12 +487,29 @@ private void requestContainer(Optional<String> preferredNode, Resource resource)
   protected ContainerLaunchContext newContainerLaunchContext(ContainerInfo containerInfo)
       throws IOException {
     Path appWorkDir = GobblinClusterUtils.getAppWorkDirPathFromConfig(this.config, this.fs, this.applicationName, this.applicationId);
+    Path containerJarsUnsharedDir = new Path(appWorkDir, GobblinYarnConfigurationKeys.CONTAINER_WORK_DIR_NAME);
+    Path jarCacheDir = this.jarCacheEnabled ? YarnHelixUtils.getJarPathCacheAndCleanIfNeeded(this.config, this.fs) : appWorkDir;
+    Path containerJarsCachedDir = new Path(jarCacheDir, GobblinYarnConfigurationKeys.CONTAINER_WORK_DIR_NAME);
+    LOGGER.info("Container cached jars root dir: " + containerJarsCachedDir);
+    LOGGER.info("Container uncached jars root dir: " + containerJarsUnsharedDir);
     Path containerWorkDir = new Path(appWorkDir, GobblinYarnConfigurationKeys.CONTAINER_WORK_DIR_NAME);
 
-    Map<String, LocalResource> resourceMap = Maps.newHashMap();
 
+    Map<String, LocalResource> resourceMap = Maps.newHashMap();
+    // Always fetch any jars from the appWorkDir for any potential snapshot jars
     addContainerLocalResources(new Path(appWorkDir, GobblinYarnConfigurationKeys.LIB_JARS_DIR_NAME), resourceMap);
-    addContainerLocalResources(new Path(containerWorkDir, GobblinYarnConfigurationKeys.APP_JARS_DIR_NAME), resourceMap);
+    if (this.config.hasPath(GobblinYarnConfigurationKeys.CONTAINER_FILES_LOCAL_KEY)) {
+      addContainerLocalResources(new Path(containerJarsUnsharedDir, GobblinYarnConfigurationKeys.APP_JARS_DIR_NAME),
+          resourceMap);
+    }
+    if (this.jarCacheEnabled) {
+      addContainerLocalResources(new Path(jarCacheDir, GobblinYarnConfigurationKeys.LIB_JARS_DIR_NAME), resourceMap);
+      if (this.config.hasPath(GobblinYarnConfigurationKeys.CONTAINER_FILES_LOCAL_KEY)) {

Review Comment:
   I don't quite understand this key.  you're checking it here in two different conditionals, but in neither one do you actually use (or even check to see) what value it holds



##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -720,26 +731,26 @@ private void addLibJars(Path srcLibJarDir, Optional<Map<String, LocalResource>>
     }
 
     for (FileStatus libJarFile : libJarFiles) {
-      Path destFilePath = new Path(destDir, libJarFile.getPath().getName());
-      this.fs.copyFromLocalFile(libJarFile.getPath(), destFilePath);
-      if (resourceMap.isPresent()) {
+      Path destFilePath = HdfsJarUploadUtils.calculateDestJarFile(this.fs, libJarFile, unsharedDir, destCacheDir);
+      if (HdfsJarUploadUtils.uploadJarToHdfs(fs, libJarFile, MAXIMUM_JAR_COPY_RETRY_TIMES_DEFAULT, destFilePath) && resourceMap.isPresent()) {
         YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+      } else {
+        LOGGER.warn("Failed to upload jar file {} to HDFS", libJarFile.getPath());
       }
     }
   }
-
-  private void addAppJars(String jarFilePathList, Optional<Map<String, LocalResource>> resourceMap,
-      Path destDir, FileSystem localFs) throws IOException {
+  private void addAppJars(String jarFilePathList, Optional<Map<String, LocalResource>> resourceMap, Path destCacheDir, Path unsharedDir,
+      FileSystem localFs) throws IOException {
     for (String jarFilePath : SPLITTER.split(jarFilePathList)) {
       Path srcFilePath = new Path(jarFilePath);
-      Path destFilePath = new Path(destDir, srcFilePath.getName());
-      if (localFs.exists(srcFilePath)) {
-        this.fs.copyFromLocalFile(srcFilePath, destFilePath);
+      FileStatus localJar = localFs.getFileStatus(srcFilePath);
+      Path destFilePath = HdfsJarUploadUtils.calculateDestJarFile(this.fs, localJar, unsharedDir, destCacheDir);
+      if (HdfsJarUploadUtils.uploadJarToHdfs(fs, localJar, MAXIMUM_JAR_COPY_RETRY_TIMES_DEFAULT, destFilePath)) {
+        if (resourceMap.isPresent()) {
+          YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+        }
       } else {
-        LOGGER.warn("The src destination " + srcFilePath + " doesn't exists");
-      }
-      if (resourceMap.isPresent()) {
-        YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+        LOGGER.warn("Failed to upload jar file {} to HDFS", srcFilePath);

Review Comment:
   again, should this be an actual failure, not merely logging?
   
   ...or do we believe there are times when it's actually OK to continue?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "phet (via GitHub)" <gi...@apache.org>.
phet commented on code in PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#discussion_r1757111038


##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java:
##########
@@ -203,16 +203,36 @@ public static void setYarnClassPath(Config config, Configuration yarnConfigurati
     }
   }
 
-  public static Path getJarPathCacheAndCleanIfNeeded(Config config, FileSystem fs) throws IOException {
+  /**
+   * Calculate the path of a jar cache on HDFS, which is retained on a monthly basis.
+   * @param config
+   * @return
+   * @throws IOException
+   */
+  public static Path calculateJarCachePath(Config config) throws IOException {

Review Comment:
   not major, but re-reading I'd almost name this `calculatePerMonthJarCachePath`



##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java:
##########
@@ -203,16 +203,36 @@ public static void setYarnClassPath(Config config, Configuration yarnConfigurati
     }
   }
 
-  public static Path getJarPathCacheAndCleanIfNeeded(Config config, FileSystem fs) throws IOException {
+  /**
+   * Calculate the path of a jar cache on HDFS, which is retained on a monthly basis.
+   * @param config
+   * @return
+   * @throws IOException
+   */
+  public static Path calculateJarCachePath(Config config) throws IOException {
     Path jarsCacheDirMonthly = new Path(config.getString(GobblinYarnConfigurationKeys.JAR_CACHE_DIR));
     String monthSuffix = new SimpleDateFormat("yyyy-MM").format(config.getLong(GobblinYarnConfigurationKeys.YARN_APPLICATION_LAUNCHER_START_TIME_KEY));
+    return new Path(jarsCacheDirMonthly, monthSuffix);
+
+  }
+
+  /**
+   * Retain the latest k jar cache paths that are children of the parent cache path.
+   * @param parentCachePath
+   * @param k the number of latest jar cache paths to retain
+   * @param fs
+   * @return
+   * @throws IllegalAccessException
+   * @throws IOException
+   */
+  public static boolean retainKLatestJarCachePaths(Path parentCachePath, int k, FileSystem fs) throws IOException {
     // Cleanup old cache if necessary
-    List<FileStatus> jarDirs = Arrays.stream(fs.exists(jarsCacheDirMonthly)
-        ? fs.listStatus(jarsCacheDirMonthly) : new FileStatus[0]).sorted().collect(Collectors.toList());
-    if (jarDirs.size() > 2) {
-      fs.delete(jarDirs.get(0).getPath(), true);
+    List<FileStatus> jarDirs =
+        Arrays.stream(fs.exists(parentCachePath) ? fs.listStatus(parentCachePath) : new FileStatus[0]).sorted().collect(Collectors.toList());
+    if (jarDirs.size() > k) {
+      return fs.delete(jarDirs.get(0).getPath(), true);

Review Comment:
   since this is not a loop, it seems it would delete at most one dir even if there are more than 1 more than k.  is that ok?  if so, document in javadoc



##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -590,6 +590,15 @@ ApplicationId setupAndSubmitApplication() throws IOException, YarnException, Int
     amContainerLaunchContext.setEnvironment(YarnHelixUtils.getEnvironmentVariables(this.yarnConfiguration));
     amContainerLaunchContext.setCommands(Lists.newArrayList(buildApplicationMasterCommand(applicationId.toString(), resource.getMemory())));
 
+    if (this.jarCacheEnabled) {
+      Path jarCachePath = YarnHelixUtils.calculateJarCachePath(this.config);
+      // Retain at least the current and last month's jars to handle executions running for ~30 days max
+      boolean cleanedSuccessfully = YarnHelixUtils.retainKLatestJarCachePaths(jarCachePath.getParent(), 2, this.fs);

Review Comment:
   does this run before or after caching jars?  e.g. do we save only two prior AND THEN potentially add one more or we've added any new one already prior to retention paring it down to two?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "Will-Lo (via GitHub)" <gi...@apache.org>.
Will-Lo commented on code in PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#discussion_r1755527804


##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java:
##########
@@ -199,6 +203,18 @@ public static void setYarnClassPath(Config config, Configuration yarnConfigurati
     }
   }
 
+  public static Path getJarPathCacheAndCleanIfNeeded(Config config, FileSystem fs) throws IOException {

Review Comment:
   I set retainKLatestCachePaths to return false if any deletion failed rather than true iff there were any deletions, but this was a good idea.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "phet (via GitHub)" <gi...@apache.org>.
phet commented on code in PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#discussion_r1757074356


##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -720,26 +731,26 @@ private void addLibJars(Path srcLibJarDir, Optional<Map<String, LocalResource>>
     }
 
     for (FileStatus libJarFile : libJarFiles) {
-      Path destFilePath = new Path(destDir, libJarFile.getPath().getName());
-      this.fs.copyFromLocalFile(libJarFile.getPath(), destFilePath);
-      if (resourceMap.isPresent()) {
+      Path destFilePath = HdfsJarUploadUtils.calculateDestJarFile(this.fs, libJarFile, unsharedDir, destCacheDir);
+      if (HdfsJarUploadUtils.uploadJarToHdfs(fs, libJarFile, MAXIMUM_JAR_COPY_RETRY_TIMES_DEFAULT, destFilePath) && resourceMap.isPresent()) {
         YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+      } else {
+        LOGGER.warn("Failed to upload jar file {} to HDFS", libJarFile.getPath());
       }
     }
   }
-
-  private void addAppJars(String jarFilePathList, Optional<Map<String, LocalResource>> resourceMap,
-      Path destDir, FileSystem localFs) throws IOException {
+  private void addAppJars(String jarFilePathList, Optional<Map<String, LocalResource>> resourceMap, Path destCacheDir, Path unsharedDir,
+      FileSystem localFs) throws IOException {
     for (String jarFilePath : SPLITTER.split(jarFilePathList)) {
       Path srcFilePath = new Path(jarFilePath);
-      Path destFilePath = new Path(destDir, srcFilePath.getName());
-      if (localFs.exists(srcFilePath)) {
-        this.fs.copyFromLocalFile(srcFilePath, destFilePath);
+      FileStatus localJar = localFs.getFileStatus(srcFilePath);
+      Path destFilePath = HdfsJarUploadUtils.calculateDestJarFile(this.fs, localJar, unsharedDir, destCacheDir);
+      if (HdfsJarUploadUtils.uploadJarToHdfs(fs, localJar, MAXIMUM_JAR_COPY_RETRY_TIMES_DEFAULT, destFilePath)) {
+        if (resourceMap.isPresent()) {
+          YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+        }
       } else {
-        LOGGER.warn("The src destination " + srcFilePath + " doesn't exists");
-      }
-      if (resourceMap.isPresent()) {
-        YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+        LOGGER.warn("Failed to upload jar file {} to HDFS", srcFilePath);

Review Comment:
   I guess that's fine to start.  how about a `// TODO: decide whether to fail-fast here, given the job may be unable to run w/o it`



##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -720,26 +731,26 @@ private void addLibJars(Path srcLibJarDir, Optional<Map<String, LocalResource>>
     }
 
     for (FileStatus libJarFile : libJarFiles) {
-      Path destFilePath = new Path(destDir, libJarFile.getPath().getName());
-      this.fs.copyFromLocalFile(libJarFile.getPath(), destFilePath);
-      if (resourceMap.isPresent()) {
+      Path destFilePath = HdfsJarUploadUtils.calculateDestJarFile(this.fs, libJarFile, unsharedDir, destCacheDir);
+      if (HdfsJarUploadUtils.uploadJarToHdfs(fs, libJarFile, MAXIMUM_JAR_COPY_RETRY_TIMES_DEFAULT, destFilePath) && resourceMap.isPresent()) {
         YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+      } else {
+        LOGGER.warn("Failed to upload jar file {} to HDFS", libJarFile.getPath());
       }
     }
   }
-
-  private void addAppJars(String jarFilePathList, Optional<Map<String, LocalResource>> resourceMap,
-      Path destDir, FileSystem localFs) throws IOException {
+  private void addAppJars(String jarFilePathList, Optional<Map<String, LocalResource>> resourceMap, Path destCacheDir, Path unsharedDir,
+      FileSystem localFs) throws IOException {
     for (String jarFilePath : SPLITTER.split(jarFilePathList)) {
       Path srcFilePath = new Path(jarFilePath);
-      Path destFilePath = new Path(destDir, srcFilePath.getName());
-      if (localFs.exists(srcFilePath)) {
-        this.fs.copyFromLocalFile(srcFilePath, destFilePath);
+      FileStatus localJar = localFs.getFileStatus(srcFilePath);
+      Path destFilePath = HdfsJarUploadUtils.calculateDestJarFile(this.fs, localJar, unsharedDir, destCacheDir);
+      if (HdfsJarUploadUtils.uploadJarToHdfs(fs, localJar, MAXIMUM_JAR_COPY_RETRY_TIMES_DEFAULT, destFilePath)) {
+        if (resourceMap.isPresent()) {
+          YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+        }
       } else {
-        LOGGER.warn("The src destination " + srcFilePath + " doesn't exists");
-      }
-      if (resourceMap.isPresent()) {
-        YarnHelixUtils.addFileAsLocalResource(this.fs, destFilePath, LocalResourceType.FILE, resourceMap.get());
+        LOGGER.warn("Failed to upload jar file {} to HDFS", srcFilePath);

Review Comment:
   I guess that's fine to start.  how about a
   ```
   // TODO: decide whether to fail-fast here, given the job may be unable to run w/o it
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "Will-Lo (via GitHub)" <gi...@apache.org>.
Will-Lo commented on code in PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#discussion_r1755520391


##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java:
##########
@@ -484,12 +487,29 @@ private void requestContainer(Optional<String> preferredNode, Resource resource)
   protected ContainerLaunchContext newContainerLaunchContext(ContainerInfo containerInfo)
       throws IOException {
     Path appWorkDir = GobblinClusterUtils.getAppWorkDirPathFromConfig(this.config, this.fs, this.applicationName, this.applicationId);
+    Path containerJarsUnsharedDir = new Path(appWorkDir, GobblinYarnConfigurationKeys.CONTAINER_WORK_DIR_NAME);
+    Path jarCacheDir = this.jarCacheEnabled ? YarnHelixUtils.getJarPathCacheAndCleanIfNeeded(this.config, this.fs) : appWorkDir;
+    Path containerJarsCachedDir = new Path(jarCacheDir, GobblinYarnConfigurationKeys.CONTAINER_WORK_DIR_NAME);
+    LOGGER.info("Container cached jars root dir: " + containerJarsCachedDir);
+    LOGGER.info("Container uncached jars root dir: " + containerJarsUnsharedDir);
     Path containerWorkDir = new Path(appWorkDir, GobblinYarnConfigurationKeys.CONTAINER_WORK_DIR_NAME);
 
-    Map<String, LocalResource> resourceMap = Maps.newHashMap();
 
+    Map<String, LocalResource> resourceMap = Maps.newHashMap();
+    // Always fetch any jars from the appWorkDir for any potential snapshot jars
     addContainerLocalResources(new Path(appWorkDir, GobblinYarnConfigurationKeys.LIB_JARS_DIR_NAME), resourceMap);
-    addContainerLocalResources(new Path(containerWorkDir, GobblinYarnConfigurationKeys.APP_JARS_DIR_NAME), resourceMap);
+    if (this.config.hasPath(GobblinYarnConfigurationKeys.CONTAINER_FILES_LOCAL_KEY)) {
+      addContainerLocalResources(new Path(containerJarsUnsharedDir, GobblinYarnConfigurationKeys.APP_JARS_DIR_NAME),
+          resourceMap);
+    }
+    if (this.jarCacheEnabled) {
+      addContainerLocalResources(new Path(jarCacheDir, GobblinYarnConfigurationKeys.LIB_JARS_DIR_NAME), resourceMap);
+      if (this.config.hasPath(GobblinYarnConfigurationKeys.CONTAINER_FILES_LOCAL_KEY)) {

Review Comment:
   Oops should have used `gobblin.yarn.container.jars` key



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "Will-Lo (via GitHub)" <gi...@apache.org>.
Will-Lo commented on code in PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#discussion_r1757344912


##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java:
##########
@@ -203,16 +203,36 @@ public static void setYarnClassPath(Config config, Configuration yarnConfigurati
     }
   }
 
-  public static Path getJarPathCacheAndCleanIfNeeded(Config config, FileSystem fs) throws IOException {
+  /**
+   * Calculate the path of a jar cache on HDFS, which is retained on a monthly basis.
+   * @param config
+   * @return
+   * @throws IOException
+   */
+  public static Path calculateJarCachePath(Config config) throws IOException {
     Path jarsCacheDirMonthly = new Path(config.getString(GobblinYarnConfigurationKeys.JAR_CACHE_DIR));
     String monthSuffix = new SimpleDateFormat("yyyy-MM").format(config.getLong(GobblinYarnConfigurationKeys.YARN_APPLICATION_LAUNCHER_START_TIME_KEY));
+    return new Path(jarsCacheDirMonthly, monthSuffix);
+
+  }
+
+  /**
+   * Retain the latest k jar cache paths that are children of the parent cache path.
+   * @param parentCachePath
+   * @param k the number of latest jar cache paths to retain
+   * @param fs
+   * @return
+   * @throws IllegalAccessException
+   * @throws IOException
+   */
+  public static boolean retainKLatestJarCachePaths(Path parentCachePath, int k, FileSystem fs) throws IOException {
     // Cleanup old cache if necessary
-    List<FileStatus> jarDirs = Arrays.stream(fs.exists(jarsCacheDirMonthly)
-        ? fs.listStatus(jarsCacheDirMonthly) : new FileStatus[0]).sorted().collect(Collectors.toList());
-    if (jarDirs.size() > 2) {
-      fs.delete(jarDirs.get(0).getPath(), true);
+    List<FileStatus> jarDirs =
+        Arrays.stream(fs.exists(parentCachePath) ? fs.listStatus(parentCachePath) : new FileStatus[0]).sorted().collect(Collectors.toList());
+    if (jarDirs.size() > k) {
+      return fs.delete(jarDirs.get(0).getPath(), true);

Review Comment:
   Changed it to use a loop, for consistency with naming convention. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "Will-Lo (via GitHub)" <gi...@apache.org>.
Will-Lo commented on code in PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#discussion_r1757279937


##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -590,6 +590,15 @@ ApplicationId setupAndSubmitApplication() throws IOException, YarnException, Int
     amContainerLaunchContext.setEnvironment(YarnHelixUtils.getEnvironmentVariables(this.yarnConfiguration));
     amContainerLaunchContext.setCommands(Lists.newArrayList(buildApplicationMasterCommand(applicationId.toString(), resource.getMemory())));
 
+    if (this.jarCacheEnabled) {
+      Path jarCachePath = YarnHelixUtils.calculateJarCachePath(this.config);
+      // Retain at least the current and last month's jars to handle executions running for ~30 days max
+      boolean cleanedSuccessfully = YarnHelixUtils.retainKLatestJarCachePaths(jarCachePath.getParent(), 2, this.fs);

Review Comment:
   It runs after caching the jars. But it uses a consistent `YARN_APPLICATION_LAUNCHER_START_TIME_KEY` in the job so no matter how many times we look at the cache path it's only creating one path at most, and that path would be the ones where the jars are being uploaded.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#issuecomment-2346915466

   ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/4030?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 55.31%. Comparing base [(`e501b62`)](https://app.codecov.io/gh/apache/gobblin/commit/e501b62ffe646885d8b33198737bc16d38c04ba3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`72eb403`)](https://app.codecov.io/gh/apache/gobblin/commit/72eb403d117db10aec4f069169caa13b0bb41b5f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 23 commits behind head on master.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #4030      +/-   ##
   ============================================
   + Coverage     45.86%   55.31%   +9.45%     
   + Complexity     3257     1582    -1675     
   ============================================
     Files           707      307     -400     
     Lines         27865    10580   -17285     
     Branches       2796     1069    -1727     
   ============================================
   - Hits          12779     5852    -6927     
   + Misses        14008     4223    -9785     
   + Partials       1078      505     -573     
   ```
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/gobblin/pull/4030?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "phet (via GitHub)" <gi...@apache.org>.
phet commented on code in PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#discussion_r1762107332


##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java:
##########
@@ -229,10 +229,11 @@ public static boolean retainKLatestJarCachePaths(Path parentCachePath, int k, Fi
     // Cleanup old cache if necessary
     List<FileStatus> jarDirs =
         Arrays.stream(fs.exists(parentCachePath) ? fs.listStatus(parentCachePath) : new FileStatus[0]).sorted().collect(Collectors.toList());
-    if (jarDirs.size() > k) {
-      return fs.delete(jarDirs.get(0).getPath(), true);
+    boolean deletesSuccessful = true;
+    for (int i = 0; i < jarDirs.size() - k; i++) {
+      deletesSuccessful = deletesSuccessful && fs.delete(jarDirs.get(i).getPath(), true);

Review Comment:
   `&=`



##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java:
##########
@@ -590,6 +590,15 @@ ApplicationId setupAndSubmitApplication() throws IOException, YarnException, Int
     amContainerLaunchContext.setEnvironment(YarnHelixUtils.getEnvironmentVariables(this.yarnConfiguration));
     amContainerLaunchContext.setCommands(Lists.newArrayList(buildApplicationMasterCommand(applicationId.toString(), resource.getMemory())));
 
+    if (this.jarCacheEnabled) {
+      Path jarCachePath = YarnHelixUtils.calculateJarCachePath(this.config);
+      // Retain at least the current and last month's jars to handle executions running for ~30 days max
+      boolean cleanedSuccessfully = YarnHelixUtils.retainKLatestJarCachePaths(jarCachePath.getParent(), 2, this.fs);

Review Comment:
   would be great to add this info to a code comment here (in the caller, since it's the combination of the two being used, so less effective in either's javadoc).  the crux is that retention of K=2 won't save us in cases where K=3 might exceed any FS quotas.



##########
gobblin-utility/src/main/java/org/apache/gobblin/util/filesystem/HdfsJarUploadUtils.java:
##########
@@ -39,15 +39,15 @@ public class HdfsJarUploadUtils {
    * given the {@link FileStatus} of a jarFile and the path of directory that contains jar.
    * Snapshot dirs should not be shared, as different jobs may be using different versions of it.
    * @param fs
-   * @param localJar
+   * @param localJarPath
    * @param unsharedJarsDir
    * @param jarCacheDir
    * @return
    * @throws IOException
    */
-  public static Path calculateDestJarFile(FileSystem fs, FileStatus localJar, Path unsharedJarsDir, Path jarCacheDir) throws IOException {
-    Path uploadDir = localJar.getPath().getName().contains("SNAPSHOT") ? unsharedJarsDir : jarCacheDir;
-    Path destJarFile = new Path(fs.makeQualified(uploadDir), localJar.getPath().getName());
+  public static Path calculateDestJarFilePath(FileSystem fs, String localJarPath, Path unsharedJarsDir, Path jarCacheDir) throws IOException {
+    Path uploadDir = localJarPath.contains("SNAPSHOT") ? unsharedJarsDir : jarCacheDir;

Review Comment:
   seeing this invoked, `localJarPath` looks more like `localJarBasename`, not the (full) path.
   
   (or simply `jarName`)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2135] Cache Gobblin YARN application jars [gobblin]

Posted by "homatthew (via GitHub)" <gi...@apache.org>.
homatthew commented on code in PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#discussion_r1728046675


##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java:
##########
@@ -572,56 +573,22 @@ private void addJars(Path jarFileDir, String jarFileList, Configuration conf) th
     for (String jarFile : SPLITTER.split(jarFileList)) {
       Path srcJarFile = new Path(jarFile);
       FileStatus[] fileStatusList = lfs.globStatus(srcJarFile);
-
       for (FileStatus status : fileStatusList) {
+        Path destJarFile = HdfsJarUploadUtils.calculateDestJarFile(fs, status, this.unsharedJarsDir, jarFileDir);
         // For each FileStatus there are chances it could fail in copying at the first attempt, due to file-existence
         // or file-copy is ongoing by other job instance since all Gobblin jobs share the same jar file directory.
         // the retryCount is to avoid cases (if any) where retry is going too far and causes job hanging.
-        int retryCount = 0;
-        boolean shouldFileBeAddedIntoDC = true;
-        Path destJarFile = calculateDestJarFile(status, jarFileDir);
-        // Adding destJarFile into HDFS until it exists and the size of file on targetPath matches the one on local path.
-        while (!this.fs.exists(destJarFile) || fs.getFileStatus(destJarFile).getLen() != status.getLen()) {
-          try {
-            if (this.fs.exists(destJarFile) && fs.getFileStatus(destJarFile).getLen() != status.getLen()) {
-              Thread.sleep(WAITING_TIME_ON_IMCOMPLETE_UPLOAD);

Review Comment:
   This key seems unused now in this class. And might as well fix the typo in `HdfsJarUploadUtils`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org