[GitHub] [hudi] slfan1989 commented on a diff in pull request #9065: [HUDI-6451] Randomly obtain a path in HoodieMemoryConfig#getDefaultSpillableMapBasePath

["slfan1989 (via GitHub)" <gi...@apache.org>]

slfan1989 commented on code in PR #9065:
URL: https://github.com/apache/hudi/pull/9065#discussion_r1244702211


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieMemoryConfig.java:
##########
@@ -140,7 +144,9 @@ private HoodieMemoryConfig() {
 
   public static String getDefaultSpillableMapBasePath() {
     String[] localDirs = FileIOUtils.getConfiguredLocalDirs();
-    return (localDirs != null && localDirs.length > 0) ? localDirs[0] : "/tmp/";
+    List<String> localDirLists = Arrays.asList(localDirs);
+    Collections.shuffle(localDirLists);
+    return !localDirLists.isEmpty() ? localDirLists.get(0) : "/tmp/";

Review Comment:
   Thank you very much for helping with the code review! `LOCAL_DIRS` is an environment variable set by YARN during Container startup, taking into consideration disk space availability and cases where a disk may be inaccessible. 
   
   - Hudi#FileIOUtils#getYarnLocalDirs
   
   ```
     private static String getYarnLocalDirs() {
       String localDirs = System.getenv("LOCAL_DIRS");
   
       if (localDirs == null) {
         throw new HoodieIOException("Yarn Local dirs can't be empty");
       }
       return localDirs;
     }
   ```
   
   - YARN
   
   LinuxContainerExecutor#prepareContainer
   ```
   builder.setExecutionAttribute(LOCALIZED_RESOURCES,
               ctx.getLocalizedResources())
           .setExecutionAttribute(USER, ctx.getUser())
           .setExecutionAttribute(CONTAINER_LOCAL_DIRS,
               ctx.getContainerLocalDirs())
           .setExecutionAttribute(CONTAINER_RUN_CMDS, ctx.getCommands())
           .setExecutionAttribute(CONTAINER_ID_STR,
               ctx.getContainer().getContainerId().toString());
   ```
   
   ContainerLaunch#call
   ```
   
   List<String> containerLocalDirs = getContainerLocalDirs(localDirs);
   
   if (truststore != null) {
      addTruststoreVars(environment, containerWorkDir);
   }
   
   prepareContainer(localResources, containerLocalDirs);
   
   // Write out the environment
   exec.writeLaunchEnv(containerScriptOutStream, environment,
           localResources, launchContext.getCommands(), containerLogDir, user, nmEnvVars);
   ```
   
   ContainerLaunch#prepareContainer
   ```
     protected List<String> getContainerLocalDirs(List<String> localDirs) {
       List<String> containerLocalDirs = new ArrayList<>(localDirs.size());
       String user = container.getUser();
       String appIdStr = app.getAppId().toString();
       String relativeContainerLocalDir = ContainerLocalizer.USERCACHE
           + Path.SEPARATOR + user + Path.SEPARATOR + ContainerLocalizer.APPCACHE
           + Path.SEPARATOR + appIdStr + Path.SEPARATOR;
   
       for (String localDir : localDirs) {
         containerLocalDirs.add(localDir + Path.SEPARATOR
             + relativeContainerLocalDir);
       }
   
       return containerLocalDirs;
     }
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org