Re: [PR] Make the creation of properties file atomic during modification [hudi]

["Rajeev-01 (via GitHub)" <gi...@apache.org>]

Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2279209679


##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
   private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
     StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
     StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+    StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);

Review Comment:
   Yeah @danny0405 there are multiple issues that can happen right now due to file creation and post copy. Let say in hdfs, our table path is running out of quota and it reached it max. For now it reached it's limit in diskspace quota and with the create and copy of actual file directly, will end up in creation of 0 bytes properties file (Since there is no issue of namespace quota) and while copying, it will fail due to quota issue. And that's how original file will be present with 0 bytes. With this rename approach, all the intermittent quota issues or namenode latency issues will be fallback to temp file and original file will be populated once the temp file is populated correctly. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org