You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/19 17:13:59 UTC

[GitHub] [hudi] nsivabalan commented on a change in pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties

nsivabalan commented on a change in pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#discussion_r753374574



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath, String payloadClassName
 
   /**
    * For serializing and de-serializing.
-   *
    */
   public HoodieTableConfig() {
     super();
   }
 
+  private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+    Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+    try (FSDataInputStream is = fs.open(cfgPath)) {
+      props.load(is);
+    } catch (IOException ioe) {
+      if (!fs.exists(cfgPath)) {

Review comment:
       Is it intention we check fs.exists() within catch block and not upfront? 

##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath, String payloadClassName
 
   /**
    * For serializing and de-serializing.
-   *
    */
   public HoodieTableConfig() {
     super();
   }
 
+  private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+    Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+    try (FSDataInputStream is = fs.open(cfgPath)) {
+      props.load(is);
+    } catch (IOException ioe) {
+      if (!fs.exists(cfgPath)) {
+        LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
+        // try the backup. this way no query ever fails if update fails midway.
+        Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+        try (FSDataInputStream is = fs.open(backupCfgPath)) {
+          props.load(is);
+        }
+      } else {
+        throw ioe;
+      }
+    }
+  }
+
+  public static void recover(FileSystem fs, Path metadataFolder) throws IOException {
+    Path cfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE);
+    Path backupCfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+    recoverIfNeeded(fs, cfgPath, backupCfgPath);
+  }
+
+  static void recoverIfNeeded(FileSystem fs, Path cfgPath, Path backupCfgPath) throws IOException {
+    if (!fs.exists(cfgPath)) {

Review comment:
       are there chances of partial/corrupted HOODIE_PROPERTIES_FILE during updates? If yes, can we do something like: irrespective of whether HOODIE_PROPERTIES_FILE exists or not, if HOODIE_PROPERTIES_FILE_BACKUP exists, we override the HOODIE_PROPERTIES_FILE with contents from HOODIE_PROPERTIES_FILE_BACKUP. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org