[GitHub] [hudi] yihua commented on pull request #5048: [HUDI-3634] Could read empty or partial HoodieCommitMetaData in downstream if using HDFS

[GitBox <gi...@apache.org>]

yihua commented on PR #5048:
URL: https://github.com/apache/hudi/pull/5048#issuecomment-1135460941

   @boneanxs there are two ways to address the issue:
   (1) if the main purpose is to make sure that the incremental reading does not skip the empty commit metadata which is being written, the read logic of commit metadata should consider such case.  Currently, `HoodieCommitMetadata.fromBytes()` uses `fromJsonString()` to parse the metadata and does not throw exception.  One way is to add a flag to control the behavior so that it can throw an exception for incremental reading, thus the empty commit metadata being written should fail the incremental read at the moment, guaranteeing no data loss.
   ```
     public static <T> T fromJsonString(String jsonStr, Class<T> clazz) throws Exception {
       if (jsonStr == null || jsonStr.isEmpty()) {
         // For empty commit file (no data or somethings bad happen).
         return clazz.newInstance();
       }
       return JsonUtils.getObjectMapper().readValue(jsonStr, clazz);
     }
   ```
   (2) If you do want to fix the write side, you can create new API name in `HoodieWrapperFileSystem` sth like `HoodieWrapperFileSystem::createImmutableFileInPath()` and differentiate the logic based on the storage scheme, i.e., `fs.getScheme()`, inside the method.  Only for HDFS, the new logic is introduced and it's guarded by the flag.  Then, the new API is called in `HoodieActiveTimeline` and `HoodieActiveTimeline` should not be aware of any file system level behavior difference. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org