[GitHub] [hudi] danny0405 commented on a change in pull request #2929: [HUDI-1886] Avoid to generates corrupted files for flink sink

[GitBox <gi...@apache.org>]

danny0405 commented on a change in pull request #2929:
URL: https://github.com/apache/hudi/pull/2929#discussion_r629011753



##########
File path: hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteFunction.java
##########
@@ -142,6 +142,22 @@
    */
   private transient TotalSizeTracer tracer;
 
+  /**
+   * Flag saying whether the write task is waiting for the checkpoint success notification
+   * after it finished a checkpoint.
+   *
+   * <p>The flag is needed because the write task does not block during the waiting time interval,
+   * some data buckets still flush out with old instant time. There are two cases that the flush may produce
+   * corrupted files if the old instant is committed successfully:
+   * 1) the write handle was writing data but interrupted, left a corrupted parquet file;
+   * 2) the write handle finished the write but was not closed, left an empty parquet file.
+   *
+   * <p>To solve, when this flag was set to true, we flush the data buffer with a new instant time = old instant time + 1ms,
+   * the new instant time would affect the write file name. The filesystem view does not recognize the file as committed because
+   * it always filters the data files based on successful commit time.

Review comment:
       It is not invalid, if the write success and there is no retry, the write data is still valid and the reader can see the data, for current instant. If there is a retry for the cases i list here, the data would be seen as invalid and thus does not cause confuse to the filesystem view, we then can go on with the write.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org