[GitHub] [iceberg] kezhuw edited a comment on issue #2033: Iceberg specific files (manifest files, manifest list file, metadata file) are not created, only Flink specific manifest file is created

[GitBox <gi...@apache.org>]

kezhuw edited a comment on issue #2033:
URL: https://github.com/apache/iceberg/issues/2033#issuecomment-766096422


   @sunhaibotb @zhijiangW @rkhachatryan  @pnowojski @kl0u @zentol @aljoscha 
   
   I think it is a bug of Flink and was introduced in [FLINK-14228](https://issues.apache.org/jira/browse/FLINK-14228)(apache/flink#9854, apache/flink#10151). The changes for [FLINK-14228](https://issues.apache.org/jira/browse/FLINK-14228) relay on `StreamTask.afterInvoke`  and `OperatorChain.closeOperators` will only be invoked after **end of input**. But that is not true after [FLIP-34: Terminate/Suspend Job with Savepoint ](https://cwiki.apache.org/confluence/x/JAglBg) long before. Task could enter a **finished** state after synchronous savepoint, that is an expected job suspension and stopping. I am also aware of [FLIP-147: Support Checkpoints After Tasks Finished](https://cwiki.apache.org/confluence/x/mw-ZCQ), I think maybe the three should make agreement on terminologies.
   
   Besides, I think FLIP-27 Source does not work with synchronous savepoint. @becketqin @StephanEwen
   
   I have pushed branch [synchronous-savepoint-conflict-with-bounded-end-input-case](https://github.com/kezhuw/flink/commits/synchronous-savepoint-conflict-with-bounded-end-input-case) in my local repository. Two test cases failed both:
   * `SavepointITCase.testStopSavepointWithBoundedInput` failed due to `BoundedOneInput.endInput` called.
   * `SavepointITCase.testStopSavepointWithFlip27Source` failed due to timeout.
   
   FLINK-14228: https://issues.apache.org/jira/browse/FLINK-14228
   FLIP-34: https://cwiki.apache.org/confluence/x/JAglBg
   FLIP-147: https://cwiki.apache.org/confluence/x/mw-ZCQ
   `StreamTask.finishTask`: https://github.com/apache/flink/blob/d20867a58d8f9ba7520d611d5fab01e6f98253a5/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L467


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org