[GitHub] flink issue #3522: [FLINK-5823] [checkpoints] State Backends also handle Che...

[StephanEwen <gi...@git.apache.org>]

Github user StephanEwen commented on the issue:

    https://github.com/apache/flink/pull/3522
  
    I would like to merge this for 1.3 with a slightly reduced scope.
    
    I would include the refactoring to make the State Backends also responsible for storing the Metadata.
    
    I would NOT merge the started work about cleanup hooks. The reason for that is that after some digging through this, I found we actually have to treat different file systems quite differently. For example for HDFS and posix-style file systems, we want to dispose the checkpoint-exclusive state via a `rm -r`.
    However, for S3 we explicitly do not want to do that, but issue DELETE commands for each state object independently, because Flink has a more consistent view on what objects exist for a checkpoint than S3 has. Also, because a `rm -r` on the S3 file system means "list objects with path prefix; foreach object: delete" which is even worse.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---