[jira] [Created] (FLINK-11133) FsCheckpointStorage is unaware about S3 entropy when creating directories

["Mark Cho (JIRA)" <ji...@apache.org>]

Mark Cho created FLINK-11133:
--------------------------------

             Summary: FsCheckpointStorage is unaware about S3 entropy when creating directories
                 Key: FLINK-11133
                 URL: https://issues.apache.org/jira/browse/FLINK-11133
             Project: Flink
          Issue Type: Bug
    Affects Versions: 1.7.0
            Reporter: Mark Cho


We currently use S3 for our checkpoint storage with S3 entropy enabled.

Entropy seems to be working correctly when writing out checkpoint metadata file (entropy key is correctly stripped from `state.checkpoints.dir`) and when writing out checkpoint data file (entropy key is correctly replaced with random string).

However, from the logs, it seems like entropy key is not stripped or replaced when `FsCheckpointStorage` creates directories in the following class:

[https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStorage.java#L83-L85]

Should FsCheckpointStorage skip initializing mkdir calls if object store like S3 is used since S3 doesn't have directory concept?

If we want to keep the `mkdir` calls in `FsCheckpointStorage`, we should handle the entropy key specified in `state.checkpoints.dir`. Currently, folder markers in S3 are being created by Hadoop FileSystem with the entropy key in the path as a result of `mkdir` calls in `FsCheckpointStorage`).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)