[jira] [Updated] (FLINK-11133) FsCheckpointStorage is unaware about S3 entropy when creating directories

["Robert Metzger (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Metzger updated FLINK-11133:
-----------------------------------
    Component/s: FileSystems

> FsCheckpointStorage is unaware about S3 entropy when creating directories
> -------------------------------------------------------------------------
>
>                 Key: FLINK-11133
>                 URL: https://issues.apache.org/jira/browse/FLINK-11133
>             Project: Flink
>          Issue Type: Bug
>          Components: FileSystems
>    Affects Versions: 1.7.0
>            Reporter: Mark Cho
>            Priority: Minor
>
> We currently use S3 for our checkpoint storage with S3 entropy enabled.
> Entropy seems to be working correctly when writing out checkpoint metadata file (entropy key is correctly stripped from `state.checkpoints.dir`) and when writing out checkpoint data file (entropy key is correctly replaced with random string).
> However, from the logs, it seems like entropy key is not stripped or replaced when `FsCheckpointStorage` creates directories in the following class:
> [https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsCheckpointStorage.java#L83-L85]
> Should FsCheckpointStorage skip initializing mkdir calls if object store like S3 is used since S3 doesn't have directory concept?
> If we want to keep the `mkdir` calls in `FsCheckpointStorage`, we should handle the entropy key specified in `state.checkpoints.dir`. Currently, folder markers in S3 are being created by Hadoop FileSystem with the entropy key in the path as a result of `mkdir` calls in `FsCheckpointStorage`).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)