[jira] [Created] (FLINK-11696) [checkpoint] Avoid to send mkdir requests to DFS from task side

["Yun Tang (JIRA)" <ji...@apache.org>]

Yun Tang created FLINK-11696:
--------------------------------

             Summary: [checkpoint] Avoid to send mkdir requests to DFS from task side
                 Key: FLINK-11696
                 URL: https://issues.apache.org/jira/browse/FLINK-11696
             Project: Flink
          Issue Type: Improvement
          Components: State Backends, Checkpointing
            Reporter: Yun Tang
            Assignee: Yun Tang


Currently, when we create checkpoint directory in the distributed file system. Not only {{CheckpointCoordinator}} but also {{FsCheckpointStorage}} in {{StreamTask}} would create the {{checkpointsDirectory}}, {{sharedStateDirectory}} and {{taskOwnedStateDirectory}}. These many {{mkdir}} RPC requests would cause a very high pressure on the distributed file system, especially when the parallelism is large or jobs continue to failover again and again.

We could avoid these {{mkdir}} requests from task side if writing to a distributed file system.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)