[jira] [Commented] (FLINK-11696) Avoid to send mkdir requests to DFS from task side

["frank wang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890620#comment-16890620 ] 

frank wang commented on FLINK-11696:
------------------------------------

We have also encountered similar problems. We originally had two options. One is to add random numbers when creating directories, and avoid creating them at the same time. One is to give the creation all to the jobmaster to complete, avoiding duplicate creation.

> Avoid to send mkdir requests to DFS from task side
> --------------------------------------------------
>
>                 Key: FLINK-11696
>                 URL: https://issues.apache.org/jira/browse/FLINK-11696
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Checkpointing
>            Reporter: Yun Tang
>            Assignee: Yun Tang
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently, when we create checkpoint directory in the distributed file system. Not only {{CheckpointCoordinator}} but also {{FsCheckpointStorage}} in {{StreamTask}} would create the {{checkpointsDirectory}}, {{sharedStateDirectory}} and {{taskOwnedStateDirectory}}. These many {{mkdir}} RPC requests would cause a very high pressure on the distributed file system, especially when the parallelism is large or jobs continue to failover again and again.
> We could avoid these {{mkdir}} requests from task side if writing to a distributed file system.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)