You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Till Rohrmann (JIRA)" <ji...@apache.org> on 2019/04/28 11:55:00 UTC

[jira] [Commented] (FLINK-12343) Allow set hdfs.replication in Yarn Configuration

    [ https://issues.apache.org/jira/browse/FLINK-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16827957#comment-16827957 ] 

Till Rohrmann commented on FLINK-12343:
---------------------------------------

Isn't the replication factor an HDFS configuration you need to specify when starting the HDFS cluster or can Yarn applications specify this on their own?

> Allow set hdfs.replication in Yarn Configuration
> ------------------------------------------------
>
>                 Key: FLINK-12343
>                 URL: https://issues.apache.org/jira/browse/FLINK-12343
>             Project: Flink
>          Issue Type: Improvement
>          Components: Command Line Client, Deployment / YARN
>    Affects Versions: 1.6.4, 1.7.2, 1.8.0
>            Reporter: Zhenqiu Huang
>            Assignee: Zhenqiu Huang
>            Priority: Minor
>
> Currently, FlinkYarnSessionCli upload jars into hdfs with default 3 replications. From our production experience, we find that 3 replications will block big job (256 containers) to launch, when the HDFS is slow due to big workload for batch pipelines. Thus, we want to make the factor customizable from FlinkYarnSessionCli by adding an option.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)