You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/01/31 06:14:07 UTC
[GitHub] [flink] HuangZhenQiu opened a new pull request #10980:
[FLINK-12343][yarn] add yarn file replication option
HuangZhenQiu opened a new pull request #10980: [FLINK-12343][yarn] add yarn file replication option
URL: https://github.com/apache/flink/pull/10980
## What is the purpose of the change
Add yarn file replication option, so that users can specify how many replications of a hdfs file (for example, the flink libs job jars) should be created.
## Brief change log
Added yarn config option yarn.file-replication, so that Flink user can add replication number for large jobs ( more than 256 containers) to efficiently download jars into each node manager and start containers.
## Verifying this change
This change added tests and can be verified as follows:
- Added integration tests for end-to-end deployment the replication option configured to 4.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][yarn] add yarn file replication option
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][yarn] add yarn file replication option
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r381112023
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -261,7 +263,8 @@ private static void testCopySingleFileFromLocal(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ DFSConfigKeys.DFS_REPLICATION_DEFAULT);
Review comment:
I just to make it use default value here. I think we can still create multiple file replications on single datanode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373809082
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
##########
@@ -122,6 +136,32 @@ private void deployPerjob(YarnConfigOptions.UserJarInclusion userJarInclusion, J
assertThat(jobResult, is(notNullValue()));
assertThat(jobResult.getSerializedThrowable().isPresent(), is(false));
+ if (configuration.getInteger(YarnConfigOptions.FILE_REPLICATION) != -1) {
Review comment:
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375630913
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java
##########
@@ -682,6 +683,10 @@ private ApplicationReport startAppMaster(
// add user code jars from the provided JobGraph
: jobGraph.getUserJars().stream().map(f -> f.toUri()).map(File::new).collect(Collectors.toSet());
+ int yarnFileReplication = yarnConfiguration.getInt(DFSConfigKeys.DFS_REPLICATION_KEY, DFSConfigKeys.DFS_REPLICATION_DEFAULT);
+ int fileReplication = flinkConfiguration.getInteger(YarnConfigOptions.FILE_REPLICATION);
Review comment:
what if user configured `YarnConfigOptions.FILE_REPLICATION` with `0`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373913383
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -199,7 +199,8 @@ static void testCopyFromLocalRecursive(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ 1);
Review comment:
Agree with @walterddr. It should be ok to use 1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r381838514
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -261,7 +263,8 @@ private static void testCopySingleFileFromLocal(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ DFSConfigKeys.DFS_REPLICATION_DEFAULT);
Review comment:
No, we could not create multiple file replication on single datanode. The file will not be accessible until enough datanode has been registered.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373913085
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -376,6 +384,14 @@ public static void writeYarnSiteConfigXML(Configuration yarnConf, File targetFol
}
}
+ public static void writeHDFSCoreSiteConfigXML(Configuration coreSite, File targetFolder) throws IOException {
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375631555
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
##########
@@ -64,32 +69,46 @@
@BeforeClass
public static void setup() {
YARN_CONFIGURATION.set(YarnTestBase.TEST_CLUSTER_NAME_KEY, "flink-yarn-tests-per-job");
- startYARNWithConfig(YARN_CONFIGURATION);
+ startYARNWithConfig(YARN_CONFIGURATION, true);
Review comment:
I'd suggest you just create a new file and write tests about yarn file replication, without any modification on `YARNITCase`. It doesn't no harm we add a new file(test case) and if we keep modifying existing file for "patching" logics we highly possibly run into mess.
Besides, still you don't test
```
configuration.setInteger(YarnConfigOptions.FILE_REPLICATION, 4);
```
with other value than `4`, especially edge case such as `< -1` `== -1`(possibly `<= -1` is ok) `== 0` `== 1`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375633169
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
##########
@@ -64,32 +69,46 @@
@BeforeClass
public static void setup() {
YARN_CONFIGURATION.set(YarnTestBase.TEST_CLUSTER_NAME_KEY, "flink-yarn-tests-per-job");
- startYARNWithConfig(YARN_CONFIGURATION);
+ startYARNWithConfig(YARN_CONFIGURATION, true);
Review comment:
A reason that I suggest add a new file is that previously we test `YARNITCase` with `withDFS == false` but now they are all `true`. I don't see an obvious reason to do it and we possibly loss test coverage.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147664218 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147761655 TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922 TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
Hash:f522e48ac9e3478ac88c6de8389d529a17c73377 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5297 TriggerType:PUSH TriggerID:f522e48ac9e3478ac88c6de8389d529a17c73377
Hash:f522e48ac9e3478ac88c6de8389d529a17c73377 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/149583795 TriggerType:PUSH TriggerID:f522e48ac9e3478ac88c6de8389d529a17c73377
Hash:4efb9fcbd694ec4b60dd8c8790a076bac9439438 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:4efb9fcbd694ec4b60dd8c8790a076bac9439438
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
* d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885)
* 570a3b05bc858f96a98da192d03e147585ce44cc Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147761655) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922)
* f522e48ac9e3478ac88c6de8389d529a17c73377 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149583795) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5297)
* 4efb9fcbd694ec4b60dd8c8790a076bac9439438 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r376202440
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -684,12 +710,28 @@ private static void start(YarnConfiguration conf, String principal, String keyta
}
+ private static void setMiniDFSCluster(String principal, String keytab, File targetTestClassesFolder) throws Exception {
+ if (miniDFSCluster == null) {
+ Configuration hdfsConfiguration = new Configuration();
+ hdfsConfiguration.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, tmpHDFS.getRoot().getAbsolutePath());
+ miniDFSCluster = new MiniDFSCluster
+ .Builder(hdfsConfiguration)
+ .numDataNodes(2)
Review comment:
Why do you set the number of datanode to 2 explicitly? I think the default replication is 3. If you only start 2 datanodes, could it work?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r376196890
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
##########
@@ -54,9 +54,9 @@
*/
public class YARNITCase extends YarnTestBase {
- private final Duration yarnAppTerminateTimeout = Duration.ofSeconds(10);
+ protected final Duration yarnAppTerminateTimeout = Duration.ofSeconds(10);
Review comment:
Do we really need to set these two variables to `protected`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375631555
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
##########
@@ -64,32 +69,46 @@
@BeforeClass
public static void setup() {
YARN_CONFIGURATION.set(YarnTestBase.TEST_CLUSTER_NAME_KEY, "flink-yarn-tests-per-job");
- startYARNWithConfig(YARN_CONFIGURATION);
+ startYARNWithConfig(YARN_CONFIGURATION, true);
Review comment:
I'd suggest you just create a new file and write class about yarn file replication, without any modification on `YARNITCase`. It doesn't no harm we add a new file(test case) and if we keep modifying existing file for "patching" logics we highly possibly run into mess.
Besides, still you don't test
```
configuration.setInteger(YarnConfigOptions.FILE_REPLICATION, 4);
```
with other value than `4`, especially edge case such as `< -1` `== -1`(possibly `<= -1` is ok) `== 0` `== 1`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147664218 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
* d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375632299
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -199,7 +199,8 @@ static void testCopyFromLocalRecursive(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ 1);
Review comment:
@walterddr what I'm concerned about is that existing logic should perfectly respect with default value, instead a configured non-default value. Otherwise it is suspected we break existing manner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-588822780
> > > Do you have a test to update the DFS_REPLICATION_KEY in hdfs-site.xml? And then the replication here should be same. I am not sure we could get a updated value from yarnConfiguration. Since YarnConfiguration do not add the hdfs-site.xml as resource by default.
> > > > Thanks for giving feedback. Resolved most of them. For the YarnConfiguration, yes you are right. By default, the hdfs-site.xml is not loaded. Thus, Currently Flink always uses default value. To honor the value frrom hdfs-site.xml, I think we need a sperate PR.
>
> I have verified that it could work. Because when we call `FileSystem.get(yarnConfiguration)`, the `core-site.xml` and `hdfs-site.xml` will be added to resource automatically.
Thanks for your testing!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r381109259
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
##########
@@ -54,9 +54,9 @@
*/
public class YARNITCase extends YarnTestBase {
- private final Duration yarnAppTerminateTimeout = Duration.ofSeconds(10);
+ protected final Duration yarnAppTerminateTimeout = Duration.ofSeconds(10);
Review comment:
Good catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373808876
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -962,6 +1000,11 @@ public static void teardown() throws Exception {
yarnCluster = null;
}
+ if (miniDFSCluster != null) {
+ LOG.info("Stopping MiniDFS Cluster");
+ miniDFSCluster.shutdown();
Review comment:
```suggestion
miniDFSCluster.shutdown();
miniDfsCluster = null;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375632453
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -199,7 +199,8 @@ static void testCopyFromLocalRecursive(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ 1);
Review comment:
To clarify breaking change could be ok but it should be explicit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r381873829
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -261,7 +263,8 @@ private static void testCopySingleFileFromLocal(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ DFSConfigKeys.DFS_REPLICATION_DEFAULT);
Review comment:
Though we don't access the file so the test pass, it seems we start a mini HDFS cluster which has one datanode so that set to 1 is more expressive?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r382097527
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -261,7 +263,8 @@ private static void testCopySingleFileFromLocal(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ DFSConfigKeys.DFS_REPLICATION_DEFAULT);
Review comment:
I think my argument is a little bit misleading. For real HDFS cluster, name node will prevent more than one replica of a block from been placed to the same data node. From my debugging experience, there is no such limitation for MiniHDFSCluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
* d846f33ba278b1d75a1eac672c7efebee1350e8f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373913342
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java
##########
@@ -682,6 +683,9 @@ private ApplicationReport startAppMaster(
// add user code jars from the provided JobGraph
: jobGraph.getUserJars().stream().map(f -> f.toUri()).map(File::new).collect(Collectors.toSet());
+ int fileReplication = yarnConfiguration.getInt(DFSConfigKeys.DFS_REPLICATION_KEY, DFSConfigKeys.DFS_REPLICATION_DEFAULT);
+ fileReplication = flinkConfiguration.getInteger(YarnConfigOptions.FILE_REPLICATION, fileReplication);
Review comment:
Yes, you are right. I changed the logic to if the configured value is less than default value -1. It will fall back to use the yarn replication configuration. The logic is tested in another test case in YarnITCase.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r376217472
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java
##########
@@ -682,6 +683,10 @@ private ApplicationReport startAppMaster(
// add user code jars from the provided JobGraph
: jobGraph.getUserJars().stream().map(f -> f.toUri()).map(File::new).collect(Collectors.toSet());
+ int yarnFileReplication = yarnConfiguration.getInt(DFSConfigKeys.DFS_REPLICATION_KEY, DFSConfigKeys.DFS_REPLICATION_DEFAULT);
+ int fileReplication = flinkConfiguration.getInteger(YarnConfigOptions.FILE_REPLICATION);
Review comment:
Do you have a test to update the `DFS_REPLICATION_KEY` in hdfs-site.xml? And then the replication here should be same. I am not sure we could get a updated value from `yarnConfiguration`. Since `YarnConfiguration` do not add the `hdfs-site.xml` as resource by default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373809067
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -199,7 +199,8 @@ static void testCopyFromLocalRecursive(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ 1);
Review comment:
I don't think this is going to be a problem since it is only testing a package-private functionality
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r381109231
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNFileReplicationITCase.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.yarn;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.yarn.configuration.YarnConfigOptions;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Test cases for the deployment of Yarn Flink clusters with customized file replication numbers.
+ */
+public class YARNFileReplicationITCase extends YARNITCase {
+
+ @BeforeClass
+ public static void setup() {
+ YARN_CONFIGURATION.set(YarnTestBase.TEST_CLUSTER_NAME_KEY, "flink-yarn-tests-per-job");
+ startYARNWithConfig(YARN_CONFIGURATION, true);
+ }
+
+ @Test
+ public void testPerJobModeWithCustomizedFileReplication() throws Exception {
+ Configuration configuration = createDefaultConfiguration(YarnConfigOptions.UserJarInclusion.DISABLED);
+ configuration.setInteger(YarnConfigOptions.FILE_REPLICATION, 4);
+
+ runTest(() -> deployPerjob(
+ configuration,
+ getTestingJobGraph()));
+ }
+
+ @Test
+ public void testPerJobModeWithDefaultFileReplication() throws Exception {
+ Configuration configuration = createDefaultConfiguration(YarnConfigOptions.UserJarInclusion.DISABLED);
+
+ runTest(() -> deployPerjob(
+ configuration,
+ getTestingJobGraph()));
+ }
+
+ @Override
+ protected void extraVerification(Configuration configuration, ApplicationId applicationId) throws Exception {
+ final FileSystem fs = FileSystem.get(getYarnConfiguration());
+
+ String suffix = ".flink/" + applicationId.toString() + "/" + flinkUberjar.getName();
+
+ Path uberJarHDFSPath = new Path(fs.getHomeDirectory(), suffix);
+ FileStatus fsStatus = fs.getFileStatus(uberJarHDFSPath);
+
+ final int flinkFileReplication = configuration.getInteger(YarnConfigOptions.FILE_REPLICATION);
+ final int replication = YARN_CONFIGURATION.getInt(DFSConfigKeys.DFS_REPLICATION_KEY, DFSConfigKeys.DFS_REPLICATION_DEFAULT);
+
+ // If YarnConfigOptions.FILE_REPLICATION is not set. The replication number should equals to yarn configuration value.
+ int expectedReplication = flinkFileReplication > 0
+ ? flinkFileReplication : replication;
+ assertEquals(expectedReplication, fsStatus.getReplication());
+
+ Path appPath = uberJarHDFSPath.getParent();
+ FileStatus[] fileStatuses = fs.listStatus(appPath, new PathFilter() {
+ @Override
+ public boolean accept(Path path) {
+ return path.getName().endsWith("taskmanager-conf.yaml");
Review comment:
Yes. I saw your change to remove the task-manager flink-conf.yaml. It reduced the container start time. It should be removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373808855
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -592,14 +608,14 @@ protected ApplicationReport getOnlyApplicationReport() throws IOException, YarnE
}
public static void startYARNSecureMode(YarnConfiguration conf, String principal, String keytab) {
- start(conf, principal, keytab);
+ start(conf, principal, keytab, false);
}
- public static void startYARNWithConfig(YarnConfiguration conf) {
- start(conf, null, null);
+ public static void startYARNWithConfig(YarnConfiguration conf, boolean withDFS) {
+ start(conf, null, null, withDFS);
}
- private static void start(YarnConfiguration conf, String principal, String keytab) {
+ private static void start(YarnConfiguration conf, String principal, String keytab, boolean withDFS) {
Review comment:
+1 to Tison's idea. whatsmore, I don't think we need to change the API here. since we can just check whether `FILE_REPLICATION` has been set in Flink Configuration. or whether `DFSConfigKeys.DFS_REPLICATION_KEY` has been set in YarnConfiguration
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r381837522
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -684,12 +710,28 @@ private static void start(YarnConfiguration conf, String principal, String keyta
}
+ private static void setMiniDFSCluster(String principal, String keytab, File targetTestClassesFolder) throws Exception {
+ if (miniDFSCluster == null) {
+ Configuration hdfsConfiguration = new Configuration();
+ hdfsConfiguration.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, tmpHDFS.getRoot().getAbsolutePath());
+ miniDFSCluster = new MiniDFSCluster
+ .Builder(hdfsConfiguration)
+ .numDataNodes(2)
Review comment:
I have checked the DFSMiniCluster code. When we set the num of datanodes to 2, the `dfs.replication` will also be updated. Make sense to leave 2 here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r376218611
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -261,7 +263,8 @@ private static void testCopySingleFileFromLocal(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ DFSConfigKeys.DFS_REPLICATION_DEFAULT);
Review comment:
By default, we only start 1 datanode. So it is not reasonable to set to `DFSConfigKeys.DFS_REPLICATION_DEFAULT` which value is 3 here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373784845
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
##########
@@ -122,6 +136,32 @@ private void deployPerjob(YarnConfigOptions.UserJarInclusion userJarInclusion, J
assertThat(jobResult, is(notNullValue()));
assertThat(jobResult.getSerializedThrowable().isPresent(), is(false));
+ if (configuration.getInteger(YarnConfigOptions.FILE_REPLICATION) != -1) {
Review comment:
Could you please add a new test case instead of modify current test cases? I think this pull request adds a new feature instead of code refactoring so that we should not touch any current test cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r381109674
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -684,12 +710,28 @@ private static void start(YarnConfiguration conf, String principal, String keyta
}
+ private static void setMiniDFSCluster(String principal, String keytab, File targetTestClassesFolder) throws Exception {
Review comment:
The original function also handles with secure hdfs access for test cases. It is not used for now. Let me add it if there is a need for secured test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r376197225
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
##########
@@ -54,9 +54,9 @@
*/
public class YARNITCase extends YarnTestBase {
- private final Duration yarnAppTerminateTimeout = Duration.ofSeconds(10);
+ protected final Duration yarnAppTerminateTimeout = Duration.ofSeconds(10);
- private final int sleepIntervalInMS = 100;
+ protected final int sleepIntervalInMS = 100;
@Rule
public final TemporaryFolder temporaryFolder = new TemporaryFolder();
Review comment:
We could add a `hotfix` to remove this unused variable `temporaryFolder`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r376199849
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -376,6 +384,14 @@ public static void writeYarnSiteConfigXML(Configuration yarnConf, File targetFol
}
}
+ private static void writeHDFSCoreSiteConfigXML(Configuration coreSite, File targetFolder) throws IOException {
Review comment:
Maybe the name `writeHDFSSiteConfigXML` makes more senses. Since we do not write the `core-site.xml` file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-588833119
The changes LGTM now.
@HuangZhenQiu Please make sure you have tested in the following cases in a real cluster
* Do not set `yarn.file-replication` in Flink configuration and do not update `dfs.replication` in hdfs-site.xml in client, then replication should be 3
* Do not set `yarn.file-replication` in Flink configuration and update `dfs.replication` to 5 in hdfs-site.xml in client, the replication should be 5
* Set `yarn.file-replication` to 7 in Flink configuration and update `dfs.replication` to 5 in hdfs-site.xml in client, then the replication should be 7
@TisonKun Let's wait for @HuangZhenQiu's final verification and merge. Thanks a lot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10980: [FLINK-12343][yarn] add
yarn file replication option
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10980: [FLINK-12343][yarn] add yarn file replication option
URL: https://github.com/apache/flink/pull/10980#issuecomment-580598389
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 (Fri Jan 31 06:16:16 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r381111087
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java
##########
@@ -171,6 +171,19 @@
" setting and use cluster default priority. Please refer to YARN's official documentation for specific" +
" settings required to enable priority scheduling for the targeted YARN version.");
+ /**
+ * Yarn session client uploads flink jar and user libs to file system (hdfs/s3) as local resource for yarn
+ * application context. The replication number changes the how many replica of each of these files in hdfs/s3.
+ * It is useful to accelerate this container bootstrap, when a Flink application needs more than one hundred
+ * of containers. If it is configured, Flink will use the default replication value in hadoop configuration.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373784864
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -376,6 +384,14 @@ public static void writeYarnSiteConfigXML(Configuration yarnConf, File targetFol
}
}
+ public static void writeHDFSCoreSiteConfigXML(Configuration coreSite, File targetFolder) throws IOException {
Review comment:
why should it be `public`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147664218 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147761655 TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922 TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
Hash:f522e48ac9e3478ac88c6de8389d529a17c73377 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5297 TriggerType:PUSH TriggerID:f522e48ac9e3478ac88c6de8389d529a17c73377
Hash:f522e48ac9e3478ac88c6de8389d529a17c73377 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/149583795 TriggerType:PUSH TriggerID:f522e48ac9e3478ac88c6de8389d529a17c73377
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
* d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885)
* 570a3b05bc858f96a98da192d03e147585ce44cc Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147761655) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922)
* f522e48ac9e3478ac88c6de8389d529a17c73377 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149583795) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5297)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375632453
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -199,7 +199,8 @@ static void testCopyFromLocalRecursive(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ 1);
Review comment:
To clarify breaking change could be ok but it should be explicit. IIUC this patch add a new feature without any change to existing logics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373784779
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -592,14 +608,14 @@ protected ApplicationReport getOnlyApplicationReport() throws IOException, YarnE
}
public static void startYARNSecureMode(YarnConfiguration conf, String principal, String keytab) {
- start(conf, principal, keytab);
+ start(conf, principal, keytab, false);
}
- public static void startYARNWithConfig(YarnConfiguration conf) {
- start(conf, null, null);
+ public static void startYARNWithConfig(YarnConfiguration conf, boolean withDFS) {
+ start(conf, null, null, withDFS);
}
- private static void start(YarnConfiguration conf, String principal, String keytab) {
+ private static void start(YarnConfiguration conf, String principal, String keytab, boolean withDFS) {
Review comment:
Could you instead use an external configuration code than such boolean flag? Or at least use an overload method so that we don't modify all of these method calls which only pass `withDFS == false`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r376218170
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java
##########
@@ -171,6 +171,19 @@
" setting and use cluster default priority. Please refer to YARN's official documentation for specific" +
" settings required to enable priority scheduling for the targeted YARN version.");
+ /**
+ * Yarn session client uploads flink jar and user libs to file system (hdfs/s3) as local resource for yarn
+ * application context. The replication number changes the how many replica of each of these files in hdfs/s3.
+ * It is useful to accelerate this container bootstrap, when a Flink application needs more than one hundred
+ * of containers. If it is configured, Flink will use the default replication value in hadoop configuration.
Review comment:
"If it is configured" -> "If it is not configured"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r376200422
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -684,12 +710,28 @@ private static void start(YarnConfiguration conf, String principal, String keyta
}
+ private static void setMiniDFSCluster(String principal, String keytab, File targetTestClassesFolder) throws Exception {
Review comment:
Why do you leave the unused two arguments `principal` and `keytab`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373784636
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -199,7 +199,8 @@ static void testCopyFromLocalRecursive(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ 1);
Review comment:
Why should we explicitly set `1` instead of rely on default value. I'm afraid it breaks some current user settings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147664218 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
* d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147664218 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147761655 TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922 TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
* d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885)
* 570a3b05bc858f96a98da192d03e147585ce44cc Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147761655) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375632453
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -199,7 +199,8 @@ static void testCopyFromLocalRecursive(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ 1);
Review comment:
To clarify breaking change could be ok but it should be explicit. IIUC this patch adds a new feature without any change to existing logics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373809209
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java
##########
@@ -682,6 +683,9 @@ private ApplicationReport startAppMaster(
// add user code jars from the provided JobGraph
: jobGraph.getUserJars().stream().map(f -> f.toUri()).map(File::new).collect(Collectors.toSet());
+ int fileReplication = yarnConfiguration.getInt(DFSConfigKeys.DFS_REPLICATION_KEY, DFSConfigKeys.DFS_REPLICATION_DEFAULT);
+ fileReplication = flinkConfiguration.getInteger(YarnConfigOptions.FILE_REPLICATION, fileReplication);
Review comment:
I think we need a test with configuration setting to `-1` (we currently only have one test to set to `4`)
my gut feeling is that this won't work - Configuration.getInteger only applies default value when the config key is `null`, however the default value according to the documentation is `-1` not `null`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147664218 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
* d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885)
* 570a3b05bc858f96a98da192d03e147585ce44cc UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375658858
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
##########
@@ -64,32 +69,46 @@
@BeforeClass
public static void setup() {
YARN_CONFIGURATION.set(YarnTestBase.TEST_CLUSTER_NAME_KEY, "flink-yarn-tests-per-job");
- startYARNWithConfig(YARN_CONFIGURATION);
+ startYARNWithConfig(YARN_CONFIGURATION, true);
Review comment:
From the perspective of long term maintenance, I agree with your point. I created another test class exends from YARNITCase. It will not touch existing test cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r376195470
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNFileReplicationITCase.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.yarn;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.yarn.configuration.YarnConfigOptions;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Test cases for the deployment of Yarn Flink clusters with customized file replication numbers.
+ */
+public class YARNFileReplicationITCase extends YARNITCase {
+
+ @BeforeClass
+ public static void setup() {
+ YARN_CONFIGURATION.set(YarnTestBase.TEST_CLUSTER_NAME_KEY, "flink-yarn-tests-per-job");
+ startYARNWithConfig(YARN_CONFIGURATION, true);
+ }
+
+ @Test
+ public void testPerJobModeWithCustomizedFileReplication() throws Exception {
+ Configuration configuration = createDefaultConfiguration(YarnConfigOptions.UserJarInclusion.DISABLED);
+ configuration.setInteger(YarnConfigOptions.FILE_REPLICATION, 4);
+
+ runTest(() -> deployPerjob(
+ configuration,
+ getTestingJobGraph()));
+ }
+
+ @Test
+ public void testPerJobModeWithDefaultFileReplication() throws Exception {
+ Configuration configuration = createDefaultConfiguration(YarnConfigOptions.UserJarInclusion.DISABLED);
+
+ runTest(() -> deployPerjob(
+ configuration,
+ getTestingJobGraph()));
+ }
+
+ @Override
+ protected void extraVerification(Configuration configuration, ApplicationId applicationId) throws Exception {
+ final FileSystem fs = FileSystem.get(getYarnConfiguration());
+
+ String suffix = ".flink/" + applicationId.toString() + "/" + flinkUberjar.getName();
+
+ Path uberJarHDFSPath = new Path(fs.getHomeDirectory(), suffix);
+ FileStatus fsStatus = fs.getFileStatus(uberJarHDFSPath);
+
+ final int flinkFileReplication = configuration.getInteger(YarnConfigOptions.FILE_REPLICATION);
+ final int replication = YARN_CONFIGURATION.getInt(DFSConfigKeys.DFS_REPLICATION_KEY, DFSConfigKeys.DFS_REPLICATION_DEFAULT);
+
+ // If YarnConfigOptions.FILE_REPLICATION is not set. The replication number should equals to yarn configuration value.
+ int expectedReplication = flinkFileReplication > 0
+ ? flinkFileReplication : replication;
+ assertEquals(expectedReplication, fsStatus.getReplication());
+
+ Path appPath = uberJarHDFSPath.getParent();
+ FileStatus[] fileStatuses = fs.listStatus(appPath, new PathFilter() {
+ @Override
+ public boolean accept(Path path) {
+ return path.getName().endsWith("taskmanager-conf.yaml");
Review comment:
The `taskmanager-conf.yaml` has been remove from 1.10. All the updated flink configuration options for taskmanager will be loaded by dynamical properties. So i think we do not need to verify the replication.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147664218 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147761655 TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922 TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
* d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885)
* 570a3b05bc858f96a98da192d03e147585ce44cc Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147761655) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r381109343
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
##########
@@ -54,9 +54,9 @@
*/
public class YARNITCase extends YarnTestBase {
- private final Duration yarnAppTerminateTimeout = Duration.ofSeconds(10);
+ protected final Duration yarnAppTerminateTimeout = Duration.ofSeconds(10);
- private final int sleepIntervalInMS = 100;
+ protected final int sleepIntervalInMS = 100;
@Rule
public final TemporaryFolder temporaryFolder = new TemporaryFolder();
Review comment:
Good catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10980: [FLINK-12343][yarn] add
yarn file replication option
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10980: [FLINK-12343][yarn] add yarn file replication option
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-588812149
>> Do you have a test to update the DFS_REPLICATION_KEY in hdfs-site.xml? And then the replication here should be same. I am not sure we could get a updated value from yarnConfiguration. Since YarnConfiguration do not add the hdfs-site.xml as resource by default.
>>> Thanks for giving feedback. Resolved most of them. For the YarnConfiguration, yes you are right. By default, the hdfs-site.xml is not loaded. Thus, Currently Flink always uses default value. To honor the value frrom hdfs-site.xml, I think we need a sperate PR.
I have verified that it could work. Because when we call `FileSystem.get(yarnConfiguration)`, the `core-site.xml` and `hdfs-site.xml` will be added to resource automatically.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][yarn] add yarn file replication option
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][yarn] add yarn file replication option
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375659177
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java
##########
@@ -682,6 +683,10 @@ private ApplicationReport startAppMaster(
// add user code jars from the provided JobGraph
: jobGraph.getUserJars().stream().map(f -> f.toUri()).map(File::new).collect(Collectors.toSet());
+ int yarnFileReplication = yarnConfiguration.getInt(DFSConfigKeys.DFS_REPLICATION_KEY, DFSConfigKeys.DFS_REPLICATION_DEFAULT);
+ int fileReplication = flinkConfiguration.getInteger(YarnConfigOptions.FILE_REPLICATION);
Review comment:
I change the condition to fileReplication = fileReplication > 0 ? fileReplication : yarnFileReplication;
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373913383
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -199,7 +199,8 @@ static void testCopyFromLocalRecursive(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ 1);
Review comment:
Agree.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373913165
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -962,6 +1000,11 @@ public static void teardown() throws Exception {
yarnCluster = null;
}
+ if (miniDFSCluster != null) {
+ LOG.info("Stopping MiniDFS Cluster");
+ miniDFSCluster.shutdown();
Review comment:
Good catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][yarn] add yarn file replication option
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][yarn] add yarn file replication option
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r381110566
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
##########
@@ -684,12 +710,28 @@ private static void start(YarnConfiguration conf, String principal, String keyta
}
+ private static void setMiniDFSCluster(String principal, String keytab, File targetTestClassesFolder) throws Exception {
+ if (miniDFSCluster == null) {
+ Configuration hdfsConfiguration = new Configuration();
+ hdfsConfiguration.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, tmpHDFS.getRoot().getAbsolutePath());
+ miniDFSCluster = new MiniDFSCluster
+ .Builder(hdfsConfiguration)
+ .numDataNodes(2)
Review comment:
Bring up more nodes will make test case runs longer and 10 seconds timeout will be not long enough. File replication number can be multiple on even 1 data node. It doesn't have to be 1:1 mapping. I put 2 here to make sure the scenarios of multiple data nodes is covered.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-588812149
>> Do you have a test to update the DFS_REPLICATION_KEY in hdfs-site.xml? And then the replication here should be same. I am not sure we could get a updated value from yarnConfiguration. Since YarnConfiguration do not add the hdfs-site.xml as resource by default.
I have verified that it could work. Because when we call `FileSystem.get(yarnConfiguration)`, the `core-site.xml` and `hdfs-site.xml` will be added to resource automatically.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375631555
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
##########
@@ -64,32 +69,46 @@
@BeforeClass
public static void setup() {
YARN_CONFIGURATION.set(YarnTestBase.TEST_CLUSTER_NAME_KEY, "flink-yarn-tests-per-job");
- startYARNWithConfig(YARN_CONFIGURATION);
+ startYARNWithConfig(YARN_CONFIGURATION, true);
Review comment:
I'd suggest you just create a new file and write class about yarn file replication, without any modification on `YARNITCase`. It doesn't no harm we add a new file(test case) and if we keep modifying existing file for "patching" logics we highly possibly run into mess.
Besides, still you don't test
```
configuration.setInteger(YarnConfigOptions.FILE_REPLICATION, 4);
```
with value expect `4` especially edge case such as `< -1` `== -1`(possibly `<= -1` is ok) `== 0` `== 1`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r381838514
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -261,7 +263,8 @@ private static void testCopySingleFileFromLocal(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ DFSConfigKeys.DFS_REPLICATION_DEFAULT);
Review comment:
No, we could not create multiple file replication on single datanode. The file will not be accessible until enough datanode has been registered.
Since we do not need to access the file here, it makes sense to use `DFSConfigKeys.DFS_REPLICATION_DEFAULT`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
wangyang0918 edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-588833119
The changes LGTM now.
@HuangZhenQiu Could you help to test the following cases in a real Yarn cluster?
* Do not set `yarn.file-replication` in Flink configuration and do not update `dfs.replication` in hdfs-site.xml in client, then replication should be 3
* Do not set `yarn.file-replication` in Flink configuration and update `dfs.replication` to 5 in hdfs-site.xml in client, the replication should be 5
* Set `yarn.file-replication` to 7 in Flink configuration and update `dfs.replication` to 5 in hdfs-site.xml in client, then the replication should be 7
@TisonKun Let's wait for @HuangZhenQiu's final verification and merge. Thanks a lot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375632299
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -199,7 +199,8 @@ static void testCopyFromLocalRecursive(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ 1);
Review comment:
@walterddr what I'm concerned about is that existing logic should perfectly respect with default value, not a configured non-default value. Otherwise it is suspected we break existing manner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-589172829
Thanks for your contribution @HuangZhenQiu ! Merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zehuaiWANG commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
zehuaiWANG commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375637375
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -199,7 +199,8 @@ static void testCopyFromLocalRecursive(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ 1);
Review comment:
I don't think that setting it to 1 is a good decision. I agree with @TisonKun 's point of view, rely on default value will be better, and it can reduce the risk of disrupting the current user settings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r373913066
##########
File path: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNITCase.java
##########
@@ -122,6 +136,32 @@ private void deployPerjob(YarnConfigOptions.UserJarInclusion userJarInclusion, J
assertThat(jobResult, is(notNullValue()));
assertThat(jobResult.getSerializedThrowable().isPresent(), is(false));
+ if (configuration.getInteger(YarnConfigOptions.FILE_REPLICATION) != -1) {
Review comment:
Agree. I actually added two more test cases in YarnITCase. The change on this function is to minimize duplicated code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r375658548
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -199,7 +199,8 @@ static void testCopyFromLocalRecursive(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ 1);
Review comment:
Sounds good. I updated to the default hadoop DFSConfigKeys.DFS_REPLICATION_DEFAULT, which is the default value when the YarnConfigOptions.FILE_REPLICATION is not set.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147664218 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147761655 TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922 TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
Hash:f522e48ac9e3478ac88c6de8389d529a17c73377 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5297 TriggerType:PUSH TriggerID:f522e48ac9e3478ac88c6de8389d529a17c73377
Hash:f522e48ac9e3478ac88c6de8389d529a17c73377 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/149583795 TriggerType:PUSH TriggerID:f522e48ac9e3478ac88c6de8389d529a17c73377
Hash:4efb9fcbd694ec4b60dd8c8790a076bac9439438 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/149591443 TriggerType:PUSH TriggerID:4efb9fcbd694ec4b60dd8c8790a076bac9439438
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
* d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885)
* 570a3b05bc858f96a98da192d03e147585ce44cc Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147761655) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922)
* f522e48ac9e3478ac88c6de8389d529a17c73377 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149583795) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5297)
* 4efb9fcbd694ec4b60dd8c8790a076bac9439438 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/149591443)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147664218 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
* d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#discussion_r381869854
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/YarnFileStageTest.java
##########
@@ -261,7 +263,8 @@ private static void testCopySingleFileFromLocal(
remotePaths,
localResources,
localResourceDirectory,
- new StringBuilder());
+ new StringBuilder(),
+ DFSConfigKeys.DFS_REPLICATION_DEFAULT);
Review comment:
@wangyang0918 given this settings to 1, is this patch looks good to you?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-580609306
<!--
Meta data
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/146878202 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706 TriggerType:PUSH TriggerID:1bca661ca3f42c4c98234ac90fb5359516c0b6c9
Hash:d7944ff95108e04a636460069880368ff96daedc Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/146962127 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d7944ff95108e04a636460069880368ff96daedc Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740 TriggerType:PUSH TriggerID:d7944ff95108e04a636460069880368ff96daedc
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/147136051 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d69bdc04371d701fac1a103e5a47902246707a61 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759 TriggerType:PUSH TriggerID:d69bdc04371d701fac1a103e5a47902246707a61
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147664218 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:d846f33ba278b1d75a1eac672c7efebee1350e8f Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885 TriggerType:PUSH TriggerID:d846f33ba278b1d75a1eac672c7efebee1350e8f
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/147761655 TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
Hash:570a3b05bc858f96a98da192d03e147585ce44cc Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922 TriggerType:PUSH TriggerID:570a3b05bc858f96a98da192d03e147585ce44cc
Hash:f522e48ac9e3478ac88c6de8389d529a17c73377 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5297 TriggerType:PUSH TriggerID:f522e48ac9e3478ac88c6de8389d529a17c73377
Hash:f522e48ac9e3478ac88c6de8389d529a17c73377 Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/149583795 TriggerType:PUSH TriggerID:f522e48ac9e3478ac88c6de8389d529a17c73377
Hash:4efb9fcbd694ec4b60dd8c8790a076bac9439438 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/149591443 TriggerType:PUSH TriggerID:4efb9fcbd694ec4b60dd8c8790a076bac9439438
Hash:4efb9fcbd694ec4b60dd8c8790a076bac9439438 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5301 TriggerType:PUSH TriggerID:4efb9fcbd694ec4b60dd8c8790a076bac9439438
-->
## CI report:
* 1bca661ca3f42c4c98234ac90fb5359516c0b6c9 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/146878202) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4706)
* d7944ff95108e04a636460069880368ff96daedc Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/146962127) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4740)
* d69bdc04371d701fac1a103e5a47902246707a61 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/147136051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4759)
* d846f33ba278b1d75a1eac672c7efebee1350e8f Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147664218) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4885)
* 570a3b05bc858f96a98da192d03e147585ce44cc Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/147761655) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4922)
* f522e48ac9e3478ac88c6de8389d529a17c73377 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/149583795) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5297)
* 4efb9fcbd694ec4b60dd8c8790a076bac9439438 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/149591443) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5301)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] HuangZhenQiu commented on issue #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
HuangZhenQiu commented on issue #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980#issuecomment-588680293
@TisonKun
Would you please take one more round of look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun closed pull request #10980:
[FLINK-12343][flink-yarn] add yarn file replication configuration
Posted by GitBox <gi...@apache.org>.
TisonKun closed pull request #10980: [FLINK-12343][flink-yarn] add yarn file replication configuration
URL: https://github.com/apache/flink/pull/10980
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services