You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by NicoK <gi...@git.apache.org> on 2017/11/06 10:08:20 UTC
[GitHub] flink pull request #4951: [FLINK-7989][yarn] do not deploy the dist.jar file...
GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/4951
[FLINK-7989][yarn] do not deploy the dist.jar file twice
## What is the purpose of the change
We always add the dist.jar ourselves, but it could also be inside a shipped folder such as the "lib/" folder and was then distributed multiple times.
This PR is based on #4939 which changed the artefact uploads a bit.
## Brief change log
- also exclude the `flink-dist*.jar` from any (recursive) upload of a shipped directory, e.g. `lib`
## Verifying this change
This change is already covered by existing tests, such as any test deploying a program on YARN (if it didn't include the `flink-dist*.jar` file, it would fail). Additionally, a manual deployment showed that `flink-dist*.jar` is only uploaded once.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (JavaDocs)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-7989
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4951.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4951
----
commit 6d40434b90d654df55dff3d267721743adc4f586
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2016-11-09T20:04:50Z
[FLINK-4228][yarn/s3] fix for yarn staging with s3a defaultFs
+ includes a new unit tests for recursive uploads to hfds:// targets
+ add a unit test for recursive file uploads to s3:// via s3a
commit df9f9b855185bafe89ac09f4accbd603807e3f56
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-11-03T14:29:07Z
[FLINK-4228][yarn/s3] turn the dependencies around
Instead of having flink-s3-fs-hadoop depend on flink-yarn_<scala_version>,
let flink-yarn depend on the s3 filesystem and implement the test there.
This is safer with regards to the scala-independent flink-s3-fs-hadoop project.
commit e5488ef719573da0fe86b8355a8b997dd3e3d80a
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-11-03T20:19:34Z
[FLINK-7989][yarn] do not add the dist.jar file twice
We always add the dist.jar ourselves, but it could also be inside a shipped
folder such as the "lib/" folder and was then distributed multiple times.
----
---
[GitHub] flink pull request #4951: [FLINK-7989][yarn] do not deploy the dist.jar file...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/4951#discussion_r149071640
--- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java ---
@@ -186,6 +186,8 @@ public void addShipFiles(List<File> shipFiles) {
for (File shipFile: shipFiles) {
// remove uberjar from ship list (by default everything in the lib/ folder is added to
// the list of files to ship, but we handle the uberjar separately.
+ // NOTE: this does not filter the uberjar if added recursively, e.g. by having the "lib"
+ // folder in the shipFiles
if (!(shipFile.getName().startsWith("flink-dist") && shipFile.getName().endsWith("jar"))) {
--- End diff --
oh - you're right.
Regarding the change of `uploadAndRegisterFiles`: I wouldn't upload the shipfiles in `addShipFiles` yet, so if we changed that, we'd have to traverse through the directories twice as we definitely need to do that for the uploads and if we wanted to filter recursively in `addShipFiles`, too, then there as well.
---
[GitHub] flink issue #4951: [FLINK-7989][yarn] do not deploy the dist.jar file twice
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/4951
FYI: travis errors are unrelated (ERROR 404 during download of http://mirror.netcologne.de/apache.org/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz)
---
[GitHub] flink issue #4951: [FLINK-7989][yarn] do not deploy the dist.jar file twice
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/4951
(rebased onto the newest #4939 which should fix the test failures shown by Travis on the previous build here)
---
[GitHub] flink issue #4951: [FLINK-7989][yarn] do not deploy the dist.jar file twice
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4951
Looks good, I think we can merge it once #4939 has been merged.
---
[GitHub] flink pull request #4951: [FLINK-7989][yarn] do not deploy the dist.jar file...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4951#discussion_r149069349
--- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java ---
@@ -186,6 +186,8 @@ public void addShipFiles(List<File> shipFiles) {
for (File shipFile: shipFiles) {
// remove uberjar from ship list (by default everything in the lib/ folder is added to
// the list of files to ship, but we handle the uberjar separately.
+ // NOTE: this does not filter the uberjar if added recursively, e.g. by having the "lib"
+ // folder in the shipFiles
if (!(shipFile.getName().startsWith("flink-dist") && shipFile.getName().endsWith("jar"))) {
--- End diff --
this filtering seems redundant now, the non-directory branch in `uploadAndRegisterFiles` should catch it.
We may want to pull the directory expansion into this method instead, and keep uploadAndRegisterFiles as a dumb upload method.
---
[GitHub] flink pull request #4951: [FLINK-7989][yarn] do not deploy the dist.jar file...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4951
---
[GitHub] flink issue #4951: [FLINK-7989][yarn] do not deploy the dist.jar file twice
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/4951
#4939 has been merged now - I rebased this PR now so we can go ahead
---