You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/07/05 18:19:00 UTC
[jira] [Commented] (FLINK-7042) Fix jar file discovery in YARN
tests
[ https://issues.apache.org/jira/browse/FLINK-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075180#comment-16075180 ]
ASF GitHub Bot commented on FLINK-7042:
---------------------------------------
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/4264
[FLINK-7042] [yarn] Fix jar file discovery flink-yarn-tests
Add dependencies for batch and streaming WordCount programs and copies the jar files into a new target/programs directory. The integration tests now directly references the program jar files rather than the prior brittle search.
This removes the flink-yarn-tests build-time dependency on the examples modules (there remains a build-time dependency on flink-dist).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink 7042_fix_jar_file_discovery_in_yarn_tests
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4264.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4264
----
commit 18e908f90fdeb4f2883536d86c5e890a61d6c39d
Author: Greg Hogan <co...@greghogan.com>
Date: 2017-07-05T16:00:30Z
[FLINK-7042] [yarn] Fix jar file discovery flink-yarn-tests
Add dependencies for batch and streaming WordCount programs and copies
the jar files into a new target/programs directory. The integration
tests now directly references the program jar files rather than the
prior brittle search.
This removes the flink-yarn-tests build-time dependency on the examples
modules (there remains a build-time dependency on flink-dist).
----
> Fix jar file discovery in YARN tests
> ------------------------------------
>
> Key: FLINK-7042
> URL: https://issues.apache.org/jira/browse/FLINK-7042
> Project: Flink
> Issue Type: Bug
> Components: YARN
> Reporter: Greg Hogan
> Assignee: Greg Hogan
> Priority: Critical
>
> Running a local {{mvn clean verify}} the following error in {{org.apache.flink.yarn.YARNSessionCapacitySchedulerITCase#perJobYarnClusterWithParallelism}} is caused by the discovery of a spurious file created by an earlier YARN test.
> {code}
> 15:45:16,627 INFO org.apache.flink.yarn.YarnTestBase - Running with args [run, -p, 2, -m, yarn-cluster, -yj, /home/ec2-user/flink-upstream/flink-yarn-tests/../flink-dist/target/flink-1.4-SNAPSHOT-bin/flink-1.4-SNAPSHOT/lib/flink-dist_2.10-1.4-SNAPSHOT.jar, -yt, /home/ec2-user/flink-upstream/flink-yarn-tests/../flink-dist/target/flink-1.4-SNAPSHOT-bin/flink-1.4-SNAPSHOT/lib, -yn, 1, -yjm, 768, -ytm, 1024, /home/ec2-user/flink-upstream/flink-yarn-tests/../flink-yarn-tests/target/flink-yarn-tests-capacityscheduler/flink-yarn-tests-capacityscheduler-localDir-nm-1_0/usercache/ec2-user/appcache/application_1498751075681_0001/filecache/13/.tmp_flink-examples-batch_2.10-1.4-SNAPSHOT-WordCount.jar.crc]
> 15:45:16,628 INFO org.apache.flink.client.CliFrontend - Using configuration directory /home/ec2-user/flink-upstream/flink-yarn-tests/../flink-dist/target/flink-1.4-SNAPSHOT-bin/flink-1.4-SNAPSHOT/conf
> 15:45:16,628 INFO org.apache.flink.client.CliFrontend - Trying to load configuration file
> 15:45:16,628 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost
> 15:45:16,629 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
> 15:45:16,629 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 1024
> 15:45:16,629 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 1024
> 15:45:16,629 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
> 15:45:16,629 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, false
> 15:45:16,629 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
> 15:45:16,629 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.web.port, 8081
> 15:45:16,629 INFO org.apache.flink.client.CliFrontend - Running 'run' command.
> 15:45:16,629 INFO org.apache.flink.client.CliFrontend - Building program from JAR file
> 15:45:16,630 ERROR org.apache.flink.client.CliFrontend - Error while running the command.
> org.apache.flink.client.program.ProgramInvocationException: Error while opening jar file '/home/ec2-user/flink-upstream/flink-yarn-tests/../flink-yarn-tests/target/flink-yarn-tests-capacityscheduler/flink-yarn-tests-capacityscheduler-localDir-nm-1_0/usercache/ec2-user/appcache/application_1498751075681_0001/filecache/13/.tmp_flink-examples-batch_2.10-1.4-SNAPSHOT-WordCount.jar.crc'. error in opening zip file
> at org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(PackagedProgram.java:562)
> at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:188)
> at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:126)
> at org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:900)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:229)
> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1083)
> at org.apache.flink.yarn.YarnTestBase$Runner.run(YarnTestBase.java:657)
> Caused by: java.util.zip.ZipException: error in opening zip file
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.<init>(ZipFile.java:219)
> at java.util.zip.ZipFile.<init>(ZipFile.java:149)
> at java.util.jar.JarFile.<init>(JarFile.java:166)
> at java.util.jar.JarFile.<init>(JarFile.java:130)
> at org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(PackagedProgram.java:557)
> ... 6 more
> 15:45:16,632 INFO org.apache.flink.yarn.YarnTestBase - Runner stopped with exception
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)