You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by greghogan <gi...@git.apache.org> on 2016/10/27 21:11:56 UTC
[GitHub] flink pull request #2708: [FLINK-4946] [scripts] Load jar files from subdire...
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/2708
[FLINK-4946] [scripts] Load jar files from subdirectories of lib
The Flink classpath is a concatenation of jar files in lib/. This commit includes files from subdirectories of lib/ in the classpath.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink 4946_load_jar_files_from_subdirectories_of_lib
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2708.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2708
----
commit 19cb4d5bee2863aedf111c1e6e69475c4205a360
Author: Greg Hogan <co...@greghogan.com>
Date: 2016-10-27T19:14:36Z
[FLINK-4946] [scripts] Load jar files from subdirectories of lib
The Flink classpath is a concatenation of jar files in lib/. This commit
includes files from subdirectories of lib/ in the classpath.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2708: [FLINK-4946] [scripts] Load jar files from subdirectories...
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/2708
@mxm I pushed a new commit that is working in YARN with recursive directories. The issue looks to have been that YARN was copying files recursively but the Java classpath can only contain simple wildcards `*` but not recursive wildcards `**`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2708: [FLINK-4946] [scripts] Load jar files from subdirectories...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2708
Thanks @greghogan! The changes look good.
+1 to merge
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2708: [FLINK-4946] [scripts] Load jar files from subdire...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/2708#discussion_r86546214
--- Diff: flink-dist/src/main/flink-bin/yarn-bin/yarn-session.sh ---
@@ -35,14 +33,15 @@ JVM_ARGS="$JVM_ARGS -Xmx512m"
# Flink CLI client
constructCLIClientClassPath() {
- for jarfile in $FLINK_LIB_DIR/*.jar ; do
- if [[ $CC_CLASSPATH = "" ]]; then
- CC_CLASSPATH=$jarfile;
- else
- CC_CLASSPATH=$CC_CLASSPATH:$jarfile
- fi
- done
- echo $CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS
+ while read -d '' -r jarfile ; do
+ if [[ $FLINK_CLASSPATH = "" ]]; then
+ CC_CLASSPATH="$jarfile";
+ else
+ CC_CLASSPATH="$CC_CLASSPATH":"$jarfile"
+ fi
+ done < <(find "$FLINK_LIB_DIR" -name '*.jar' -print0)
+
+ echo $CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS
--- End diff --
Definitely preferable if `**` works \U0001f44d
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2708: [FLINK-4946] [scripts] Load jar files from subdire...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2708#discussion_r86545553
--- Diff: flink-dist/src/main/flink-bin/yarn-bin/yarn-session.sh ---
@@ -35,14 +33,15 @@ JVM_ARGS="$JVM_ARGS -Xmx512m"
# Flink CLI client
constructCLIClientClassPath() {
- for jarfile in $FLINK_LIB_DIR/*.jar ; do
- if [[ $CC_CLASSPATH = "" ]]; then
- CC_CLASSPATH=$jarfile;
- else
- CC_CLASSPATH=$CC_CLASSPATH:$jarfile
- fi
- done
- echo $CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS
+ while read -d '' -r jarfile ; do
+ if [[ $FLINK_CLASSPATH = "" ]]; then
+ CC_CLASSPATH="$jarfile";
+ else
+ CC_CLASSPATH="$CC_CLASSPATH":"$jarfile"
+ fi
+ done < <(find "$FLINK_LIB_DIR" -name '*.jar' -print0)
+
+ echo $CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS
--- End diff --
Recursing over the lib folder for uploading is not necessary because Yarn already recursively uploads it into the home directory using `FileUtil#copyFromLocalFile`. We just have to make sure to generate the classpath including all subdirectories. The easiest fix would be to use `lib/**` instead of `lib/*` which would be a one-line change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2708: [FLINK-4946] [scripts] Load jar files from subdirectories...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2708
@greghogan It would be helpful to look at the Yarn container files which include the container environment variables. This would help us to figure out why the recursive `**` is not working. Another solution would be to recurse into directories and add the files to the classpath one-by-one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2708: [FLINK-4946] [scripts] Load jar files from subdire...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/2708#discussion_r86542605
--- Diff: flink-dist/src/main/flink-bin/yarn-bin/yarn-session.sh ---
@@ -35,14 +33,15 @@ JVM_ARGS="$JVM_ARGS -Xmx512m"
# Flink CLI client
constructCLIClientClassPath() {
- for jarfile in $FLINK_LIB_DIR/*.jar ; do
- if [[ $CC_CLASSPATH = "" ]]; then
- CC_CLASSPATH=$jarfile;
- else
- CC_CLASSPATH=$CC_CLASSPATH:$jarfile
- fi
- done
- echo $CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS
+ while read -d '' -r jarfile ; do
+ if [[ $FLINK_CLASSPATH = "" ]]; then
+ CC_CLASSPATH="$jarfile";
+ else
+ CC_CLASSPATH="$CC_CLASSPATH":"$jarfile"
+ fi
+ done < <(find "$FLINK_LIB_DIR" -name '*.jar' -print0)
+
+ echo $CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS
--- End diff --
I think this is not enough for the YARN side of things. We need to update the logic in `AbstractYarnClusterDescriptor` (and maybe `FlinkYarnSessionCli` as well).
In `AbstractYarnClusterDescriptor` there is `addLibFolderToShipFiles` which adds the lib directory to the list of ship files. These are uploaded and the classpath of the containers has it as `*` (see line 620 of the descriptor).
We need to update this to upload the sub directories as well and add these to the class path. Maybe it's best to just iterate over the lib folder recursively and add each JAR file. @mxm What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2708: [FLINK-4946] [scripts] Load jar files from subdire...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2708
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2708: [FLINK-4946] [scripts] Load jar files from subdirectories...
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/2708
@uce @mxm after adding the recursive wildcard the nested jar files are copied to the application directory within HDFS. This is my first time testing Flink on YARN and I am getting a `ClassNotFoundException` from a class within the `flink-gelly` jar even when it is copied to `lib` (not nested).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---