You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by haohui <gi...@git.apache.org> on 2017/03/23 21:48:28 UTC
[GitHub] flink pull request #3604: [FLINK-5998] Un-fat Hadoop from Flink fat jar
GitHub user haohui opened a pull request:
https://github.com/apache/flink/pull/3604
[FLINK-5998] Un-fat Hadoop from Flink fat jar
This PR implements FLINK-5998.
It marks all Hadoop dependency in the dist jar as `provided` so that users can plug in the jars from their own Hadoop distribution.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/haohui/flink FLINK-5998
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3604.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3604
----
commit a8f145cce141450edeb92c5ea9bed6d0441dc94f
Author: Haohui Mai <wh...@apache.org>
Date: 2017-03-23T21:46:45Z
[FLINK-5998] Un-fat Hadoop from Flink fat jar.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3604: [FLINK-5998] Un-fat Hadoop from Flink fat jar
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3604
Yes, the idea is that users in some environments can even delete `flink-dist-hadoop.jar` entirely, and just configure the classpath correctly to point to the Hadoop lib folder of their hadoop distribution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3604: [FLINK-5998] Un-fat Hadoop from Flink fat jar
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3604
Success!
I got it to work on my CDH 5.4.2 virtual machine with the following environment variables set
```
export HADOOP_CLASSPATH=/usr/lib/hadoop-yarn/*:/usr/lib/hadoop/*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop/lib/*:/etc/hadoop/conf:/usr/lib/hadoop-hdfs/*
```
and it was using CDH's Hadoop version:
```
2017-05-04 10:49:58,243 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Hadoop version: 2.6.0-cdh5.4.2
```
I'll merge this change to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3604: [FLINK-5998] Un-fat Hadoop from Flink fat jar
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3604
Mh, not sure.
What I would try is updating the assembly descriptor(s) in the flink-dist project, so that it creates two assemblies.
Afaik you need to define all dependencies as regular deps in flink-dist, and then include subsets of them into different assemblies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3604: [FLINK-5998] Un-fat Hadoop from Flink fat jar
Posted by haohui <gi...@git.apache.org>.
Github user haohui commented on the issue:
https://github.com/apache/flink/pull/3604
@rmetzger ping...would you mind taking a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3604: [FLINK-5998] Un-fat Hadoop from Flink fat jar
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3604
Sorry for the delay. I'm now checking out the change....
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3604: [FLINK-5998] Un-fat Hadoop from Flink fat jar
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/3604
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3604: [FLINK-5998] Un-fat Hadoop from Flink fat jar
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3604
I have tested this change locally, but it does not exactly what my intention was.
What I want is that the `lib/` folder contains a `flink-dist.jar` with all the Flink runtime dependencies, and a `flink-dist-hadoop.jar` with the packaged Hadoop dependencies.
This way, users have a good out of the box experience, but they can use their own Hadoop jars easily.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3604: [FLINK-5998] Un-fat Hadoop from Flink fat jar
Posted by haohui <gi...@git.apache.org>.
Github user haohui commented on the issue:
https://github.com/apache/flink/pull/3604
@rmetzger -- thanks for the clarification. Do you think that it is sufficient to mark the dependency of `flink-shaded-hadoop2` as provided and produces a dedicated jar for it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3604: [FLINK-5998] Un-fat Hadoop from Flink fat jar
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/3604
@rmetzger, how would users know which Hadoop dependencies to include in `flink-dist-hadoop.jar`? Would they be copying multiple component jars into `lib/`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3604: [FLINK-5998] Un-fat Hadoop from Flink fat jar
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/3604
@haohui I think this goes into the right direction, but the `flink-shaded-hadoop2-1.3.jar` should be in `/lib`, not in `opt`. That way, we preserve the original behavior (Hadoop dependencies are available by default), but make it easy to swap/remove the Hadoop dependency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3604: [FLINK-5998] Un-fat Hadoop from Flink fat jar
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/3604
@rmetzger thanks for the clarification. Sounds good!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---