You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by rmetzger <gi...@git.apache.org> on 2015/11/17 22:39:57 UTC
[GitHub] flink pull request: [FLINK-3032] Fix jackson-core dependency confl...
GitHub user rmetzger opened a pull request:
https://github.com/apache/flink/pull/1372
[FLINK-3032] Fix jackson-core dependency conflict with Hadoop 2.7.1.
This commit is also changing how we build the "flink-shaded-hadoop" artifact.
In the past, we were including all Hadoop dependencies into a fat jar, without relocating all of them.
Maven was not able to see Hadoop's dependencies and classes ended up in the classpath multiple times.
With this change, only shaded Hadoop dependencies are included into the jar. The shade plugin will also
remove only the shaded dependencies from the pom file.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/flink flink3032
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1372.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1372
----
commit ad42f13f34f1c026436e050570e9572403dca375
Author: Robert Metzger <rm...@apache.org>
Date: 2015-11-17T13:16:27Z
[FLINK-3032] Fix jackson-core dependency conflict with Hadoop 2.7.1.
This commit is also changing how we build the "flink-shaded-hadoop" artifact.
In the past, we were including all Hadoop dependencies into a fat jar, without relocating all of them.
Maven was not able to see Hadoop's dependencies and classes ended up in the classpath multiple times.
With this change, only shaded Hadoop dependencies are included into the jar. The shade plugin will also
remove only the shaded dependencies from the pom file.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3032] Fix jackson-core dependency confl...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1372#discussion_r45125382
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java ---
@@ -320,6 +320,8 @@ public static void logEnvironmentInfo(Logger log, String componentName, String[]
}
}
+ log.info(" Classpath: " + System.getProperty("java.class.path"));
+
--- End diff --
Printing the classpath at startup is useful for debugging these issues. many other systems (kafka, hadoop) do it as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3032] Fix jackson-core dependency confl...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1372#discussion_r45125791
--- Diff: pom.xml ---
@@ -350,6 +351,24 @@ under the License.
<artifactId>httpclient</artifactId>
<version>4.2</version>
</dependency>
+
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ <version>${jackson.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ <version>${jackson.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-annotations</artifactId>
+ <version>${jackson.version}</version>
+ </dependency>
--- End diff --
Manage all fasterxml jackson dependencies to the same version. Hadoop (through some Azure dependency) and flink-runtime (through the metrics library and the web frontend) depend on jackson.
The managed version seems to be supported by all users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3032] Fix jackson-core dependency confl...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1372#issuecomment-158125356
I'm manually closing the PR since its merged on master. I'll push it to 0.10 once travis gives me green light
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3032] Fix jackson-core dependency confl...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1372#discussion_r45125511
--- Diff: flink-shaded-hadoop/pom.xml ---
@@ -72,6 +72,7 @@ under the License.
<shadedArtifactAttached>false</shadedArtifactAttached>
<createDependencyReducedPom>true</createDependencyReducedPom>
<dependencyReducedPomLocation>${project.basedir}/target/dependency-reduced-pom.xml</dependencyReducedPomLocation>
+ <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
--- End diff --
WIth this change, transitive dependencies are promoted to look like immediate dependencies.
Only with this flag enabled, I can properly include dependencies such as netty.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3032] Fix jackson-core dependency confl...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger closed the pull request at:
https://github.com/apache/flink/pull/1372
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3032] Fix jackson-core dependency confl...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1372#discussion_r45125614
--- Diff: flink-shaded-hadoop/pom.xml ---
@@ -101,6 +102,17 @@ under the License.
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer"/>
</transformers>
+ <artifactSet>
+ <includes>
+ <include>com.google.guava:*</include>
+ <include>com.google.protobuf:*</include>
+ <include>com.google.code.findbugs:*</include>
+ <include>asm:asm</include>
+ <include>io.netty:netty:*</include>
+ <include>org.apache.curator:*</include>
+ <include>org.apache.hadoop:*</include>
+ </includes>
+ </artifactSet>
--- End diff --
These are the artifacts the shading is taking into consideration. Only those dependencies are removed from the pom file and included into the final jar. They are all relocated (except for hadoop of course).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3032] Fix jackson-core dependency confl...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1372#issuecomment-157707910
I tested the change against:
- HDP 2.3.2 (Hadoop 2.7.1)
- Cloudera 5.3.0.0 (Hadoop 2.5.0) with Kerberos
- Cloudera 5.4.2.0 (Hadoop 2.6.0)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3032] Fix jackson-core dependency confl...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1372#issuecomment-157713077
Thanks for digging into this and fixing this.
+1 to merge this
One thing that would be a good sanity check is to see how many overlapping classes the final fat jar shading finds, and make sure there are no others than the Guava/ASM ones that we shade in a fine granular way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---