You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2015/01/28 09:29:32 UTC
spark git commit: [SPARK-4809] Rework Guava library shading.
Repository: spark
Updated Branches:
refs/heads/master d74373225 -> 37a5e272f
[SPARK-4809] Rework Guava library shading.
The current way of shading Guava is a little problematic. Code that
depends on "spark-core" does not see the transitive dependency, yet
classes in "spark-core" actually depend on Guava. So it's a little
tricky to run unit tests that use spark-core classes, since you need
a compatible version of Guava in your dependencies when running the
tests. This can become a little tricky, and is kind of a bad user
experience.
This change modifies the way Guava is shaded so that it's applied
uniformly across the Spark build. This means Guava is shaded inside
spark-core itself, so that the dependency issues above are solved.
Aside from that, all Spark sub-modules have their Guava references
relocated, so that they refer to the relocated classes now packaged
inside spark-core. Before, this was only done by the time the assembly
was built, so projects that did not end up inside the assembly (such
as streaming backends) could still reference the original location
of Guava classes.
The Guava classes are added to the "first" artifact Spark generates
(network-common), so that all downstream modules have the needed
classes available. Since "network-common" is a dependency of spark-core,
all Spark apps should get the relocated classes automatically.
Author: Marcelo Vanzin <va...@cloudera.com>
Closes #3658 from vanzin/SPARK-4809 and squashes the following commits:
3c93e42 [Marcelo Vanzin] Shade Guava in the network-common artifact.
5d69ec9 [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
b3104fc [Marcelo Vanzin] Add comment.
941848f [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
f78c48a [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
8053dd4 [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
107d7da [Marcelo Vanzin] Add fix for SPARK-5052 (PR #3874).
40b8723 [Marcelo Vanzin] Merge branch 'master' into SPARK-4809
4a4ed42 [Marcelo Vanzin] [SPARK-4809] Rework Guava library shading.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/37a5e272
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/37a5e272
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/37a5e272
Branch: refs/heads/master
Commit: 37a5e272f898e946c09c2e7de5d1bda6f27a8f39
Parents: d743732
Author: Marcelo Vanzin <va...@cloudera.com>
Authored: Wed Jan 28 00:29:29 2015 -0800
Committer: Patrick Wendell <pa...@databricks.com>
Committed: Wed Jan 28 00:29:29 2015 -0800
----------------------------------------------------------------------
assembly/pom.xml | 22 ---------
core/pom.xml | 48 --------------------
examples/pom.xml | 103 ++++++++++++++-----------------------------
network/common/pom.xml | 24 +++++++---
network/shuffle/pom.xml | 1 -
pom.xml | 22 ++++++++-
streaming/pom.xml | 8 ++++
7 files changed, 81 insertions(+), 147 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/37a5e272/assembly/pom.xml
----------------------------------------------------------------------
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 594fa0c..1bb5a67 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -43,12 +43,6 @@
</properties>
<dependencies>
- <!-- Promote Guava to compile scope in this module so it's included while shading. -->
- <dependency>
- <groupId>com.google.guava</groupId>
- <artifactId>guava</artifactId>
- <scope>compile</scope>
- </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
@@ -133,22 +127,6 @@
<goal>shade</goal>
</goals>
<configuration>
- <relocations>
- <relocation>
- <pattern>com.google</pattern>
- <shadedPattern>org.spark-project.guava</shadedPattern>
- <includes>
- <include>com.google.common.**</include>
- </includes>
- <excludes>
- <exclude>com/google/common/base/Absent*</exclude>
- <exclude>com/google/common/base/Function</exclude>
- <exclude>com/google/common/base/Optional*</exclude>
- <exclude>com/google/common/base/Present*</exclude>
- <exclude>com/google/common/base/Supplier</exclude>
- </excludes>
- </relocation>
- </relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
http://git-wip-us.apache.org/repos/asf/spark/blob/37a5e272/core/pom.xml
----------------------------------------------------------------------
diff --git a/core/pom.xml b/core/pom.xml
index 1984682..3c51b2d 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -106,16 +106,6 @@
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
</dependency>
- <!--
- Promote Guava to "compile" so that maven-shade-plugin picks it up (for packaging the Optional
- class exposed in the Java API). The plugin will then remove this dependency from the published
- pom, so that Guava does not pollute the client's compilation classpath.
- -->
- <dependency>
- <groupId>com.google.guava</groupId>
- <artifactId>guava</artifactId>
- <scope>compile</scope>
- </dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
@@ -352,44 +342,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-shade-plugin</artifactId>
- <executions>
- <execution>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- <configuration>
- <shadedArtifactAttached>false</shadedArtifactAttached>
- <artifactSet>
- <includes>
- <include>com.google.guava:guava</include>
- </includes>
- </artifactSet>
- <filters>
- <!-- See comment in the guava dependency declaration above. -->
- <filter>
- <artifact>com.google.guava:guava</artifact>
- <includes>
- <include>com/google/common/base/Absent*</include>
- <include>com/google/common/base/Function</include>
- <include>com/google/common/base/Optional*</include>
- <include>com/google/common/base/Present*</include>
- <include>com/google/common/base/Supplier</include>
- </includes>
- </filter>
- </filters>
- </configuration>
- </execution>
- </executions>
- </plugin>
- <!--
- Copy guava to the build directory. This is needed to make the SPARK_PREPEND_CLASSES
- option work in compute-classpath.sh, since it would put the non-shaded Spark classes in
- the runtime classpath.
- -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/spark/blob/37a5e272/examples/pom.xml
----------------------------------------------------------------------
diff --git a/examples/pom.xml b/examples/pom.xml
index 4b92147..8caad2b 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -35,12 +35,6 @@
<url>http://spark.apache.org/</url>
<dependencies>
- <!-- Promote Guava to compile scope in this module so it's included while shading. -->
- <dependency>
- <groupId>com.google.guava</groupId>
- <artifactId>guava</artifactId>
- <scope>compile</scope>
- </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
@@ -310,69 +304,40 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
- <executions>
- <execution>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- <configuration>
- <shadedArtifactAttached>false</shadedArtifactAttached>
- <outputFile>${project.build.directory}/scala-${scala.binary.version}/spark-examples-${project.version}-hadoop${hadoop.version}.jar</outputFile>
- <artifactSet>
- <includes>
- <include>*:*</include>
- </includes>
- </artifactSet>
- <filters>
- <filter>
- <artifact>com.google.guava:guava</artifact>
- <excludes>
- <!--
- Exclude all Guava classes so they're picked up from the main assembly. The
- dependency still needs to be compile-scoped so that the relocation below
- works.
- -->
- <exclude>**</exclude>
- </excludes>
- </filter>
- <filter>
- <artifact>*:*</artifact>
- <excludes>
- <exclude>META-INF/*.SF</exclude>
- <exclude>META-INF/*.DSA</exclude>
- <exclude>META-INF/*.RSA</exclude>
- </excludes>
- </filter>
- </filters>
- <relocations>
- <relocation>
- <pattern>com.google</pattern>
- <shadedPattern>org.spark-project.guava</shadedPattern>
- <includes>
- <include>com.google.common.**</include>
- </includes>
- <excludes>
- <exclude>com.google.common.base.Optional**</exclude>
- </excludes>
- </relocation>
- <relocation>
- <pattern>org.apache.commons.math3</pattern>
- <shadedPattern>org.spark-project.commons.math3</shadedPattern>
- </relocation>
- </relocations>
- <transformers>
- <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
- <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
- <resource>reference.conf</resource>
- </transformer>
- <transformer implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
- <resource>log4j.properties</resource>
- </transformer>
- </transformers>
- </configuration>
- </execution>
- </executions>
+ <configuration>
+ <shadedArtifactAttached>false</shadedArtifactAttached>
+ <outputFile>${project.build.directory}/scala-${scala.binary.version}/spark-examples-${project.version}-hadoop${hadoop.version}.jar</outputFile>
+ <artifactSet>
+ <includes>
+ <include>*:*</include>
+ </includes>
+ </artifactSet>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ <relocations combine.children="append">
+ <relocation>
+ <pattern>org.apache.commons.math3</pattern>
+ <shadedPattern>org.spark-project.commons.math3</shadedPattern>
+ </relocation>
+ </relocations>
+ <transformers>
+ <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
+ <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
+ <resource>reference.conf</resource>
+ </transformer>
+ <transformer implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
+ <resource>log4j.properties</resource>
+ </transformer>
+ </transformers>
+ </configuration>
</plugin>
</plugins>
</build>
http://git-wip-us.apache.org/repos/asf/spark/blob/37a5e272/network/common/pom.xml
----------------------------------------------------------------------
diff --git a/network/common/pom.xml b/network/common/pom.xml
index 245a96b..5a9bbe1 100644
--- a/network/common/pom.xml
+++ b/network/common/pom.xml
@@ -48,10 +48,15 @@
<artifactId>slf4j-api</artifactId>
<scope>provided</scope>
</dependency>
+ <!--
+ Promote Guava to "compile" so that maven-shade-plugin picks it up (for packaging the Optional
+ class exposed in the Java API). The plugin will then remove this dependency from the published
+ pom, so that Guava does not pollute the client's compilation classpath.
+ -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
- <scope>provided</scope>
+ <scope>compile</scope>
</dependency>
<!-- Test dependencies -->
@@ -88,11 +93,6 @@
<version>2.2</version>
<executions>
<execution>
- <goals>
- <goal>test-jar</goal>
- </goals>
- </execution>
- <execution>
<id>test-jar-on-test-compile</id>
<phase>test-compile</phase>
<goals>
@@ -101,6 +101,18 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <configuration>
+ <shadedArtifactAttached>false</shadedArtifactAttached>
+ <artifactSet>
+ <includes>
+ <include>com.google.guava:guava</include>
+ </includes>
+ </artifactSet>
+ </configuration>
+ </plugin>
</plugins>
</build>
</project>
http://git-wip-us.apache.org/repos/asf/spark/blob/37a5e272/network/shuffle/pom.xml
----------------------------------------------------------------------
diff --git a/network/shuffle/pom.xml b/network/shuffle/pom.xml
index 5bfa1ac..c2d0300 100644
--- a/network/shuffle/pom.xml
+++ b/network/shuffle/pom.xml
@@ -52,7 +52,6 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
- <scope>provided</scope>
</dependency>
<!-- Test dependencies -->
http://git-wip-us.apache.org/repos/asf/spark/blob/37a5e272/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 05cb379..4adfdf3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1264,7 +1264,10 @@
</execution>
</executions>
</plugin>
- <!-- The shade plug-in is used here to create effective pom's (see SPARK-3812). -->
+ <!--
+ The shade plug-in is used here to create effective pom's (see SPARK-3812), and also
+ remove references from the shaded libraries from artifacts published by Spark.
+ -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
@@ -1276,6 +1279,23 @@
<include>org.spark-project.spark:unused</include>
</includes>
</artifactSet>
+ <relocations>
+ <relocation>
+ <pattern>com.google.common</pattern>
+ <shadedPattern>org.spark-project.guava</shadedPattern>
+ <excludes>
+ <!--
+ These classes cannot be relocated, because the Java API exposes the
+ "Optional" type; the others are referenced by the Optional class.
+ -->
+ <exclude>com/google/common/base/Absent*</exclude>
+ <exclude>com/google/common/base/Function</exclude>
+ <exclude>com/google/common/base/Optional*</exclude>
+ <exclude>com/google/common/base/Present*</exclude>
+ <exclude>com/google/common/base/Supplier</exclude>
+ </excludes>
+ </relocation>
+ </relocations>
</configuration>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/spark/blob/37a5e272/streaming/pom.xml
----------------------------------------------------------------------
diff --git a/streaming/pom.xml b/streaming/pom.xml
index 22b0d71..98f5b41 100644
--- a/streaming/pom.xml
+++ b/streaming/pom.xml
@@ -95,6 +95,14 @@
</execution>
</executions>
</plugin>
+
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <configuration>
+ <shadeTestJar>true</shadeTestJar>
+ </configuration>
+ </plugin>
</plugins>
<resources>
<resource>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org