You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/08/21 01:23:22 UTC
git commit: [SPARK-2848] Shade Guava in uber-jars.
Repository: spark
Updated Branches:
refs/heads/master d9e94146a -> c9f743957
[SPARK-2848] Shade Guava in uber-jars.
For further discussion, please check the JIRA entry.
This change moves Guava classes to a different package so that they don't conflict with the user-provided Guava (or the Hadoop-provided one). Since one class (Optional) was exposed through Spark's public API, that class was forked from Guava at the current dependency version (14.0.1) so that it can be kept going forward (until the API is cleaned).
Note this change has a few implications:
- *all* classes in the final jars will reference the relocated classes. If Hadoop classes are included (i.e. "-Phadoop-provided" is not activated), those will also reference the Guava 14 classes (instead of the Guava 11 classes from the Hadoop classpath).
- if the Guava version in Spark is ever changed, the new Guava will still reference the forked Optional class; this may or may not be a problem, but in the long term it's better to think about removing Optional from the public API.
For the end user, there are two visible implications:
- Guava is not provided as a transitive dependency anymore (since it's "provided" in Spark)
- At runtime, unless they provide their own, they'll either have no Guava or Hadoop's version of Guava (11), depending on how they set up their classpath.
Note that this patch does not change the sbt deliverables; those will still contain guava in its original package, and provide guava as a compile-time dependency. This assumes that maven is the canonical build, and sbt-built artifacts are not (officially) published.
Author: Marcelo Vanzin <va...@cloudera.com>
Closes #1813 from vanzin/SPARK-2848 and squashes the following commits:
9bdffb0 [Marcelo Vanzin] Undo sbt build changes.
819b445 [Marcelo Vanzin] Review feedback.
05e0a3d [Marcelo Vanzin] Merge branch 'master' into SPARK-2848
fef4370 [Marcelo Vanzin] Unfork Optional.java.
d3ea8e1 [Marcelo Vanzin] Exclude asm classes from final jar.
637189b [Marcelo Vanzin] Add hacky filter to prefer Spark's copy of Optional.
2fec990 [Marcelo Vanzin] Shade Guava in the sbt build.
616998e [Marcelo Vanzin] Shade Guava in the maven build, fork Guava's Optional.java.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c9f74395
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c9f74395
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c9f74395
Branch: refs/heads/master
Commit: c9f743957fa963bc1dbed7a44a346ffce1a45cf2
Parents: d9e9414
Author: Marcelo Vanzin <va...@cloudera.com>
Authored: Wed Aug 20 16:23:10 2014 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Wed Aug 20 16:23:10 2014 -0700
----------------------------------------------------------------------
assembly/pom.xml | 18 ++++++++++++++++++
core/pom.xml | 35 +++++++++++++++++++++++++++++++++++
examples/pom.xml | 26 +++++++++++++++++++++++++-
pom.xml | 16 ++++++++++++++++
project/SparkBuild.scala | 4 ++--
project/plugins.sbt | 4 ++++
6 files changed, 100 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/c9f74395/assembly/pom.xml
----------------------------------------------------------------------
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 9fbb037..de7b752 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -43,6 +43,12 @@
</properties>
<dependencies>
+ <!-- Promote Guava to compile scope in this module so it's included while shading. -->
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <scope>compile</scope>
+ </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
@@ -113,6 +119,18 @@
<goal>shade</goal>
</goals>
<configuration>
+ <relocations>
+ <relocation>
+ <pattern>com.google</pattern>
+ <shadedPattern>org.spark-project.guava</shadedPattern>
+ <includes>
+ <include>com.google.common.**</include>
+ </includes>
+ <excludes>
+ <exclude>com.google.common.base.Optional**</exclude>
+ </excludes>
+ </relocation>
+ </relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
http://git-wip-us.apache.org/repos/asf/spark/blob/c9f74395/core/pom.xml
----------------------------------------------------------------------
diff --git a/core/pom.xml b/core/pom.xml
index 6d8be37..83c708d 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -68,9 +68,15 @@
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
</dependency>
+ <!--
+ Promote Guava to "compile" so that maven-shade-plugin picks it up (for packaging the Optional
+ class exposed in the Java API). The plugin will then remove this dependency from the published
+ pom, so that Guava does not pollute the client's compilation classpath.
+ -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
+ <scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
@@ -322,6 +328,35 @@
</arguments>
</configuration>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <shadedArtifactAttached>false</shadedArtifactAttached>
+ <artifactSet>
+ <includes>
+ <include>com.google.guava:guava</include>
+ </includes>
+ </artifactSet>
+ <filters>
+ <!-- See comment in the guava dependency declaration above. -->
+ <filter>
+ <artifact>com.google.guava:guava</artifact>
+ <includes>
+ <include>com/google/common/base/Optional*</include>
+ </includes>
+ </filter>
+ </filters>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
</plugins>
<resources>
http://git-wip-us.apache.org/repos/asf/spark/blob/c9f74395/examples/pom.xml
----------------------------------------------------------------------
diff --git a/examples/pom.xml b/examples/pom.xml
index 8c4c128..9b12cb0 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -46,8 +46,14 @@
</dependencies>
</profile>
</profiles>
-
+
<dependencies>
+ <!-- Promote Guava to compile scope in this module so it's included while shading. -->
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <scope>compile</scope>
+ </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
@@ -210,6 +216,12 @@
</artifactSet>
<filters>
<filter>
+ <artifact>com.google.guava:guava</artifact>
+ <excludes>
+ <exclude>com/google/common/base/Optional*</exclude>
+ </excludes>
+ </filter>
+ <filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
@@ -226,6 +238,18 @@
<goal>shade</goal>
</goals>
<configuration>
+ <relocations>
+ <relocation>
+ <pattern>com.google</pattern>
+ <shadedPattern>org.spark-project.guava</shadedPattern>
+ <includes>
+ <include>com.google.common.**</include>
+ </includes>
+ <excludes>
+ <exclude>com.google.common.base.Optional**</exclude>
+ </excludes>
+ </relocation>
+ </relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
http://git-wip-us.apache.org/repos/asf/spark/blob/c9f74395/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 7ed07ad..9cbf3ea 100644
--- a/pom.xml
+++ b/pom.xml
@@ -260,6 +260,7 @@
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>14.0.1</version>
+ <scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
@@ -1017,6 +1018,21 @@
<profiles>
+ <!--
+ This profile is enabled automatically by the sbt built. It changes the scope for the guava
+ dependency, since we don't shade it in the artifacts generated by the sbt build.
+ -->
+ <profile>
+ <id>sbt</id>
+ <dependencies>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+
<!-- Ganglia integration is not included by default due to LGPL-licensed code -->
<profile>
<id>spark-ganglia-lgpl</id>
http://git-wip-us.apache.org/repos/asf/spark/blob/c9f74395/project/SparkBuild.scala
----------------------------------------------------------------------
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 49d52ae..4c696d3 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -61,7 +61,7 @@ object SparkBuild extends PomBuild {
def backwardCompatibility = {
import scala.collection.mutable
var isAlphaYarn = false
- var profiles: mutable.Seq[String] = mutable.Seq.empty
+ var profiles: mutable.Seq[String] = mutable.Seq("sbt")
if (Properties.envOrNone("SPARK_GANGLIA_LGPL").isDefined) {
println("NOTE: SPARK_GANGLIA_LGPL is deprecated, please use -Pspark-ganglia-lgpl flag.")
profiles ++= Seq("spark-ganglia-lgpl")
@@ -116,7 +116,7 @@ object SparkBuild extends PomBuild {
retrieveManaged := true,
retrievePattern := "[type]s/[artifact](-[revision])(-[classifier]).[ext]",
publishMavenStyle := true,
-
+
resolvers += Resolver.mavenLocal,
otherResolvers <<= SbtPomKeys.mvnLocalRepository(dotM2 => Seq(Resolver.file("dotM2", dotM2))),
publishLocalConfiguration in MavenCompile <<= (packagedArtifacts, deliverLocal, ivyLoggingLevel) map {
http://git-wip-us.apache.org/repos/asf/spark/blob/c9f74395/project/plugins.sbt
----------------------------------------------------------------------
diff --git a/project/plugins.sbt b/project/plugins.sbt
index 2a61f56..8096c61 100644
--- a/project/plugins.sbt
+++ b/project/plugins.sbt
@@ -26,3 +26,7 @@ addSbtPlugin("com.alpinenow" % "junit_xml_listener" % "0.5.1")
addSbtPlugin("com.eed3si9n" % "sbt-unidoc" % "0.3.1")
addSbtPlugin("com.cavorite" % "sbt-avro" % "0.3.2")
+
+libraryDependencies += "org.ow2.asm" % "asm" % "5.0.3"
+
+libraryDependencies += "org.ow2.asm" % "asm-commons" % "5.0.3"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org