You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/09/12 23:54:50 UTC
git commit: [SPARK-3217] Add Guava to classpath when
SPARK_PREPEND_CLASSES is set.
Repository: spark
Updated Branches:
refs/heads/master 1d767967e -> af2583826
[SPARK-3217] Add Guava to classpath when SPARK_PREPEND_CLASSES is set.
When that option is used, the compiled classes from the build directory
are prepended to the classpath. Now that we avoid packaging Guava, that
means we have classes referencing the original Guava location in the app's
classpath, so errors happen.
For that case, add Guava manually to the classpath.
Note: if Spark is compiled with "-Phadoop-provided", it's tricky to
make things work with SPARK_PREPEND_CLASSES, because you need to add
the Hadoop classpath using SPARK_CLASSPATH and that means the older
Hadoop Guava overrides the newer one Spark needs. So someone using
SPARK_PREPEND_CLASSES needs to remember to not use that profile.
Author: Marcelo Vanzin <va...@cloudera.com>
Closes #2141 from vanzin/SPARK-3217 and squashes the following commits:
b967324 [Marcelo Vanzin] [SPARK-3217] Add Guava to classpath when SPARK_PREPEND_CLASSES is set.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/af258382
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/af258382
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/af258382
Branch: refs/heads/master
Commit: af2583826c15d2a4e2732017ea20feeff0fb79f6
Parents: 1d76796
Author: Marcelo Vanzin <va...@cloudera.com>
Authored: Fri Sep 12 14:54:42 2014 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Fri Sep 12 14:54:42 2014 -0700
----------------------------------------------------------------------
bin/compute-classpath.sh | 1 +
core/pom.xml | 27 +++++++++++++++++++++++++++
2 files changed, 28 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/af258382/bin/compute-classpath.sh
----------------------------------------------------------------------
diff --git a/bin/compute-classpath.sh b/bin/compute-classpath.sh
index 15c6779..0f63e36 100755
--- a/bin/compute-classpath.sh
+++ b/bin/compute-classpath.sh
@@ -43,6 +43,7 @@ if [ -n "$SPARK_PREPEND_CLASSES" ]; then
echo "NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark"\
"classes ahead of assembly." >&2
CLASSPATH="$CLASSPATH:$FWDIR/core/target/scala-$SCALA_VERSION/classes"
+ CLASSPATH="$CLASSPATH:$FWDIR/core/target/jars/*"
CLASSPATH="$CLASSPATH:$FWDIR/repl/target/scala-$SCALA_VERSION/classes"
CLASSPATH="$CLASSPATH:$FWDIR/mllib/target/scala-$SCALA_VERSION/classes"
CLASSPATH="$CLASSPATH:$FWDIR/bagel/target/scala-$SCALA_VERSION/classes"
http://git-wip-us.apache.org/repos/asf/spark/blob/af258382/core/pom.xml
----------------------------------------------------------------------
diff --git a/core/pom.xml b/core/pom.xml
index b2b788a..2a81f6d 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -351,6 +351,33 @@
</execution>
</executions>
</plugin>
+ <!--
+ Copy guava to the build directory. This is needed to make the SPARK_PREPEND_CLASSES
+ option work in compute-classpath.sh, since it would put the non-shaded Spark classes in
+ the runtime classpath.
+ -->
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>copy-dependencies</id>
+ <phase>package</phase>
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>${project.build.directory}</outputDirectory>
+ <overWriteReleases>false</overWriteReleases>
+ <overWriteSnapshots>false</overWriteSnapshots>
+ <overWriteIfNewer>true</overWriteIfNewer>
+ <useSubDirectoryPerType>true</useSubDirectoryPerType>
+ <includeArtifactIds>guava</includeArtifactIds>
+ <silent>true</silent>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
</plugins>
<resources>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org