git commit: SPARK-1802. Audit dependency graph when Spark is built with -Phive

[pw...@apache.org]

Repository: spark
Updated Branches:
  refs/heads/master 7120a2979 -> 8586bf564


SPARK-1802. Audit dependency graph when Spark is built with -Phive

This initial commit resolves the conflicts in the Hive profiles as noted in https://issues.apache.org/jira/browse/SPARK-1802 .

Most of the fix was to note that Hive drags in Avro, and so if the hive module depends on Spark's version of the `avro-*` dependencies, it will pull in our exclusions as needed too. But I found we need to copy some exclusions between the two Avro dependencies to get this right. And then had to squash some commons-logging intrusions.

This turned up another annoying find, that `hive-exec` is basically an "assembly" artifact that _also_ packages all of its transitive dependencies. This means the final assembly shows lots of collisions between itself and its dependencies, and even other project dependencies. I have a TODO to examine whether that is going to be a deal-breaker or not.

In the meantime I'm going to tack on a second commit to this PR that will also fix some similar, last collisions in the YARN profile.

Author: Sean Owen <so...@cloudera.com>

Closes #744 from srowen/SPARK-1802 and squashes the following commits:

a856604 [Sean Owen] Resolve JAR version conflicts specific to Hive profile


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8586bf56
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8586bf56
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8586bf56

Branch: refs/heads/master
Commit: 8586bf564fe010dfc19ef26874472a6f85e355fb
Parents: 7120a29
Author: Sean Owen <so...@cloudera.com>
Authored: Mon May 12 14:17:25 2014 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Mon May 12 14:17:25 2014 -0700

----------------------------------------------------------------------
 pom.xml          | 16 ++++++++++++++++
 sql/hive/pom.xml | 21 +++++++++++++++++++++
 2 files changed, 37 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/8586bf56/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 5542a32..3554efa 100644
--- a/pom.xml
+++ b/pom.xml
@@ -535,6 +535,22 @@
             <groupId>io.netty</groupId>
             <artifactId>netty</artifactId>
           </exclusion>
+          <exclusion>
+            <groupId>org.mortbay.jetty</groupId>
+            <artifactId>jetty</artifactId>
+          </exclusion>
+          <exclusion>
+            <groupId>org.mortbay.jetty</groupId>
+            <artifactId>jetty-util</artifactId>
+          </exclusion>
+          <exclusion>
+            <groupId>org.mortbay.jetty</groupId>
+            <artifactId>servlet-api</artifactId>
+          </exclusion>
+          <exclusion>
+            <groupId>org.apache.velocity</groupId>
+            <artifactId>velocity</artifactId>
+          </exclusion>
         </exclusions>
       </dependency>
       <!-- See SPARK-1556 for info on this dependency: -->

http://git-wip-us.apache.org/repos/asf/spark/blob/8586bf56/sql/hive/pom.xml
----------------------------------------------------------------------
diff --git a/sql/hive/pom.xml b/sql/hive/pom.xml
index 889d249..8b32451 100644
--- a/sql/hive/pom.xml
+++ b/sql/hive/pom.xml
@@ -52,6 +52,12 @@
       <groupId>org.apache.hive</groupId>
       <artifactId>hive-exec</artifactId>
       <version>${hive.version}</version>
+      <exclusions>
+        <exclusion>
+          <groupId>commons-logging</groupId>
+          <artifactId>commons-logging</artifactId>
+        </exclusion>
+      </exclusions>
     </dependency>
     <dependency>
       <groupId>org.codehaus.jackson</groupId>
@@ -61,6 +67,21 @@
       <groupId>org.apache.hive</groupId>
       <artifactId>hive-serde</artifactId>
       <version>${hive.version}</version>
+      <exclusions>
+        <exclusion>
+          <groupId>commons-logging</groupId>
+          <artifactId>commons-logging</artifactId>
+        </exclusion>
+        <exclusion>
+          <groupId>commons-logging</groupId>
+          <artifactId>commons-logging-api</artifactId>
+        </exclusion>
+      </exclusions>
+    </dependency>
+    <!-- hive-serde already depends on avro, but this brings in customized config of avro deps from parent -->
+    <dependency>
+      <groupId>org.apache.avro</groupId>
+      <artifactId>avro</artifactId>
     </dependency>
     <dependency>
       <groupId>org.scalatest</groupId>