You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by va...@apache.org on 2018/03/02 01:26:57 UTC
spark git commit: [SPARK-23551][BUILD] Exclude
`hadoop-mapreduce-client-core` dependency from `orc-mapreduce`
Repository: spark
Updated Branches:
refs/heads/master cdcccd7b4 -> 34811e0b9
[SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`
## What changes were proposed in this pull request?
This PR aims to prevent `orc-mapreduce` dependency from making IDEs and maven confused.
**BEFORE**
Please note that `2.6.4` at `Spark Project SQL`.
```
$ mvn dependency:tree -Phadoop-2.7 -Dincludes=org.apache.hadoop:hadoop-mapreduce-client-core
...
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Catalyst 2.4.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) spark-catalyst_2.11 ---
[INFO] org.apache.spark:spark-catalyst_2.11:jar:2.4.0-SNAPSHOT
[INFO] \- org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] \- org.apache.hadoop:hadoop-client:jar:2.7.3:compile
[INFO] \- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3:compile
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project SQL 2.4.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) spark-sql_2.11 ---
[INFO] org.apache.spark:spark-sql_2.11:jar:2.4.0-SNAPSHOT
[INFO] \- org.apache.orc:orc-mapreduce:jar:nohive:1.4.3:compile
[INFO] \- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.4:compile
```
**AFTER**
```
$ mvn dependency:tree -Phadoop-2.7 -Dincludes=org.apache.hadoop:hadoop-mapreduce-client-core
...
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Catalyst 2.4.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) spark-catalyst_2.11 ---
[INFO] org.apache.spark:spark-catalyst_2.11:jar:2.4.0-SNAPSHOT
[INFO] \- org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] \- org.apache.hadoop:hadoop-client:jar:2.7.3:compile
[INFO] \- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3:compile
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project SQL 2.4.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) spark-sql_2.11 ---
[INFO] org.apache.spark:spark-sql_2.11:jar:2.4.0-SNAPSHOT
[INFO] \- org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] \- org.apache.hadoop:hadoop-client:jar:2.7.3:compile
[INFO] \- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3:compile
```
## How was this patch tested?
1. Pass the Jenkins with `dev/test-dependencies.sh` with the existing dependencies.
2. Manually do the following and see the change.
```
mvn dependency:tree -Phadoop-2.7 -Dincludes=org.apache.hadoop:hadoop-mapreduce-client-core
```
Author: Dongjoon Hyun <do...@apache.org>
Closes #20704 from dongjoon-hyun/SPARK-23551.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/34811e0b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/34811e0b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/34811e0b
Branch: refs/heads/master
Commit: 34811e0b908449fd59bca476604612b1d200778d
Parents: cdcccd7
Author: Dongjoon Hyun <do...@apache.org>
Authored: Thu Mar 1 17:26:39 2018 -0800
Committer: Marcelo Vanzin <va...@cloudera.com>
Committed: Thu Mar 1 17:26:39 2018 -0800
----------------------------------------------------------------------
pom.xml | 4 ++++
1 file changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/34811e0b/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index b839616..0a711f2 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1754,6 +1754,10 @@
<artifactId>hadoop-common</artifactId>
</exclusion>
<exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-mapreduce-client-core</artifactId>
+ </exclusion>
+ <exclusion>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
</exclusion>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org