You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2019/04/23 14:08:21 UTC
[spark] branch master updated: [SPARK-27481][BUILD] Upgrade
commons-logging to 1.1.3 for hadoop-3.2
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7cc15af [SPARK-27481][BUILD] Upgrade commons-logging to 1.1.3 for hadoop-3.2
7cc15af is described below
commit 7cc15af15632150c0abf7f9f3c43f7a54f9b6c98
Author: Yuming Wang <yu...@ebay.com>
AuthorDate: Tue Apr 23 07:08:01 2019 -0700
[SPARK-27481][BUILD] Upgrade commons-logging to 1.1.3 for hadoop-3.2
## What changes were proposed in this pull request?
hadoop-2.7 gets `commons-logging` version from `hive-metastore`:
```
[INFO] +- org.spark-project.hive:hive-metastore:jar:1.2.1.spark2:compile
[INFO] | +- com.jolbox:bonecp:jar:0.8.0.RELEASE:compile
[INFO] | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile
```
But Hive removes `commons-logging` since [HIVE-12237(Hive 2.0.0)](https://issues.apache.org/jira/browse/HIVE-12237), so hadoop-3.2 gets `commons-logging` from `commons-httpclient`:
```
[INFO] +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] | \- commons-logging:commons-logging:jar:1.0.4:compile
```
Thus. we may hint `LogConfigurationException`:
```
bin/spark-sql --conf spark.sql.hive.metastore.version=1.2.2 --conf spark.sql.hive.metastore.jars=file:///apache/hive-1.2.2-bin/lib/*
...
Caused by: org.apache.commons.logging.LogConfigurationException: Invalid class loader hierarchy. You have more than one version of 'org.apache.commons.logging.Log' visible, which is not allowed.
at org.apache.commons.logging.impl.LogFactoryImpl.getLogConstructor(LogFactoryImpl.java:385)
... 43 more
```
This pr upgrade `commons-logging` to 1.1.3 for hadoop-3.2 to fix this issue.
## How was this patch tested?
manual tests
Closes #24388 from wangyum/SPARK-27481.
Authored-by: Yuming Wang <yu...@ebay.com>
Signed-off-by: Sean Owen <se...@databricks.com>
---
pom.xml | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/pom.xml b/pom.xml
index 5879a76..5153957 100644
--- a/pom.xml
+++ b/pom.xml
@@ -475,6 +475,12 @@
<version>1.9.3</version>
</dependency>
<dependency>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ <!-- Hive uses commons-logging 1.1.3 from 0.13 to 1.2 -->
+ <version>1.1.3</version>
+ </dependency>
+ <dependency>
<groupId>org.apache.ivy</groupId>
<artifactId>ivy</artifactId>
<version>${ivy.version}</version>
@@ -1609,10 +1615,6 @@
</exclusion>
<exclusion>
<groupId>${hive.group}</groupId>
- <artifactId>hive-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>${hive.group}</groupId>
<artifactId>hive-metastore</artifactId>
</exclusion>
<exclusion>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org