You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2022/03/30 14:28:30 UTC
[spark] branch master updated: [SPARK-38696][BUILD] Add `commons-collections` back for hadoop-3 profile
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6b29b28 [SPARK-38696][BUILD] Add `commons-collections` back for hadoop-3 profile
6b29b28 is described below
commit 6b29b28deffd11edd65b69e0f5c79ed51d483b66
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Wed Mar 30 07:26:21 2022 -0700
[SPARK-38696][BUILD] Add `commons-collections` back for hadoop-3 profile
### What changes were proposed in this pull request?
- SPARK-37968 replaced `commons-collections 3.x` with `commons-collections4`.
- SPARK-37600 upgrades to Apache Hadoop 3.3.2.
This PR adds it back for `hadoop-3` profile because Apache Hadoop 3.3.2 is still using it. Since SPARK-37968 didn't remove it from `hadoop-2` profile , this is a regression in `hadoop-3` profile only.
### Why are the changes needed?
[HADOOP-17139](https://issues.apache.org/jira/browse/HADOOP-17139) added `commons-collections` usage at 3.3.2. Without this patch, `ClassNotFound` error happens while using S3A filesystem.
https://github.com/apache/hadoop/blob/f91452b289aea1418f56d242c046b58d9f214a1d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CopyFromLocalOperation.java#L41
```
import org.apache.commons.collections.comparators.ReverseComparator;
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
This is a kind of dependency recovery. This should pass CIs.
Closes #36009 from dongjoon-hyun/SPARK-38696.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
core/pom.xml | 4 ++++
dev/deps/spark-deps-hadoop-3-hive-2.3 | 1 +
pom.xml | 10 ++++++++--
3 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/core/pom.xml b/core/pom.xml
index 24294a2..c1e494f 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -193,6 +193,10 @@
<artifactId>commons-io</artifactId>
</dependency>
<dependency>
+ <groupId>commons-collections</groupId>
+ <artifactId>commons-collections</artifactId>
+ </dependency>
+ <dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-collections4</artifactId>
</dependency>
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 27b0ec5..08fa631 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -38,6 +38,7 @@ chill-java/0.10.0//chill-java-0.10.0.jar
chill_2.12/0.10.0//chill_2.12-0.10.0.jar
commons-cli/1.5.0//commons-cli-1.5.0.jar
commons-codec/1.15//commons-codec-1.15.jar
+commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-collections4/4.4//commons-collections4-4.4.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
commons-compress/1.21//commons-compress-1.21.jar
diff --git a/pom.xml b/pom.xml
index 6c5b321..d45d6ca 100644
--- a/pom.xml
+++ b/pom.xml
@@ -160,7 +160,8 @@
<commons.httpcore.version>4.4.14</commons.httpcore.version>
<commons.math3.version>3.6.1</commons.math3.version>
<!-- managed up from 3.2.1 for SPARK-11652 -->
- <commons.collections.version>4.4</commons.collections.version>
+ <commons.collections.version>3.2.2</commons.collections.version>
+ <commons.collections4.version>4.4</commons.collections4.version>
<scala.version>2.12.15</scala.version>
<scala.binary.version>2.12</scala.binary.version>
<scalatest-maven-plugin.version>2.0.2</scalatest-maven-plugin.version>
@@ -620,9 +621,14 @@
<version>${commons.math3.version}</version>
</dependency>
<dependency>
+ <groupId>commons-collections</groupId>
+ <artifactId>commons-collections</artifactId>
+ <version>${commons.collections.version}</version>
+ </dependency>
+ <dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-collections4</artifactId>
- <version>${commons.collections.version}</version>
+ <version>${commons.collections4.version}</version>
</dependency>
<dependency>
<groupId>commons-beanutils</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org