You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@orc.apache.org by do...@apache.org on 2023/05/21 07:54:40 UTC
[orc] branch main updated: ORC-1430: Use Hadoop 3.3.5 shaded clients
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/orc.git
The following commit(s) were added to refs/heads/main by this push:
new b166bf4b7 ORC-1430: Use Hadoop 3.3.5 shaded clients
b166bf4b7 is described below
commit b166bf4b7dfe12e6ddc6bbe77f4e6611ce2240f8
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Sun May 21 00:54:30 2023 -0700
ORC-1430: Use Hadoop 3.3.5 shaded clients
### What changes were proposed in this pull request?
Currently, Apache ORC project uses three properties.
```
<hadoop.version>2.7.3</hadoop.version>
<min.hadoop.version>2.7.3</min.hadoop.version>
<tools.hadoop.version>2.7.3</tools.hadoop.version>
```
This aims the following for Apache ORC 2.0.0.
1. Use Hadoop 3.3.5 shaded clients.
2. Remove `min.hadoop.version` and `tools.hadoop.version` in favor of `hadoop.version`
3. Ban non-shaded clients from now.
```
<bannedDependencies>
<excludes>
<exclude>org.apache.hadoop:hadoop-common</exclude>
<exclude>org.apache.hadoop:hadoop-hdfs-client</exclude>
<exclude>org.apache.hadoop:hadoop-mapreduce-client-core</exclude>
<exclude>org.apache.hadoop:hadoop-mapreduce-client-jobclient</exclude>
</excludes>
<searchTransitive>true</searchTransitive>
</bannedDependencies>
```
Note that all changes are `pom.xml` files. There is no code change.
### Why are the changes needed?
- Hadoop 3's shaded client removes lots of complexity from the downstream clients.
- It's stable because Apache Spark community has been using Hadoop 3's shaded client from Apache Spark 3.2.0 (October 13, 2021) via https://issues.apache.org/jira/browse/SPARK-33212.
### How was this patch tested?
Pass the CIs.
Also, I validated there is no side-effect at Apache Spark side. The following is the change set when Apache Spark upgrades from Apache ORC 1.8.3 (AS-IS) to Apache ORC 2.0.0-SNAPSHOT.
```
-aircompressor/0.21//aircompressor-0.21.jar
+aircompressor/0.24//aircompressor-0.24.jar
-orc-core/1.8.3/shaded-protobuf/orc-core-1.8.3-shaded-protobuf.jar
-orc-mapreduce/1.8.3/shaded-protobuf/orc-mapreduce-1.8.3-shaded-protobuf.jar
-orc-shims/1.8.3//orc-shims-1.8.3.jar
+orc-core/2.0.0-SNAPSHOT/shaded-protobuf/orc-core-2.0.0-SNAPSHOT-shaded-protobuf.jar
+orc-mapreduce/2.0.0-SNAPSHOT/shaded-protobuf/orc-mapreduce-2.0.0-SNAPSHOT-shaded-protobuf.jar
+orc-shims/2.0.0-SNAPSHOT//orc-shims-2.0.0-SNAPSHOT.jar
```
Closes #1509 from dongjoon-hyun/ORC-1430.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
java/bench/core/pom.xml | 12 ---
java/bench/hive/pom.xml | 24 +++---
java/bench/pom.xml | 69 ---------------
java/bench/spark/pom.xml | 12 ---
java/core/pom.xml | 30 +------
java/examples/pom.xml | 17 +---
java/mapreduce/pom.xml | 45 +---------
java/pom.xml | 214 ++++-------------------------------------------
java/shims/pom.xml | 26 +-----
java/tools/pom.xml | 32 -------
10 files changed, 34 insertions(+), 447 deletions(-)
diff --git a/java/bench/core/pom.xml b/java/bench/core/pom.xml
index 9fad2ef2a..7214a4715 100644
--- a/java/bench/core/pom.xml
+++ b/java/bench/core/pom.xml
@@ -55,18 +55,6 @@
<groupId>org.apache.commons</groupId>
<artifactId>commons-csv</artifactId>
</dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-core</artifactId>
- </dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
diff --git a/java/bench/hive/pom.xml b/java/bench/hive/pom.xml
index 2db4c9458..461eda61b 100644
--- a/java/bench/hive/pom.xml
+++ b/java/bench/hive/pom.xml
@@ -47,26 +47,26 @@
<groupId>org.apache.avro</groupId>
<artifactId>avro-mapred</artifactId>
</dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-core</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- </dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<classifier>core</classifier>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-registry</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-serde</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
diff --git a/java/bench/pom.xml b/java/bench/pom.xml
index 6f4eca8eb..538a820ae 100644
--- a/java/bench/pom.xml
+++ b/java/bench/pom.xml
@@ -34,7 +34,6 @@
<properties>
<avro.version>1.11.1</avro.version>
- <hadoop.version>3.3.5</hadoop.version>
<hive.version>3.1.3</hive.version>
<jmh.version>1.20</jmh.version>
<junit.version>5.9.3</junit.version>
@@ -109,74 +108,6 @@
<artifactId>commons-csv</artifactId>
<version>1.10.0</version>
</dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.version}</version>
- <exclusions>
- <exclusion>
- <groupId>com.sun.jersey</groupId>
- <artifactId>jersey-server</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.sun.jersey</groupId>
- <artifactId>jersey-core</artifactId>
- </exclusion>
- <exclusion>
- <groupId>commons-beanutils</groupId>
- <artifactId>commons-beanutils</artifactId>
- </exclusion>
- <exclusion>
- <groupId>commons-beanutils</groupId>
- <artifactId>commons-beanutils-core</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.servlet</groupId>
- <artifactId>servlet-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>jdk.tools</groupId>
- <artifactId>jdk.tools</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.mortbay.jetty</groupId>
- <artifactId>servlet-api</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- <version>${hadoop.version}</version>
- <scope>runtime</scope>
- <exclusions>
- <exclusion>
- <groupId>com.sun.jersey</groupId>
- <artifactId>jersey-core</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.sun.jersey</groupId>
- <artifactId>jersey-server</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.servlet</groupId>
- <artifactId>servlet-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.fusesource.leveldbjni</groupId>
- <artifactId>leveldbjni-all</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.mortbay.jetty</groupId>
- <artifactId>servlet-api</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-common</artifactId>
diff --git a/java/bench/spark/pom.xml b/java/bench/spark/pom.xml
index 7e8f6f4cc..b4e863818 100644
--- a/java/bench/spark/pom.xml
+++ b/java/bench/spark/pom.xml
@@ -52,18 +52,6 @@
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-yarn-common</artifactId>
- </dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
diff --git a/java/core/pom.xml b/java/core/pom.xml
index fc82c21d2..8bb4ea613 100644
--- a/java/core/pom.xml
+++ b/java/core/pom.xml
@@ -49,11 +49,12 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
+ <artifactId>hadoop-client-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
+ <artifactId>hadoop-client-runtime</artifactId>
+ <scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
@@ -152,16 +153,11 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<configuration>
- <ignoredUnusedDeclaredDependencies>
- <ignoredUnusedDeclaredDependency>org.apache.hadoop:hadoop-hdfs</ignoredUnusedDeclaredDependency>
- </ignoredUnusedDeclaredDependencies>
<ignoredUsedUndeclaredDependencies>
<ignoredUsedUndeclaredDependency>com.google.auto.service:auto-service-annotations</ignoredUsedUndeclaredDependency>
</ignoredUsedUndeclaredDependencies>
<ignoredDependencies>
<ignoredDependency>org.apache.hive:hive-storage-api</ignoredDependency>
- <ignoredDependency>org.apache.hadoop:hadoop-client-api</ignoredDependency>
- <ignoredDependency>org.apache.hadoop:hadoop-client-runtime</ignoredDependency>
<ignoredDependency>com.google.auto.service:auto-service</ignoredDependency>
</ignoredDependencies>
</configuration>
@@ -178,25 +174,5 @@
<directory>${build.dir}/core</directory>
</build>
</profile>
- <profile>
- <id>java17</id>
- <activation>
- <jdk>[17,)</jdk>
- </activation>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client-api</artifactId>
- <version>${hadoop.version}</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client-runtime</artifactId>
- <version>${hadoop.version}</version>
- <scope>test</scope>
- </dependency>
- </dependencies>
- </profile>
</profiles>
</project>
diff --git a/java/examples/pom.xml b/java/examples/pom.xml
index 60c714030..3d11dc036 100644
--- a/java/examples/pom.xml
+++ b/java/examples/pom.xml
@@ -51,12 +51,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <scope>compile</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
+ <artifactId>hadoop-client-api</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
@@ -111,16 +106,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
- <configuration>
- <ignoredUnusedDeclaredDependencies>
- <ignoredUnusedDeclaredDependency>com.google.guava:guava</ignoredUnusedDeclaredDependency>
- <ignoredUnusedDeclaredDependency>org.apache.hadoop:hadoop-hdfs</ignoredUnusedDeclaredDependency>
- <ignoredUnusedDeclaredDependency>org.apache.hadoop:hadoop-common</ignoredUnusedDeclaredDependency>
- </ignoredUnusedDeclaredDependencies>
- <ignoredDependencies>
- <ignoredDependency>org.apache.hadoop:hadoop-client-api</ignoredDependency>
- </ignoredDependencies>
- </configuration>
</plugin>
</plugins>
<sourceDirectory>${basedir}/src/java</sourceDirectory>
diff --git a/java/mapreduce/pom.xml b/java/mapreduce/pom.xml
index b2fd37a2d..330566ee1 100644
--- a/java/mapreduce/pom.xml
+++ b/java/mapreduce/pom.xml
@@ -49,11 +49,11 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
+ <artifactId>hadoop-client-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-core</artifactId>
+ <artifactId>hadoop-client-runtime</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
@@ -66,17 +66,6 @@
<artifactId>slf4j-api</artifactId>
<scope>test</scope>
</dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- <version>${min.hadoop.version}</version>
- <scope>test</scope>
- </dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
@@ -119,16 +108,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
- <configuration>
- <ignoredUnusedDeclaredDependencies>
- <ignoredUnusedDeclaredDependency>org.apache.hadoop:hadoop-hdfs</ignoredUnusedDeclaredDependency>
- <ignoredUnusedDeclaredDependency>org.apache.hadoop:hadoop-mapreduce-client-jobclient</ignoredUnusedDeclaredDependency>
- </ignoredUnusedDeclaredDependencies>
- <ignoredDependencies>
- <ignoredDependency>org.apache.hadoop:hadoop-client-api</ignoredDependency>
- <ignoredDependency>org.apache.hadoop:hadoop-client-runtime</ignoredDependency>
- </ignoredDependencies>
- </configuration>
</plugin>
</plugins>
<sourceDirectory>${basedir}/src/java</sourceDirectory>
@@ -142,25 +121,5 @@
<directory>${build.dir}/mapreduce</directory>
</build>
</profile>
- <profile>
- <id>java17</id>
- <activation>
- <jdk>[17,)</jdk>
- </activation>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client-api</artifactId>
- <version>${hadoop.version}</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client-runtime</artifactId>
- <version>${hadoop.version}</version>
- <scope>test</scope>
- </dependency>
- </dependencies>
- </profile>
</profiles>
</project>
diff --git a/java/pom.xml b/java/pom.xml
index fff0dabb0..5f58db60a 100644
--- a/java/pom.xml
+++ b/java/pom.xml
@@ -62,7 +62,7 @@
<properties>
<checkstyle.version>10.11.0</checkstyle.version>
<example.dir>${project.basedir}/../../examples</example.dir>
- <hadoop.version>2.7.3</hadoop.version>
+ <hadoop.version>3.3.5</hadoop.version>
<junit.version>5.9.3</junit.version>
<maven-assembly-plugin.version>3.6.0</maven-assembly-plugin.version>
<maven-shade-plugin.version>3.4.1</maven-shade-plugin.version>
@@ -71,7 +71,6 @@
<maven.compiler.useIncrementalCompilation>false</maven.compiler.useIncrementalCompilation>
<maven.version>3.8.8</maven.version>
- <min.hadoop.version>2.7.3</min.hadoop.version>
<mockito.version>4.11.0</mockito.version>
<!-- Build Properties -->
<project.build.outputTimestamp>2023-05-15T16:29:49Z</project.build.outputTimestamp>
@@ -79,7 +78,6 @@
<storage-api.version>2.8.1</storage-api.version>
<surefire.version>3.0.0-M5</surefire.version>
<test.tmp.dir>${project.build.directory}/testing-tmp</test.tmp.dir>
- <tools.hadoop.version>3.3.5</tools.hadoop.version>
<zookeeper.version>3.8.1</zookeeper.version>
</properties>
@@ -155,195 +153,14 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${min.hadoop.version}</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>com.google.code.findbugs</groupId>
- <artifactId>jsr305</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.sun.jersey</groupId>
- <artifactId>jersey-json</artifactId>
- </exclusion>
- <exclusion>
- <groupId>commons-beanutils</groupId>
- <artifactId>commons-beanutils-core</artifactId>
- </exclusion>
- <exclusion>
- <groupId>commons-daemon</groupId>
- <artifactId>commons-daemon</artifactId>
- </exclusion>
- <exclusion>
- <groupId>commons-digester</groupId>
- <artifactId>commons-digester</artifactId>
- </exclusion>
- <exclusion>
- <groupId>commons-el</groupId>
- <artifactId>commons-el</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.servlet</groupId>
- <artifactId>servlet-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.servlet.jsp</groupId>
- <artifactId>jsp-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>jdk.tools</groupId>
- <artifactId>jdk.tools</artifactId>
- </exclusion>
- <exclusion>
- <groupId>net.java.dev.jets3t</groupId>
- <artifactId>jets3t</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.tukaani</groupId>
- <artifactId>xz</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.curator</groupId>
- <artifactId>curator-recipes</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.mortbay.jetty</groupId>
- <artifactId>jetty</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.mortbay.jetty</groupId>
- <artifactId>jetty-util</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.avro</groupId>
- <artifactId>avro</artifactId>
- </exclusion>
- <exclusion>
- <groupId>tomcat</groupId>
- <artifactId>jasper-compiler</artifactId>
- </exclusion>
- <exclusion>
- <groupId>tomcat</groupId>
- <artifactId>jasper-runtime</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-log4j12</artifactId>
- </exclusion>
- <exclusion>
- <groupId>log4j</groupId>
- <artifactId>log4j</artifactId>
- </exclusion>
- </exclusions>
+ <artifactId>hadoop-client-api</artifactId>
+ <version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- <version>${min.hadoop.version}</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>com.google.code.findbugs</groupId>
- <artifactId>jsr305</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.sun.jersey</groupId>
- <artifactId>jersey-server</artifactId>
- </exclusion>
- <exclusion>
- <groupId>commons-daemon</groupId>
- <artifactId>commons-daemon</artifactId>
- </exclusion>
- <exclusion>
- <groupId>io.netty</groupId>
- <artifactId>netty</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.servlet</groupId>
- <artifactId>servlet-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.servlet.jsp</groupId>
- <artifactId>jsp-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.avro</groupId>
- <artifactId>avro</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.fusesource.leveldbjni</groupId>
- <artifactId>leveldbjni-all</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.mortbay.jetty</groupId>
- <artifactId>jetty</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.mortbay.jetty</groupId>
- <artifactId>jetty-util</artifactId>
- </exclusion>
- <exclusion>
- <groupId>tomcat</groupId>
- <artifactId>jasper-runtime</artifactId>
- </exclusion>
- <exclusion>
- <groupId>xerces</groupId>
- <artifactId>xercesImpl</artifactId>
- </exclusion>
- <exclusion>
- <groupId>log4j</groupId>
- <artifactId>log4j</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-core</artifactId>
- <version>${min.hadoop.version}</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>com.google.code.findbugs</groupId>
- <artifactId>jsr305</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.servlet</groupId>
- <artifactId>servlet-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.servlet.jsp</groupId>
- <artifactId>jsp-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.mortbay.jetty</groupId>
- <artifactId>jetty</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.mortbay.jetty</groupId>
- <artifactId>jetty-util</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.avro</groupId>
- <artifactId>avro</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-log4j12</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
- <version>${min.hadoop.version}</version>
- <scope>test</scope>
- <exclusions>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-log4j12</artifactId>
- </exclusion>
- </exclusions>
+ <artifactId>hadoop-client-runtime</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
@@ -811,6 +628,15 @@
<requireMavenVersion>
<version>${maven.version}</version>
</requireMavenVersion>
+ <bannedDependencies>
+ <excludes>
+ <exclude>org.apache.hadoop:hadoop-common</exclude>
+ <exclude>org.apache.hadoop:hadoop-hdfs-client</exclude>
+ <exclude>org.apache.hadoop:hadoop-mapreduce-client-core</exclude>
+ <exclude>org.apache.hadoop:hadoop-mapreduce-client-jobclient</exclude>
+ </excludes>
+ <searchTransitive>true</searchTransitive>
+ </bannedDependencies>
</rules>
</configuration>
</execution>
@@ -973,16 +799,6 @@
<module>bench</module>
</modules>
</profile>
- <profile>
- <id>java17</id>
- <activation>
- <jdk>[17,)</jdk>
- </activation>
- <properties>
- <hadoop.version>3.3.5</hadoop.version>
- <min.hadoop.version>3.3.5</min.hadoop.version>
- </properties>
- </profile>
<profile>
<id>java8</id>
<activation>
diff --git a/java/shims/pom.xml b/java/shims/pom.xml
index 407289ea7..89d1c6646 100644
--- a/java/shims/pom.xml
+++ b/java/shims/pom.xml
@@ -34,13 +34,7 @@
<!-- inter-project -->
<dependency>
<groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
+ <artifactId>hadoop-client-api</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
@@ -82,11 +76,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
- <configuration>
- <ignoredDependencies>
- <ignoredDependency>org.apache.hadoop:hadoop-hdfs</ignoredDependency>
- </ignoredDependencies>
- </configuration>
</plugin>
</plugins>
<sourceDirectory>${basedir}/src/java</sourceDirectory>
@@ -100,18 +89,5 @@
<directory>${build.dir}/shims</directory>
</build>
</profile>
- <profile>
- <id>java17</id>
- <activation>
- <jdk>[17,)</jdk>
- </activation>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client-api</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- </dependencies>
- </profile>
</profiles>
</project>
diff --git a/java/tools/pom.xml b/java/tools/pom.xml
index f761088c6..aa68b9b9f 100644
--- a/java/tools/pom.xml
+++ b/java/tools/pom.xml
@@ -63,18 +63,6 @@
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${tools.hadoop.version}</version>
- <scope>compile</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs-client</artifactId>
- <version>${tools.hadoop.version}</version>
- <scope>compile</scope>
- </dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
@@ -208,25 +196,5 @@
<directory>${build.dir}/tools</directory>
</build>
</profile>
- <profile>
- <id>java17</id>
- <activation>
- <jdk>[17,)</jdk>
- </activation>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client-api</artifactId>
- <version>${hadoop.version}</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client-runtime</artifactId>
- <version>${hadoop.version}</version>
- <scope>test</scope>
- </dependency>
- </dependencies>
- </profile>
</profiles>
</project>