You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2023/02/13 14:28:02 UTC
[spark] branch branch-3.4 updated: [SPARK-42422][BUILD] Upgrade `maven-shade-plugin` to 3.4.1
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 4a73f25c7a8 [SPARK-42422][BUILD] Upgrade `maven-shade-plugin` to 3.4.1
4a73f25c7a8 is described below
commit 4a73f25c7a8b8553d4da84a08c56da5a5e9fb46d
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Mon Feb 13 08:27:29 2023 -0600
[SPARK-42422][BUILD] Upgrade `maven-shade-plugin` to 3.4.1
### What changes were proposed in this pull request?
This pr aims upgrade `maven-shade-plugin` from 3.2.4 to 3.4.1
### Why are the changes needed?
The `maven-shade-plugin` was [built by Java 8](https://github.com/apache/maven-shade-plugin/commit/33273411d30377773bc866bba46ec5f2fc39e60b) from 3.4.1, all other changes as follows:
- https://github.com/apache/maven-shade-plugin/releases/tag/maven-shade-plugin-3.3.0
- https://github.com/apache/maven-shade-plugin/compare/maven-shade-plugin-3.3.0...maven-shade-plugin-3.4.1
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Manual check:
There are 6 modules actually use shade function, checked the maven compilation logs manually:
1. spark-core
Before
```
[INFO] --- maven-shade-plugin:3.2.4:shade (default) spark-core_2.12 ---
[INFO] Including org.eclipse.jetty:jetty-plus:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-security:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-util:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-server:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-io:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-http:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-continuation:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-servlet:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-proxy:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-client:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-servlets:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
```
After
```
[INFO] --- maven-shade-plugin:3.4.1:shade (default) spark-core_2.12 ---
[INFO] Including org.eclipse.jetty:jetty-plus:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-security:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-util:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-server:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-io:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-http:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-continuation:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-servlet:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-proxy:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-client:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-servlets:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
```
2. spark-network-yarn
Before
```
[INFO] --- maven-shade-plugin:3.2.4:shade (default) spark-network-yarn_2.12 ---
[INFO] Including org.apache.spark:spark-network-shuffle_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.spark:spark-network-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including io.netty:netty-all:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-buffer:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-http:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-http2:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-socks:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-common:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-handler:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-unix-common:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-handler-proxy:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-resolver:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-classes-epoll:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-classes-kqueue:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.87.Final in the shaded jar.
[INFO] Including org.apache.commons:commons-lang3:jar:3.12.0 in the shaded jar.
[INFO] Including org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 in the shaded jar.
[INFO] Including org.rocksdb:rocksdbjni:jar:7.9.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.14.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.14.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.14.2 in the shaded jar.
[INFO] Including org.apache.commons:commons-crypto:jar:1.1.0 in the shaded jar.
[INFO] Including com.google.crypto.tink:tink:jar:1.7.0 in the shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.8.9 in the shaded jar.
[INFO] Including io.dropwizard.metrics:metrics-core:jar:4.2.15 in the shaded jar.
[INFO] Including org.roaringbitmap:RoaringBitmap:jar:0.9.39 in the shaded jar.
[INFO] Including org.roaringbitmap:shims:jar:0.9.39 in the shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:3.0.0 in the shaded jar.
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
```
After
```
[INFO] --- maven-shade-plugin:3.4.1:shade (default) spark-network-yarn_2.12 ---
[INFO] Including org.apache.spark:spark-network-shuffle_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.spark:spark-network-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including io.netty:netty-all:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-buffer:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-http:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-http2:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-socks:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-common:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-handler:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-unix-common:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-handler-proxy:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-resolver:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-classes-epoll:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-classes-kqueue:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.87.Final in the shaded jar.
[INFO] Including org.apache.commons:commons-lang3:jar:3.12.0 in the shaded jar.
[INFO] Including org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 in the shaded jar.
[INFO] Including org.rocksdb:rocksdbjni:jar:7.9.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.14.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.14.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.14.2 in the shaded jar.
[INFO] Including org.apache.commons:commons-crypto:jar:1.1.0 in the shaded jar.
[INFO] Including com.google.crypto.tink:tink:jar:1.7.0 in the shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.8.9 in the shaded jar.
[INFO] Including io.dropwizard.metrics:metrics-core:jar:4.2.15 in the shaded jar.
[INFO] Including org.roaringbitmap:RoaringBitmap:jar:0.9.39 in the shaded jar.
[INFO] Including org.roaringbitmap:shims:jar:0.9.39 in the shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:3.0.0 in the shaded jar.
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
```
3. spark-protobuf
Before
```
[INFO] --- maven-shade-plugin:3.2.4:shade (default) spark-protobuf_2.12 ---
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
```
After
```
[INFO] --- maven-shade-plugin:3.4.1:shade (default) spark-protobuf_2.12 ---
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
```
4. spark-connect-common
Before
```
[INFO] --- maven-shade-plugin:3.2.4:shade (default) spark-connect-common_2.12 ---
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
```
After
```
[INFO] --- maven-shade-plugin:3.4.1:shade (default) spark-connect-common_2.12 ---
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
```
5. spark-connect
Before
```
[INFO] --- maven-shade-plugin:3.2.4:shade (default) spark-connect_2.12 ---
[INFO] Including org.apache.spark:spark-connect-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
[INFO] Including org.checkerframework:checker-qual:jar:3.12.0 in the shaded jar.
[INFO] Including com.google.errorprone:error_prone_annotations:jar:2.7.1 in the shaded jar.
[INFO] Including com.google.j2objc:j2objc-annotations:jar:1.3 in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
[INFO] Including io.grpc:grpc-netty:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.9.0 in the shaded jar.
[INFO] Including com.google.android:annotations:jar:4.1.1.4 in the shaded jar.
[INFO] Including org.codehaus.mojo:animal-sniffer-annotations:jar:1.19 in the shaded jar.
[INFO] Including io.perfmark:perfmark-api:jar:0.25.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.api.grpc:proto-google-common-protos:jar:2.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar.
```
After
```
[INFO] --- maven-shade-plugin:3.4.1:shade (default) spark-connect_2.12 ---
[INFO] Including org.apache.spark:spark-connect-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
[INFO] Including org.checkerframework:checker-qual:jar:3.12.0 in the shaded jar.
[INFO] Including com.google.errorprone:error_prone_annotations:jar:2.7.1 in the shaded jar.
[INFO] Including com.google.j2objc:j2objc-annotations:jar:1.3 in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
[INFO] Including io.grpc:grpc-netty:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.9.0 in the shaded jar.
[INFO] Including com.google.android:annotations:jar:4.1.1.4 in the shaded jar.
[INFO] Including org.codehaus.mojo:animal-sniffer-annotations:jar:1.19 in the shaded jar.
[INFO] Including io.perfmark:perfmark-api:jar:0.25.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.api.grpc:proto-google-common-protos:jar:2.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar.
```
6. spark-connect-client-jvm
Before
```
[INFO] --- maven-shade-plugin:3.2.4:shade (default) spark-connect-client-jvm_2.12 ---
[INFO] Including org.apache.spark:spark-connect-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-netty:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
```
After
```
[INFO] --- maven-shade-plugin:3.4.1:shade (default) spark-connect-client-jvm_2.12 ---
[INFO] Including org.apache.spark:spark-connect-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-netty:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
```
Also checked the file numbers and file names in jars after decompression, the result is the same.
Closes #39994 from LuciferYang/SPARK-42422.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: Sean Owen <sr...@gmail.com>
(cherry picked from commit 9b625fdc7ddfb9eb29e0cc48ebbd0693a8f52f23)
Signed-off-by: Sean Owen <sr...@gmail.com>
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index e8eb7c4447a..d29d017dac8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -3151,7 +3151,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
- <version>3.2.4</version>
+ <version>3.4.1</version>
<dependencies>
<dependency>
<groupId>org.ow2.asm</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org