You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2022/12/09 14:14:30 UTC
[spark] branch master updated: [SPARK-41458][BUILD][YARN][SHUFFLE] Correctly transform the SPI services for Yarn Shuffle Service
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new be52d67fbe9 [SPARK-41458][BUILD][YARN][SHUFFLE] Correctly transform the SPI services for Yarn Shuffle Service
be52d67fbe9 is described below
commit be52d67fbe98110eeabf1b2a7c16741dceefdca6
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Fri Dec 9 08:14:17 2022 -0600
[SPARK-41458][BUILD][YARN][SHUFFLE] Correctly transform the SPI services for Yarn Shuffle Service
### What changes were proposed in this pull request?
Correctly transform the SPI services for Yarn Shuffle Service by configuring `ServicesResourceTransformer`.
### Why are the changes needed?
SPARK-12807 relocated the Jackson classes, but did not handle SPI services properly.
It affects Spark 2.0 and above, so this PR is for 3.2/3.3/master.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually checked the output jar.
Before:
<img width="772" alt="Xnip2022-12-09_12-49-31" src="https://user-images.githubusercontent.com/26535726/206632421-acbec562-c600-4497-83a3-f9b2f6caba74.png">
After:
<img width="1209" alt="Xnip2022-12-09_12-53-39" src="https://user-images.githubusercontent.com/26535726/206632440-4c8ed745-dbc8-4b6e-a9e7-f285521aa8b4.png">
Closes #38989 from pan3793/SPARK-41458.
Authored-by: Cheng Pan <ch...@apache.org>
Signed-off-by: Sean Owen <sr...@gmail.com>
---
common/network-yarn/pom.xml | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index a4073969dbf..a77732bb8b8 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -136,6 +136,11 @@
<goals>
<goal>shade</goal>
</goals>
+ <configuration>
+ <transformers>
+ <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
+ </transformers>
+ </configuration>
</execution>
</executions>
</plugin>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org