You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "yadavay-amzn (via GitHub)" <gi...@apache.org> on 2026/04/11 02:20:15 UTC
[PR] [SPARK-56374][BUILD] Align SBT assembly shade rules with Maven [spark]
yadavay-amzn opened a new pull request, #55306:
URL: https://github.com/apache/spark/pull/55306
### What changes were proposed in this pull request?
Add missing shade rules to `project/SparkBuild.scala` to align SBT assembly output with Maven for three connect modules:
1. **SparkConnect (server)**: Add `com.google.common` → `org.sparkproject.guava` and `com.google.thirdparty` → `org.sparkproject.guava.thirdparty` relocations. Maven's `sql/connect/server/pom.xml` has these but SBT was missing them.
2. **SparkConnectClient (jvm)**: Add `org.apache.arrow` → `org.sparkproject.connect.client.org.apache.arrow` relocation. Maven's `connector/connect/client/jvm/pom.xml` has this but SBT was missing it.
3. **SparkConnectJdbc**: Add `org.apache.arrow` relocation for consistency with Maven's `sql/connect/client/jdbc/pom.xml`.
### Why are the changes needed?
SBT assembly shade rules were out of sync with Maven, causing differences in the assembled JARs:
- **Server**: Without the guava relocation, grpc classes in the server assembly reference unshaded `com.google.common.*`. At runtime these fail to resolve because Guava is shaded to `org.sparkproject.guava` in `spark-network-common`. Verified by inspecting `ManagedChannelImpl.class` — after the fix, references correctly point to `org/sparkproject/guava/base/MoreObjects` instead of `com/google/common/base/MoreObjects`.
- **Client JVM**: Arrow classes were not being shaded. After the fix, they appear under `org/sparkproject/connect/client/org/apache/arrow/`.
Other Maven modules with shade rules (`core`, `sql/core`, `network-yarn`, root `pom.xml`) were verified to be intentionally different in SBT — those modules don't produce separate assembly JARs in SBT, and their shading is handled at a different level in the SBT build architecture.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Built all three affected SBT assemblies and verified the output JARs:
```
build/sbt connect/assembly # ✅ success
build/sbt connect-client-jvm/assembly # ✅ success
build/sbt connect-client-jdbc/assembly # ✅ success
```
Verified shading in the output JARs:
- Server assembly: `strings ManagedChannelImpl.class` shows `org/sparkproject/guava/base/MoreObjects` (was `com/google/common/base/MoreObjects` before fix)
- Client JVM assembly: `jar tf` shows arrow classes under `org/sparkproject/connect/client/org/apache/arrow/` with zero unshaded `org/apache/arrow` entries
### Was this patch authored or co-authored using generative AI tooling?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56374][BUILD] Align SBT assembly shade rules with Maven [spark]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn closed pull request #55306: [SPARK-56374][BUILD] Align SBT assembly shade rules with Maven
URL: https://github.com/apache/spark/pull/55306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56374][BUILD] Align SBT assembly shade rules with Maven [spark]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn closed pull request #55306: [SPARK-56374][BUILD] Align SBT assembly shade rules with Maven
URL: https://github.com/apache/spark/pull/55306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56374][BUILD] Align SBT assembly shade rules with Maven [spark]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn closed pull request #55306: [SPARK-56374][BUILD] Align SBT assembly shade rules with Maven
URL: https://github.com/apache/spark/pull/55306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org