[I] Shaded jars should exclude the entire Scala toolchain (org.scala-lang:*), not just scala-library [sedona]

["james-willis (via GitHub)" <gi...@apache.org>]

james-willis opened a new issue, #3095:
URL: https://github.com/apache/sedona/issues/3095

   ## Expected behavior
   
   The shaded fat jars (`sedona-spark-shaded`, `sedona-flink-shaded`, `sedona-snowflake`) should never bundle unrelocated Scala toolchain classes (`scala-library`, `scala-reflect`, `scala-compiler`). The runtime (Spark/Flink) provides its own Scala distribution, and an unrelocated copy inside the fat jar shadows it.
   
   ## Actual behavior
   
   The maven-shade-plugin `artifactSet` in all three shaded modules only excludes `org.scala-lang:scala-library`:
   
   - `spark-shaded/pom.xml`
   - `flink-shaded/pom.xml`
   - `snowflake/pom.xml`
   
   If any dependency (present or future) transitively pulls in `org.scala-lang:scala-reflect` or `scala-compiler` at compile scope, it gets silently bundled unrelocated into the shaded jar.
   
   The published Sedona 1.9.0 shaded jars are currently clean (verified: no `scala/reflect/` or `scala/tools/` entries), so this is preventive hardening rather than an active bug upstream. However, we hit the failure mode for real in a downstream build derived from this pom: a transitive dependency chain (`geomesa-z3 -> scala-logging -> scala-reflect:2.13.8`) silently bundled an old unrelocated `scala.reflect.internal.*` (including all 16 `StdNames` classes) into the shaded jar. Under `spark-submit` that stale copy shadowed Spark's `scala-reflect-2.13.16`, and any Scala-compiler user on the classpath broke — e.g. almond/Ammonite kernels failed with:
   
   ```
   java.lang.NoSuchMethodError: scala.reflect.internal.StdNames$TypeNames.PermittedSubclassesATTR()
   ```
   
   (`PermittedSubclassesATTR` was added to `StdNames` in Scala 2.13.12, so any pre-2.13.12 `scala-reflect` copy shadowing a newer one produces this.)
   
   ## Proposed fix
   
   Change the exclude in all three shaded modules from:
   
   ```xml
   <exclude>org.scala-lang:scala-library</exclude>
   ```
   
   to:
   
   ```xml
   <exclude>org.scala-lang:*</exclude>
   ```
   
   This drops `scala-library`, `scala-reflect`, and `scala-compiler` from the fat jar. The groupId match is exact, so `org.scala-lang.modules:scala-collection-compat` (which Spark does not ship and must stay bundled) is unaffected.
   
   ## Steps to reproduce the problem
   
   Add any compile-scope dependency that transitively pulls `org.scala-lang:scala-reflect` to `common` or `spark/common`, build `spark-shaded`, and observe unrelocated `scala/reflect/**` classes in the jar (`unzip -l ... | grep scala/reflect`).
   
   ## Settings
   
   Sedona version = 1.9.0 (master)
   
   API type = Scala, Java
   
   Apache Spark version = 3.5 / 4.0
   
   Environment = any


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Shaded jars should exclude the entire Scala toolchain (org.scala-lang:*), not just scala-library [sedona]

["jiayuasu (via GitHub)" <gi...@apache.org>]

jiayuasu closed issue #3095: Shaded jars should exclude the entire Scala toolchain (org.scala-lang:*), not just scala-library
URL: https://github.com/apache/sedona/issues/3095


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org