You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "ilyaavgul-tomtom (via GitHub)" <gi...@apache.org> on 2026/03/31 15:15:51 UTC
[I] CompileException: Unknown variable or type "org.apache.spark.sql.sedona_sql.UDT.GeometryUDT$$.MODULE$" with Sedona 1.8.1 on Databricks 17.3 LTS [sedona]
ilyaavgul-tomtom opened a new issue, #2820:
URL: https://github.com/apache/sedona/issues/2820
### Problem
After upgrading from Databricks Runtime 16.4 to 17.3 LTS, Spark whole-stage codegen fails with:
```
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 45, Column 66:
Unknown variable or type "org.apache.spark.sql.sedona_sql.UDT.GeometryUDT$$.MODULE$"
```
The same cluster config, jars, and application code work without issues on DBR 16.4.
### Setup
- **Sedona:** 1.8.1 (sedona-spark-shaded-4.0_2.13-1.8.1.jar + geotools-wrapper-1.8.1-33.1.jar) installed via init script to /databricks/jars/
- **Scala:** 2.13.15
- **Spark config:**
```
spark.driver.maxResultSize 8g
spark.databricks.delta.preview.enabled true
spark.network.timeout 600s
spark.shuffle.io.retryWait 30s
spark.cleaner.referenceTracking.cleanCheckpoints true
spark.sql.shuffle.partitions auto
```
### Reproduction
The error triggers when calling `.map()` on a DataFrame produced by `Adapter.toDf()` after a spatial join via the RDD API:
```scala
import org.apache.sedona.core.spatialOperator.{JoinQuery, SpatialPredicate}
import org.apache.sedona.sql.utils.Adapter
import org.locationtech.jts.geom.Geometry
val joinResult = JoinQuery.SpatialJoinQueryFlat(blueSpatialRdd, redSpatialRdd, true, SpatialPredicate.INTERSECTS)
val df = Adapter.toDf(joinResult, Seq("redId"), Seq("blueId"), spark)
// This triggers whole-stage codegen which fails on DBR 17.3 LTS
df.map(row => row.getAs[Geometry]("leftgeometry")).collect()
```
**DBR 16.4:** works
**DBR 17.3 LTS:** fails with `Unknown variable or type "org.apache.spark.sql.sedona_sql.UDT.GeometryUDT$$.MODULE$"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CompileException: Unknown variable or type "org.apache.spark.sql.sedona_sql.UDT.GeometryUDT$$.MODULE$" with Sedona 1.8.1 on Databricks 17.3 LTS [sedona]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2820:
URL: https://github.com/apache/sedona/issues/2820#issuecomment-4163499991
👋 **Welcome to Apache Sedona!**
Thank you for contributing your first issue 🙌
We appreciate your effort to make Sedona even better.
💬 Join our [Sedona Discord Server](https://discord.gg/9A3k5dEBsY)
to connect with other developers, ask questions, and share ideas.
We're glad you're here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CompileException: Unknown variable or type "org.apache.spark.sql.sedona_sql.UDT.GeometryUDT$$.MODULE$" with Sedona 1.8.1 on Databricks 17.3 LTS [sedona]
Posted by "ilyaavgul-tomtom (via GitHub)" <gi...@apache.org>.
ilyaavgul-tomtom closed issue #2820: CompileException: Unknown variable or type "org.apache.spark.sql.sedona_sql.UDT.GeometryUDT$$.MODULE$" with Sedona 1.8.1 on Databricks 17.3 LTS
URL: https://github.com/apache/sedona/issues/2820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CompileException: Unknown variable or type "org.apache.spark.sql.sedona_sql.UDT.GeometryUDT$$.MODULE$" with Sedona 1.8.1 on Databricks 17.3 LTS [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #2820:
URL: https://github.com/apache/sedona/issues/2820#issuecomment-4214940589
Can you try to add the following command before the collect()?
```
implicit val enc = Encoders.kryo[Geometry]
val geometries = df.map(row => row.getAs[Geometry]("leftgeometry")).collect()
```
We've tested on DBR 14, 16 and 17, this setting works in all cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CompileException: Unknown variable or type "org.apache.spark.sql.sedona_sql.UDT.GeometryUDT$$.MODULE$" with Sedona 1.8.1 on Databricks 17.3 LTS [sedona]
Posted by "ilyaavgul-tomtom (via GitHub)" <gi...@apache.org>.
ilyaavgul-tomtom commented on issue #2820:
URL: https://github.com/apache/sedona/issues/2820#issuecomment-4258558671
Thanks, that seems to work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org