You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "jiayuasu (via GitHub)" <gi...@apache.org> on 2026/09/09 17:55:52 UTC
[I] R interface rejects Spark 4.x with "Unsupported Spark version" [sedona]
jiayuasu opened a new issue, #3338:
URL: https://github.com/apache/sedona/issues/3338
### Expected behavior
`library(apache.sedona)` followed by `spark_connect()` should work against Spark 4.0.x and 4.1.x, requesting the `sedona-spark-shaded-4.x_2.13` artifact (or using the jars listed in `SEDONA_JAR_FILES`), the same way it does for Spark 3.5.
### Actual behavior
`spark_connect()` fails while sparklyr collects extension dependencies, before any jar or Maven coordinate is considered:
```
Error in spark_dependencies(...): Unsupported Spark version: 4.0
```
Root cause is in `R/R/dependencies.R`. `spark_dependencies()` only accepts Spark major version 3 and defaults the Scala suffix to `2.12`. Widening the guard alone is not enough: sparklyr 1.9.5 hands every extension `scala_version = "2.12"` for any Spark release at or above 3.0, so a naive fix would request the non-existent `sedona-spark-shaded-4.0_2.12`. Spark 4 and Sedona's Spark 4 artifacts are Scala 2.13 only, so the R interface has to pin `2.13` for Spark 4 regardless of what sparklyr passes.
Setting `SEDONA_JAR_FILES` does not help, because the version check runs before the environment variable is consulted.
### Steps to reproduce
```shell
# EMR release emr-spark-8.0.0 (Spark 4.0.2, Scala 2.13.16, Java 17), apache.sedona 1.9.1 from CRAN, sparklyr 1.9.5
Rscript -e 'library(sparklyr); library(apache.sedona); sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark")'
# Error in spark_dependencies(spark_version, scala_version, ...) : Unsupported Spark version: 4.0
```
With the guard widened and the Scala suffix pinned to `2.13` for Spark 4, the same call connects on Spark 4.0.2, resolves `org.apache.sedona:sedona-spark-shaded-4.0_2.13:1.9.1` from Maven Central when `SEDONA_JAR_FILES` is unset, and `sdf_sql(sc, "SELECT ST_Point(0.0, 0.0)")` returns a geometry.
### Sedona version
1.9.1
### Apache Spark version
4.0.2 (4.1.x is affected the same way)
### API type
R
### Scala version
2.13
### JRE version
17
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] R interface rejects Spark 4.x with "Unsupported Spark version" [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu closed issue #3338: R interface rejects Spark 4.x with "Unsupported Spark version"
URL: https://github.com/apache/sedona/issues/3338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org