You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "minhpham1810 (via GitHub)" <gi...@apache.org> on 2026/09/07 23:11:45 UTC
[PR] [GH-1024] Add Sedona R instructions to the AWS EMR setup page [sedona]
minhpham1810 opened a new pull request, #3334:
URL: https://github.com/apache/sedona/pull/3334
## Did you read the Contributor Guide?
- Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/)
## Is this PR related to a ticket?
- Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #1024
## What changes were proposed in this PR?
Adds a `## Use Sedona in R` section to `docs/setup/emr.md` and its Chinese mirror `docs/setup/emr.zh.md`.
There is currently no R deployment guidance anywhere in the mkdocs site — both "Install Sedona R" nav entries point at the generated pkgdown site, which documents the R API but not how to get Sedona onto a cluster. The EMR page covers Scala and Python and stops there.
The new section covers three things:
- extending the page's existing bootstrap script to install R, `sparklyr` and `apache.sedona`;
- connecting with `spark_connect(master = "yarn", spark_home = "/usr/lib/spark")`, pointing `SEDONA_JAR_FILES` at the two jars the bootstrap script already downloads into `/jars`;
- a one-liner to verify the R side.
The Sedona-specific claims are grounded in `R/R/dependencies.R`: `apache.sedona` registers itself as a `sparklyr` extension whose initializer calls `SedonaContext.create(spark_session(sc), "r")`, so there is no manual registration step in R; `SEDONA_JAR_FILES` is split on `:` and, when set, replaces **both** Maven coordinates, which is why the `geotools-wrapper` jar has to be listed alongside the Sedona jar; and the R interface rejects Spark 4.x outright, so this is Spark 3.x only.
## Two open questions for reviewers
I don't have an EMR cluster to validate against, and these two would change the text:
1. **Which EMR release should this target?** The page still documents EMR 6.9.0 / Spark 3.3.0 with `sedona-spark-shaded-3.3_2.12`. EMR 7.x ships Spark 3.5. I wrote the R section against the page's existing baseline rather than silently changing it, but the whole page looks due a refresh — happy to do that here or in a follow-up, whichever you prefer.
2. **Is `SEDONA_JAR_FILES` actually necessary here?** The page's `spark-defaults` classification already sets `spark.yarn.dist.jars` to the same two `/jars` paths. If `spark_connect()` inherits that, the `Sys.setenv()` block is redundant and should be presented as a no-internet fallback rather than as the default path.
Three further assumptions on the AWS side are flagged in https://github.com/apache/sedona/issues/1024#issuecomment-5576489007 — whether the EMR AMI already ships R, whether core/task nodes need the R packages in YARN client mode, and how R is expected to be driven on EMR now that Studio has no R kernel and the RStudio integration is deprecated.
## How was this patch tested?
- `prek run --files docs/setup/emr.md docs/setup/emr.zh.md` — all hooks pass (markdownlint, codespell, editorconfig-checker, license, whitespace).
- `mkdocs build` — succeeds; the `{{ sedona.current_version }}` / `{{ sedona.current_geotools }}` macros resolve to `1.9.1` and `1.9.1-33.5` in both locales, the three new headings anchor, and both new `!!!note` admonitions render.
- External links checked for lychee: all resolve 200 after redirects.
- The R snippets themselves were not executed against a live EMR cluster — see the open questions above.
## Did this PR include necessary documentation updates?
- Yes, I have updated the documentation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01LGHuMH6Jk2ForsfJPEvev6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-1024] Add Sedona R instructions to the AWS EMR setup page [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #3334:
URL: https://github.com/apache/sedona/pull/3334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-1024] Add Sedona R instructions to the AWS EMR setup page [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #3334:
URL: https://github.com/apache/sedona/pull/3334#discussion_r3975645610
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
+
+!!!note
+ The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version.
+
+### Extend the initialization script
+
+Add the following to the bootstrap script above, so that R and the two R packages are available on the node you run R from:
+
+```bash
+# Install R and the Sedona R interface
+sudo yum install -y R
+sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org")'
Review Comment:
This doesn't work on EMR 7.x as written. The R `curl` package fails to configure because the libcurl headers aren't on the image, and that takes httr, sparklyr and apache.sedona down with it:
```
Configuration failed because libcurl was not found. Try installing:
* rpm: libcurl-devel (Fedora, CentOS, RHEL)
ERROR: dependency 'curl' is not available for package 'httr'
ERROR: dependency 'httr' is not available for package 'sparklyr'
ERROR: dependency 'sparklyr' is not available for package 'apache.sedona'
```
`R -e` still exits 0, so the bootstrap reports success and you only find out at `library(sparklyr)`.
```bash
sudo yum install -y R libcurl-devel
```
is enough. I re-ran with that and all 38 packages built on both nodes (about 1.5 min for yum and 6.5 min for `install.packages` per node). openssl and xml2 are fine already because R-core-devel brings their headers.
Since this runs on every node, it may also be worth making the install fail loudly and use all cores:
```r
install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org", Ncpus = parallel::detectCores())
stopifnot(all(c("sparklyr", "apache.sedona") %in% rownames(installed.packages())))
```
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
+
+!!!note
+ The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version.
+
+### Extend the initialization script
+
+Add the following to the bootstrap script above, so that R and the two R packages are available on the node you run R from:
+
+```bash
+# Install R and the Sedona R interface
+sudo yum install -y R
+sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org")'
+```
+
+### Connect to the cluster from R
+
+EMR installs Spark under `/usr/lib/spark`. Point `SEDONA_JAR_FILES` at the jars the bootstrap script already downloaded into `/jars` so that `sparklyr` uses them instead of resolving the Sedona coordinates from Maven Central every time you connect:
+
+```r
+library(sparklyr)
+library(apache.sedona)
+
+Sys.setenv(
+ "SEDONA_JAR_FILES" = paste(
+ "/jars/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar",
Review Comment:
This is the Spark 3.3 jar, but since #3300 the bootstrap on master downloads `sedona-spark-shaded-3.5_2.12`. The hunk merges cleanly so nothing flags it, and `spark_connect()` dies with `FileNotFoundException: File file:/jars/sedona-spark-shaded-3.3_2.12-1.9.1.jar does not exist`. Rebase and change this to `3.5_2.12`.
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
+
+!!!note
+ The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version.
+
+### Extend the initialization script
+
+Add the following to the bootstrap script above, so that R and the two R packages are available on the node you run R from:
+
+```bash
+# Install R and the Sedona R interface
+sudo yum install -y R
+sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org")'
+```
+
+### Connect to the cluster from R
+
+EMR installs Spark under `/usr/lib/spark`. Point `SEDONA_JAR_FILES` at the jars the bootstrap script already downloaded into `/jars` so that `sparklyr` uses them instead of resolving the Sedona coordinates from Maven Central every time you connect:
+
+```r
+library(sparklyr)
+library(apache.sedona)
+
+Sys.setenv(
+ "SEDONA_JAR_FILES" = paste(
+ "/jars/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar",
+ "/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar",
+ sep = ":"
+ )
+)
+
+sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark")
+```
+
+!!!note
+ `SEDONA_JAR_FILES` holds a `:`-separated list and replaces *both* Maven coordinates that `apache.sedona` would otherwise request, which is why the GeoTools wrapper jar has to be listed next to the Sedona jar. If you leave `SEDONA_JAR_FILES` unset, every connection downloads `org.apache.sedona:sedona-spark-shaded-<spark version>_<scala version>:{{ sedona.current_version }}` and `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}`. That requires outbound internet access from the driver and can take long enough to exceed the default `sparklyr.connect.timeout`.
Review Comment:
This isn't what the code does. In `dependencies.R` the geotools-wrapper coordinate goes into `packages` unconditionally; `SEDONA_JAR_FILES` only drops the Sedona coordinate. With the env var set exactly like the snippet above, spark-submit was launched with
```
--jars /jars/sedona-spark-shaded-3.5_2.12-1.9.1.jar,/jars/geotools-wrapper-1.9.1-33.5.jar
--packages org.datasyslab:geotools-wrapper:1.9.1-33.5
```
and `~/.ivy2/jars` went from empty to containing the geotools-wrapper jar. So the driver still needs Maven Central (or a warm Ivy cache), and listing the geotools jar is harmless but not for the reason given here. Please reword this, and the "instead of resolving ... from Maven Central" sentence above the code block.
"every connection downloads" is also too strong: Ivy caches, so it's the first connection per user. On EMR that first connect took 26.7 s with the default 60 s timeout (both jars, about 109 MB), the second 24.3 s. The timeout warning is still fair for slow egress, just say "first connection".
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
+
+!!!note
+ The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version.
+
+### Extend the initialization script
+
+Add the following to the bootstrap script above, so that R and the two R packages are available on the node you run R from:
+
+```bash
+# Install R and the Sedona R interface
+sudo yum install -y R
+sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org")'
+```
+
+### Connect to the cluster from R
+
+EMR installs Spark under `/usr/lib/spark`. Point `SEDONA_JAR_FILES` at the jars the bootstrap script already downloaded into `/jars` so that `sparklyr` uses them instead of resolving the Sedona coordinates from Maven Central every time you connect:
Review Comment:
Worth saying here that `SEDONA_JAR_FILES` (or letting sparklyr pull `--packages`) is required, not an optimisation. `spark.yarn.dist.jars` only distributes the jars to executors; a yarn-client driver never gets them on its classpath. With plain sparklyr and nothing adding `--jars`, the driver logged `ClassNotFoundException: org.apache.sedona.viz.sql.SedonaVizExtensions` and `ST_AsText` was unresolved. The Jupyter section works only because EMR runs Livy in cluster deploy mode. Without a sentence on this, readers will drop the env var assuming spark-defaults covers it, which is what your open question 2 asks.
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
+
+!!!note
+ The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version.
+
+### Extend the initialization script
+
+Add the following to the bootstrap script above, so that R and the two R packages are available on the node you run R from:
Review Comment:
A bootstrap action runs on every node, not just the one you run R from. That's fine in practice, and workers only need the R packages for `spark_apply()` (an executor-side aggregation ran without sparklyr on the core node), but the wording suggests otherwise.
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
+
+!!!note
+ The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version.
Review Comment:
Maybe "apache.sedona 1.9.1 supports Spark 3.x only". Spark 4 support went in with #3340 for the next R release, so this will be stale soon without the version.
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
Review Comment:
Confirmed, no change needed. One side effect worth knowing: attaching the package sets `spark.kryo.registrator` to `SedonaVizKryoRegistrator` through an R option, which sparklyr passes as `--conf`, overriding the `SedonaKryoRegistrator` from spark-defaults. Harmless, it's a superset.
##########
docs/setup/emr.zh.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
注意:您不需要再调用 `SedonaRegistrator.registerAll(spark)` 或 `SedonaContext.create(spark)`,因为配置中的 `org.apache.sedona.sql.SedonaSqlExtensions` 已经为您完成了这些工作。
+
+## 在 R 中使用 Sedona
Review Comment:
Same edits needed here: libcurl-devel, `3.5_2.12`, and the two reworded notes. The translation itself reads fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-1024] Add Sedona R instructions to the AWS EMR setup page [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #3334:
URL: https://github.com/apache/sedona/pull/3334#discussion_r3975645610
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
+
+!!!note
+ The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version.
+
+### Extend the initialization script
+
+Add the following to the bootstrap script above, so that R and the two R packages are available on the node you run R from:
+
+```bash
+# Install R and the Sedona R interface
+sudo yum install -y R
+sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org")'
Review Comment:
**issue (blocking):** `sudo yum install -y R` is not enough on EMR 7.x. sparklyr never installs, and the bootstrap still reports success.
The R `curl` package needs the libcurl headers, which the EMR 7.9.0 image (Amazon Linux 2023) does not ship. Bootstrap stderr on both nodes:
```
Configuration failed because libcurl was not found. Try installing:
* rpm: libcurl-devel (Fedora, CentOS, RHEL)
ERROR: configuration failed for package 'curl'
ERROR: dependency 'curl' is not available for package 'httr'
ERROR: dependency 'httr' is not available for package 'sparklyr'
ERROR: dependency 'sparklyr' is not available for package 'apache.sedona'
```
`sudo R -e 'install.packages(...)'` exits 0 on package warnings, so the cluster comes up with 34 of 38 packages and no sparklyr; the first visible error is `library(sparklyr)` failing later. Change the line to:
```bash
sudo yum install -y R libcurl-devel
```
Verified on a second EMR 7.9.0 cluster: all 38 packages install on both nodes (yum 86-88 s, `install.packages` 379-383 s per node, cluster ready in about 14 minutes). openssl and xml2 already build because `R-core-devel` pulls their headers.
**suggestion:** make the install fail loudly and use the available cores, so a broken install fails the bootstrap action instead of surfacing at `library()` time:
```r
install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org", Ncpus = parallel::detectCores())
stopifnot(all(c("sparklyr", "apache.sedona") %in% rownames(installed.packages())))
```
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
+
+!!!note
+ The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version.
+
+### Extend the initialization script
+
+Add the following to the bootstrap script above, so that R and the two R packages are available on the node you run R from:
+
+```bash
+# Install R and the Sedona R interface
+sudo yum install -y R
+sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org")'
+```
+
+### Connect to the cluster from R
+
+EMR installs Spark under `/usr/lib/spark`. Point `SEDONA_JAR_FILES` at the jars the bootstrap script already downloaded into `/jars` so that `sparklyr` uses them instead of resolving the Sedona coordinates from Maven Central every time you connect:
+
+```r
+library(sparklyr)
+library(apache.sedona)
+
+Sys.setenv(
+ "SEDONA_JAR_FILES" = paste(
+ "/jars/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar",
Review Comment:
**issue (blocking):** this names `sedona-spark-shaded-3.3_2.12`, but the page's bootstrap now downloads `sedona-spark-shaded-3.5_2.12`.
The branch predates #3300, which moved this page to EMR 7.9.0 / Spark 3.5. The hunk is append-only, so git merges it cleanly with the mismatch intact. Run live, `spark_connect()` fails with:
```
java.io.FileNotFoundException: File file:/jars/sedona-spark-shaded-3.3_2.12-1.9.1.jar does not exist
```
Please rebase and use `3.5_2.12` here. This also settles open question 1: the page already targets EMR 7.9.0, and every EMR 7.x release ships Spark 3.5 (7.14 ships 3.5.8).
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
+
+!!!note
+ The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version.
+
+### Extend the initialization script
+
+Add the following to the bootstrap script above, so that R and the two R packages are available on the node you run R from:
+
+```bash
+# Install R and the Sedona R interface
+sudo yum install -y R
+sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org")'
+```
+
+### Connect to the cluster from R
+
+EMR installs Spark under `/usr/lib/spark`. Point `SEDONA_JAR_FILES` at the jars the bootstrap script already downloaded into `/jars` so that `sparklyr` uses them instead of resolving the Sedona coordinates from Maven Central every time you connect:
+
+```r
+library(sparklyr)
+library(apache.sedona)
+
+Sys.setenv(
+ "SEDONA_JAR_FILES" = paste(
+ "/jars/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar",
+ "/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar",
+ sep = ":"
+ )
+)
+
+sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark")
+```
+
+!!!note
+ `SEDONA_JAR_FILES` holds a `:`-separated list and replaces *both* Maven coordinates that `apache.sedona` would otherwise request, which is why the GeoTools wrapper jar has to be listed next to the Sedona jar. If you leave `SEDONA_JAR_FILES` unset, every connection downloads `org.apache.sedona:sedona-spark-shaded-<spark version>_<scala version>:{{ sedona.current_version }}` and `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}`. That requires outbound internet access from the driver and can take long enough to exceed the default `sparklyr.connect.timeout`.
Review Comment:
**issue:** `SEDONA_JAR_FILES` does not replace both Maven coordinates, only the Sedona one.
In `R/R/dependencies.R`, `packages` is initialised with the geotools-wrapper coordinate before the `SEDONA_JAR_FILES` branch and is never cleared. The live `spark-submit` process on EMR 7.9.0, with the env var set exactly as this snippet shows:
```
--jars /jars/sedona-spark-shaded-3.5_2.12-1.9.1.jar,/jars/geotools-wrapper-1.9.1-33.5.jar
--packages org.datasyslab:geotools-wrapper:1.9.1-33.5
```
`~/.ivy2/jars` was empty before the connect and held `org.datasyslab_geotools-wrapper-1.9.1-33.5.jar` after it. So the stated reason for listing the GeoTools jar is wrong, the driver still needs Maven Central (or a warm Ivy cache) even with the variable set, and the sentence above the code block about avoiding Maven resolution is only half true. Please describe the actual behaviour. Listing the GeoTools jar is harmless, since both copies are the same version.
**suggestion (non-blocking):** "every connection downloads" overstates it. Ivy caches under `~/.ivy2`, so only the first connection per OS user downloads. Measured with `SEDONA_JAR_FILES` unset and the default 60 s `sparklyr.connect.timeout`: first connect 26.7 s (both jars, about 109 MB), second connect 24.3 s from cache. Suggested wording: the first connection downloads about 110 MB, which on a slow egress path can exceed the 60-second default.
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
+
+!!!note
+ The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version.
Review Comment:
**nitpick:** qualify this with the package version, e.g. "apache.sedona 1.9.1 supports Spark 3.x only". Spark 4 support merged in #3340 for the next R release, and the qualifier keeps this sentence from going stale.
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
+
+!!!note
+ The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version.
+
+### Extend the initialization script
+
+Add the following to the bootstrap script above, so that R and the two R packages are available on the node you run R from:
+
+```bash
+# Install R and the Sedona R interface
+sudo yum install -y R
+sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org")'
+```
+
+### Connect to the cluster from R
+
+EMR installs Spark under `/usr/lib/spark`. Point `SEDONA_JAR_FILES` at the jars the bootstrap script already downloaded into `/jars` so that `sparklyr` uses them instead of resolving the Sedona coordinates from Maven Central every time you connect:
Review Comment:
**suggestion:** state explicitly that `SEDONA_JAR_FILES` is required in yarn-client mode, because `spark.yarn.dist.jars` never reaches the driver.
This answers open question 2. Connecting with plain sparklyr (no `apache.sedona` attached, so nothing adds `--jars` or `--packages`), the session had `spark.yarn.dist.jars` set correctly from the page's `spark-defaults`, yet the driver logged `ClassNotFoundException: org.apache.sedona.viz.sql.SedonaVizExtensions` and `ST_AsText` failed with `UNRESOLVED_ROUTINE`. The Jupyter path on this page works only because EMR runs Livy sessions in cluster deploy mode (`livy.spark.deploy-mode cluster`). One sentence here prevents readers from dropping the variable as redundant.
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
Review Comment:
**note:** the claim holds. `apache.sedona` sets an R option that sparklyr forwards as `--conf`, so the session ran with `spark.kryo.registrator = org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator`, overriding the page's `SedonaKryoRegistrator`. Harmless, since the Viz registrator is a superset; no change needed.
##########
docs/setup/emr.zh.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
注意:您不需要再调用 `SedonaRegistrator.registerAll(spark)` 或 `SedonaContext.create(spark)`,因为配置中的 `org.apache.sedona.sql.SedonaSqlExtensions` 已经为您完成了这些工作。
+
+## 在 R 中使用 Sedona
Review Comment:
**todo:** the mirror needs the same edits as `emr.md`: `libcurl-devel` on the yum line, `3.5_2.12` in `SEDONA_JAR_FILES`, and the reworded notes about which coordinate `SEDONA_JAR_FILES` replaces and about first-connection downloads. The translation itself is faithful.
##########
docs/setup/emr.md:
##########
@@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```
Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.
+
+## Use Sedona in R
+
+The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.
+
+!!!note
+ The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version.
+
+### Extend the initialization script
+
+Add the following to the bootstrap script above, so that R and the two R packages are available on the node you run R from:
Review Comment:
**nitpick:** "on the node you run R from" is misleading. A bootstrap action runs on every node, so the packages land everywhere, and worker nodes need them only for `spark_apply()`. An executor-side aggregation over 100,000 points ran fine with sparklyr absent from the core node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org