You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "minhpham1810 (via GitHub)" <gi...@apache.org> on 2026/09/07 23:05:37 UTC

Re: [I] Install instructions Sedona/R for AWS EMR? [sedona]

minhpham1810 commented on issue #1024:
URL: https://github.com/apache/sedona/issues/1024#issuecomment-5576489007

   I've drafted a "Use Sedona in R" section for `docs/setup/emr.md` (plus the `emr.zh.md` mirror), since there is currently no R deployment guidance anywhere in the mkdocs site — both "Install Sedona R" nav entries point at the generated pkgdown site, which covers the R API but not how to get Sedona onto a cluster.
   
   The parts I could verify against the repo:
   
   - `apache.sedona` registers itself as a `sparklyr` extension, and the extension's initializer calls `SedonaContext.create(spark_session(sc), "r")`. So attaching the package before `spark_connect()` is all that's needed — there's no R equivalent of `SedonaContext.create()` to call by hand.
   - `SEDONA_JAR_FILES` is split on `:`, and when it's set it replaces **both** Maven coordinates. So the `geotools-wrapper` jar has to be listed alongside the Sedona jar, otherwise GeoTools silently goes missing.
   - Unset, every connection resolves `sedona-spark-shaded-<spark>_<scala>` and `geotools-wrapper` from Maven Central, which needs outbound internet from the driver and can exceed the default `sparklyr.connect.timeout`. Pointing `SEDONA_JAR_FILES` at the two jars the page's existing bootstrap script already downloads to `/jars` avoids that entirely.
   - The R interface rejects Spark 4.x outright (`stop("Unsupported Spark version: ")` in `R/R/dependencies.R`), so this is Spark 3.x only.
   - YARN client mode with `spark_home = "/usr/lib/spark"` is already the pattern in the R vignette, and that happens to be EMR's Spark home.
   
   What I **can't** verify without a live cluster, and would like a second pair of eyes on before this is merged:
   
   1. **Which EMR release should this target?** The page still documents EMR 6.9.0 / Spark 3.3.0 with `sedona-spark-shaded-3.3_2.12`. EMR 7.x ships Spark 3.5. I wrote the R section against the page's existing baseline rather than silently changing it, but the whole page is arguably due a refresh.
   2. **Does the EMR AMI already ship R?** I assumed a bootstrap `sudo yum install -y R` is needed. If R is present on the AMI at the targeted release, that line should come out.
   3. **Which nodes need `sparklyr` + `apache.sedona`?** In YARN client mode I believe only the primary node (where the driver runs) does, but a bootstrap action runs everywhere. Worth confirming rather than guessing in the docs.
   4. **How is R actually meant to be driven on EMR?** EMR Studio notebooks have no R/Spark kernel, and AWS has since deprecated the EMR–RStudio integration, so both links in @gregleleu's reply above have aged. Is the expected workflow "SSH to the primary node and run `R`", or an edge node? This decides whether the section needs an "accessing R" paragraph at all.
   5. **Is `SEDONA_JAR_FILES` even necessary here?** The page's `spark-defaults` classification already sets `spark.yarn.dist.jars` to the same two `/jars` paths. If `spark_connect()` inherits that, the `Sys.setenv()` block is redundant and should be presented as a no-internet fallback rather than the default.
   
   @gregleleu, since you answered the original question — would you be willing to sanity-check the AWS-side assumptions above? Happy to adjust the draft either way.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org