You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@sedona.apache.org by ji...@apache.org on 2023/12/19 20:25:00 UTC
(sedona) branch fix-examples updated: Update examples
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch fix-examples
in repository https://gitbox.apache.org/repos/asf/sedona.git
The following commit(s) were added to refs/heads/fix-examples by this push:
new 39216ec5a Update examples
39216ec5a is described below
commit 39216ec5ab7f034a385e3b3b5d1bffdea504d190
Author: Jia Yu <ji...@apache.org>
AuthorDate: Tue Dec 19 12:24:52 2023 -0800
Update examples
---
examples/README.md | 12 +-
examples/spark-rdd-colocation-mining/.gitignore | 3 -
examples/spark-rdd-colocation-mining/README.md | 62 ----
examples/spark-rdd-colocation-mining/build.sbt | 85 ------
.../spark-rdd-colocation-mining/colocationMap.png | Bin 30259 -> 0 bytes
.../spark-rdd-colocation-mining/project/.gitignore | 2 -
.../project/build.properties | 1 -
.../project/plugins.sbt | 1 -
.../src/test/resources/arealandmark.png | Bin 24322 -> 0 bytes
.../src/test/resources/colocation.png | Bin 489130 -> 0 bytes
.../src/test/resources/colocationMap.png | Bin 29529 -> 0 bytes
.../src/test/resources/colocationResult.png | Bin 576150 -> 0 bytes
.../src/test/resources/nyccolocation.png | Bin 1444124 -> 0 bytes
examples/spark-sql/.gitignore | 3 +-
examples/spark-sql/build.sbt | 85 ------
examples/spark-sql/colocationMap.png | Bin 0 -> 41586 bytes
examples/spark-sql/dependency-reduced-pom.xml | 166 +++++++++++
examples/spark-sql/pom.xml | 324 +++++++++++++++++++++
examples/spark-sql/project/.gitignore | 2 -
examples/spark-sql/project/build.properties | 1 -
examples/spark-sql/project/plugins.sbt | 1 -
examples/spark-sql/src/main/scala/Main.scala | 58 ++++
.../src/main/scala/RddExample.scala} | 66 ++---
.../spark-sql/src/main/scala/ScalaExample.scala | 153 ----------
examples/spark-sql/src/main/scala/SqlExample.scala | 128 ++++++++
examples/spark-sql/src/main/scala/VizExample.scala | 139 +++++++++
.../src/test/resources/arealm.csv | 0
.../nyc-area-landmark-shapefile/nyu_2451_34514.cst | 0
.../nyc-area-landmark-shapefile/nyu_2451_34514.dbf | Bin
.../nyc-area-landmark-shapefile/nyu_2451_34514.prj | 0
.../nyc-area-landmark-shapefile/nyu_2451_34514.shp | Bin
.../nyc-area-landmark-shapefile/nyu_2451_34514.shx | Bin
.../nyc-area-landmark-shapefile/wfsrequest.txt | 0
.../src/test/resources/primaryroads-polygon.csv | 0
.../src/test/resources/raster/FAA_UTM18N_NAD83.tif | Bin 0 -> 2785537 bytes
.../spark-sql/src/test/resources/raster/test2.tiff | Bin 174803 -> 0 bytes
.../resources/yellow_tripdata_2009-01-subset.csv | 0
.../src/test/resources/zcta510.csv | 0
examples/spark-viz/.gitignore | 2 -
examples/spark-viz/build.sbt | 85 ------
examples/spark-viz/project/.gitignore | 2 -
examples/spark-viz/project/build.properties | 1 -
examples/spark-viz/project/plugins.sbt | 1 -
.../spark-viz/src/main/scala/ScalaExample.scala | 223 --------------
.../src/test/resources/babylon.point.properties | 9 -
.../src/test/resources/babylon.polygon.properties | 9 -
.../test/resources/babylon.rectangle.properties | 9 -
47 files changed, 843 insertions(+), 790 deletions(-)
diff --git a/examples/README.md b/examples/README.md
index a5995f99b..f8b17af1b 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -1,27 +1,27 @@
# Sedona Template Project
-This repository contains template projects for RDD, SQL and Viz. The template projects have been configured properly.
+This repository contains template projects for Sedona on Spark and Flink. The template projects have been configured properly.
Note that, although the template projects are written in Scala, the same APIs can be used in Java as well.
## Folder structure
The folder structure of this repository is as follows.
-* rdd-colocation-mining: a scala template shows how to use Sedona RDD API in Spatial Data Mining
-* sql: a scala template shows how to use Sedona DataFrame and SQL API
-* viz: a scala template shows how to use Sedona Viz RDD and SQL API
+* spark-sql: a Scala template shows how to use Sedona RDD, DataFrame and SQL API
+* flink-sql: a Java template show how to use Sedona SQL via Flink Table APIs
+
## Compile and package
### Prerequisites
Please make sure you have the following software installed on your local machine:
-* For Scala: Scala 2.12, SBT
+* For Scala: Scala 2.12
* For Java: JDK 1.8, Apache Maven 3
### Compile
-Run a terminal command `sbt assembly` within the folder of each template
+Run a terminal command `mvn clean package` within the folder of each template
### Submit your fat jar to Spark
diff --git a/examples/spark-rdd-colocation-mining/.gitignore b/examples/spark-rdd-colocation-mining/.gitignore
deleted file mode 100644
index d10b13418..000000000
--- a/examples/spark-rdd-colocation-mining/.gitignore
+++ /dev/null
@@ -1,3 +0,0 @@
-/target/
-/.idea/
-/spark-warehouse/
diff --git a/examples/spark-rdd-colocation-mining/README.md b/examples/spark-rdd-colocation-mining/README.md
deleted file mode 100644
index 407501004..000000000
--- a/examples/spark-rdd-colocation-mining/README.md
+++ /dev/null
@@ -1,62 +0,0 @@
-# Sedona-Analysis
-
-This is a template project that shows how to use Sedona in Spatial Data Mining
-
-## Example
-
-### Spatial Co-Location Pattern Mining
-
-#### What is spatial co-location pattern mining
-Spatial co-location is defined as two or more species are often located in a neighborhood relationship. Ripley's K function is often used in judging co-location. It usually executes multiple times and form a 2-dimension curve for observation.
-
-***In Africa, lions co-locate with zebras.***
-
-<img src="https://github.com/jiayuasu/GeoSparkTemplateProject/raw/master/geospark-analysis/src/test/resources/colocation.png" width="300">
-
-#### Ripley's K function
-We use Ripley's K function to calculate **Multivariate Spatial Patterns**
-
-Here are some materials regarding how to use Ripley's K function and its transformation L function.
-
-Single type K function:
-
-* [ArcGIS documents](http://pro.arcgis.com/en/pro-app/tool-reference/spatial-statistics/h-how-multi-distance-spatial-cluster-analysis-ripl.htm)
-* Ripley's paper: Ripley, B.D. (1976). The second-order analysis of stationary point processes, Journal of Applied Probability
-13, 255–266.
-
-Multivariate K function and L function:
-
-* [Philip M. Dixon, Encyclopedia of Environmetrics](https://www3.nd.edu/~mhaenggi/ee87021/Dixon-K-Function.pdf)
-* [A blog about the co-location of Pokémon and Pokéstops in PokémonGo](http://blog.jlevente.com/understanding-the-cross-k-function/)
-
-#### New York City Taxi Trip and Area Landmarks
-
-The data scientist in NYC Taxi Company has a guess that the taxi pickup points are co-located with these area landmarks such as airports, museums, hospitals, colleges and so on. In other words, many taxi trips start from area landmarks. He wants to use a quantitative metric to measure the degree of co-location pattern.
-
-**Dataset**
-
-* NYC Taxi trip pickup points (CSV): http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml
-* NYC Area landmarks (Shapefile): https://geo.nyu.edu/catalog/nyu_2451_34514
-
-**Code**
-
-Run the code in "ScalaExample". You will obtain a visualized co-location map and the result of 10 iterations Ripley's L function.
-
-You can download more NYC taxi trip data to obtain a detailed analyze result.
-
-**Result**
-
-Visualized co-location map, use the subset in the template project. The output image is in the root folder:
-
-<img src="https://github.com/jiayuasu/GeoSparkTemplateProject/raw/master/geospark-analysis/src/test/resources/colocationMap.png" width="300">
-
-Visualized co-location map, use all 1.3 billion taxi trip pickup points:
-
-<img src="https://github.com/jiayuasu/GeoSparkTemplateProject/raw/master/geospark-analysis/src/test/resources/nyccolocation.png" width="300">
-
-Ripley's L function result:
-
-<img src="https://github.com/jiayuasu/GeoSparkTemplateProject/raw/master/geospark-analysis/src/test/resources/colocationResult.png" width="500">
-
-Conclusion:
-New York City taxi trip pickup points co-locate with New York City area landmarks
\ No newline at end of file
diff --git a/examples/spark-rdd-colocation-mining/build.sbt b/examples/spark-rdd-colocation-mining/build.sbt
deleted file mode 100644
index 0a138fa66..000000000
--- a/examples/spark-rdd-colocation-mining/build.sbt
+++ /dev/null
@@ -1,85 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-import sbt.Keys.{libraryDependencies, version}
-
-
-lazy val root = (project in file(".")).
- settings(
- name := "sedona-analysis",
-
- version := "0.1.0",
-
- scalaVersion := "2.12.12",
-
- organization := "org.apache.sedona",
-
- publishMavenStyle := true
- )
-
-val SparkVersion = "3.4.1"
-
-val SparkCompatibleVersion = "3.4"
-
-val HadoopVersion = "2.7.2"
-
-val SedonaVersion = "1.5.0"
-
-val ScalaCompatibleVersion = "2.12"
-
-// Change the dependency scope to "provided" when you run "sbt assembly"
-val dependencyScope = "compile"
-
-val geotoolsVersion = "1.5.0-28.2"
-
-//val jacksonVersion = "2.10.0"
-
-logLevel := Level.Warn
-
-logLevel in assembly := Level.Error
-
-libraryDependencies ++= Seq(
-// "com.fasterxml.jackson.core" % "jackson-databind" % jacksonVersion % dependencyScope,
-// "com.fasterxml.jackson.core" % "jackson-core" % jacksonVersion % dependencyScope,
- "org.apache.spark" %% "spark-core" % SparkVersion % dependencyScope exclude("org.apache.hadoop", "*"),
- "org.apache.spark" %% "spark-sql" % SparkVersion % dependencyScope exclude("org.apache.hadoop", "*"),
- "org.apache.hadoop" % "hadoop-mapreduce-client-core" % HadoopVersion % dependencyScope,
- "org.apache.hadoop" % "hadoop-common" % HadoopVersion % dependencyScope,
- "org.apache.hadoop" % "hadoop-hdfs" % HadoopVersion % dependencyScope,
- "org.apache.sedona" % "sedona-spark-".concat(SparkCompatibleVersion).concat("_").concat(ScalaCompatibleVersion) % SedonaVersion changing(),
- // The following GeoTools packages are only required if you need CRS transformation. Under GNU Lesser General Public License (LGPL) license
- "org.datasyslab" % "geotools-wrapper" % geotoolsVersion % "compile"
-)
-
-assemblyMergeStrategy in assembly := {
- case PathList("org.apache.sedona", "sedona-core", xs@_*) => MergeStrategy.first
- case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
- case PathList("META-INF","services",xs @ _*) => MergeStrategy.filterDistinctLines
- case path if path.endsWith(".SF") => MergeStrategy.discard
- case path if path.endsWith(".DSA") => MergeStrategy.discard
- case path if path.endsWith(".RSA") => MergeStrategy.discard
- case _ => MergeStrategy.first
-}
-
-resolvers ++= Seq(
- "Maven Central" at "https://repo.maven.apache.org/maven2/",
- "Open Source Geospatial Foundation Repository" at "https://repo.osgeo.org/repository/release/",
- "Apache Software Foundation Snapshots" at "https://repository.apache.org/content/groups/snapshots",
- "Java.net repository" at "https://download.java.net/maven/2"
-)
\ No newline at end of file
diff --git a/examples/spark-rdd-colocation-mining/colocationMap.png b/examples/spark-rdd-colocation-mining/colocationMap.png
deleted file mode 100644
index 407045bcf..000000000
Binary files a/examples/spark-rdd-colocation-mining/colocationMap.png and /dev/null differ
diff --git a/examples/spark-rdd-colocation-mining/project/.gitignore b/examples/spark-rdd-colocation-mining/project/.gitignore
deleted file mode 100644
index 6a60ed534..000000000
--- a/examples/spark-rdd-colocation-mining/project/.gitignore
+++ /dev/null
@@ -1,2 +0,0 @@
-/target/
-/project/
diff --git a/examples/spark-rdd-colocation-mining/project/build.properties b/examples/spark-rdd-colocation-mining/project/build.properties
deleted file mode 100644
index 331a838c9..000000000
--- a/examples/spark-rdd-colocation-mining/project/build.properties
+++ /dev/null
@@ -1 +0,0 @@
-sbt.version = 1.9.7
\ No newline at end of file
diff --git a/examples/spark-rdd-colocation-mining/project/plugins.sbt b/examples/spark-rdd-colocation-mining/project/plugins.sbt
deleted file mode 100644
index b64685617..000000000
--- a/examples/spark-rdd-colocation-mining/project/plugins.sbt
+++ /dev/null
@@ -1 +0,0 @@
-addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "2.1.4")
\ No newline at end of file
diff --git a/examples/spark-rdd-colocation-mining/src/test/resources/arealandmark.png b/examples/spark-rdd-colocation-mining/src/test/resources/arealandmark.png
deleted file mode 100644
index 57a57309d..000000000
Binary files a/examples/spark-rdd-colocation-mining/src/test/resources/arealandmark.png and /dev/null differ
diff --git a/examples/spark-rdd-colocation-mining/src/test/resources/colocation.png b/examples/spark-rdd-colocation-mining/src/test/resources/colocation.png
deleted file mode 100644
index 7ae1baa67..000000000
Binary files a/examples/spark-rdd-colocation-mining/src/test/resources/colocation.png and /dev/null differ
diff --git a/examples/spark-rdd-colocation-mining/src/test/resources/colocationMap.png b/examples/spark-rdd-colocation-mining/src/test/resources/colocationMap.png
deleted file mode 100644
index b516d3909..000000000
Binary files a/examples/spark-rdd-colocation-mining/src/test/resources/colocationMap.png and /dev/null differ
diff --git a/examples/spark-rdd-colocation-mining/src/test/resources/colocationResult.png b/examples/spark-rdd-colocation-mining/src/test/resources/colocationResult.png
deleted file mode 100644
index e73a1daad..000000000
Binary files a/examples/spark-rdd-colocation-mining/src/test/resources/colocationResult.png and /dev/null differ
diff --git a/examples/spark-rdd-colocation-mining/src/test/resources/nyccolocation.png b/examples/spark-rdd-colocation-mining/src/test/resources/nyccolocation.png
deleted file mode 100644
index 8cd4be3a9..000000000
Binary files a/examples/spark-rdd-colocation-mining/src/test/resources/nyccolocation.png and /dev/null differ
diff --git a/examples/spark-sql/.gitignore b/examples/spark-sql/.gitignore
index 0ad2ff7d1..2d513a010 100644
--- a/examples/spark-sql/.gitignore
+++ b/examples/spark-sql/.gitignore
@@ -1,3 +1,2 @@
-/target/
/.idea/
-/spark-warehouse/
\ No newline at end of file
+/target/
diff --git a/examples/spark-sql/build.sbt b/examples/spark-sql/build.sbt
deleted file mode 100644
index 68b1c1c6c..000000000
--- a/examples/spark-sql/build.sbt
+++ /dev/null
@@ -1,85 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-import sbt.Keys.{libraryDependencies, version}
-
-
-lazy val root = (project in file(".")).
- settings(
- name := "SedonaSQLTemplate",
-
- version := "0.1.0",
-
- scalaVersion := "2.12.11",
-
- organization := "org.apache.sedona",
-
- publishMavenStyle := true
- )
-
-val SparkVersion = "3.4.1"
-
-val SparkCompatibleVersion = "3.4"
-
-val HadoopVersion = "2.7.2"
-
-val SedonaVersion = "1.5.0"
-
-val ScalaCompatibleVersion = "2.12"
-
-// Change the dependency scope to "provided" when you run "sbt assembly"
-val dependencyScope = "compile"
-
-val geotoolsVersion = "1.5.0-28.2"
-
-//val jacksonVersion = "2.10.0"
-
-logLevel := Level.Warn
-
-logLevel in assembly := Level.Error
-
-libraryDependencies ++= Seq(
-// "com.fasterxml.jackson.core" % "jackson-databind" % jacksonVersion % dependencyScope,
-// "com.fasterxml.jackson.core" % "jackson-core" % jacksonVersion % dependencyScope,
- "org.apache.spark" %% "spark-core" % SparkVersion % dependencyScope exclude("org.apache.hadoop", "*"),
- "org.apache.spark" %% "spark-sql" % SparkVersion % dependencyScope exclude("org.apache.hadoop", "*"),
- "org.apache.hadoop" % "hadoop-mapreduce-client-core" % HadoopVersion % dependencyScope,
- "org.apache.hadoop" % "hadoop-common" % HadoopVersion % dependencyScope,
- "org.apache.hadoop" % "hadoop-hdfs" % HadoopVersion % dependencyScope,
- "org.apache.sedona" % "sedona-spark-".concat(SparkCompatibleVersion).concat("_").concat(ScalaCompatibleVersion) % SedonaVersion changing(),
- // The following GeoTools packages are only required if you need CRS transformation. Under GNU Lesser General Public License (LGPL) license
- "org.datasyslab" % "geotools-wrapper" % geotoolsVersion % "compile"
-)
-
-assemblyMergeStrategy in assembly := {
- case PathList("org.apache.sedona", "sedona-core", xs@_*) => MergeStrategy.first
- case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
- case PathList("META-INF","services",xs @ _*) => MergeStrategy.filterDistinctLines
- case path if path.endsWith(".SF") => MergeStrategy.discard
- case path if path.endsWith(".DSA") => MergeStrategy.discard
- case path if path.endsWith(".RSA") => MergeStrategy.discard
- case _ => MergeStrategy.first
-}
-
-resolvers ++= Seq(
- "Maven Central" at "https://repo.maven.apache.org/maven2/",
- "Open Source Geospatial Foundation Repository" at "https://repo.osgeo.org/repository/release/",
- "Apache Software Foundation Snapshots" at "https://repository.apache.org/content/groups/snapshots",
- "Java.net repository" at "https://download.java.net/maven/2"
-)
\ No newline at end of file
diff --git a/examples/spark-sql/colocationMap.png b/examples/spark-sql/colocationMap.png
new file mode 100644
index 000000000..3e065f648
Binary files /dev/null and b/examples/spark-sql/colocationMap.png differ
diff --git a/examples/spark-sql/dependency-reduced-pom.xml b/examples/spark-sql/dependency-reduced-pom.xml
new file mode 100644
index 000000000..dd5a2f610
--- /dev/null
+++ b/examples/spark-sql/dependency-reduced-pom.xml
@@ -0,0 +1,166 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.apache.sedona</groupId>
+ <artifactId>sedona-spark-example</artifactId>
+ <name>${project.groupId}:${project.artifactId}</name>
+ <version>1.0.0</version>
+ <description>Maven Example for SedonaDB</description>
+ <build>
+ <sourceDirectory>src/main/scala</sourceDirectory>
+ <resources>
+ <resource>
+ <directory>src/test/resources/</directory>
+ </resource>
+ </resources>
+ <plugins>
+ <plugin>
+ <groupId>net.alchim31.maven</groupId>
+ <artifactId>scala-maven-plugin</artifactId>
+ <version>3.2.1</version>
+ <executions>
+ <execution>
+ <id>scala-compile-first</id>
+ <phase>process-resources</phase>
+ <goals>
+ <goal>add-source</goal>
+ <goal>compile</goal>
+ </goals>
+ <configuration>
+ <args>
+ <arg>-dependencyfile</arg>
+ <arg>${project.build.directory}/.scala_dependencies</arg>
+ </args>
+ </configuration>
+ </execution>
+ <execution>
+ <id>scala-test-compile</id>
+ <phase>process-test-resources</phase>
+ <goals>
+ <goal>testCompile</goal>
+ </goals>
+ <configuration>
+ <args>
+ <arg>-dependencyfile</arg>
+ <arg>${project.build.directory}/.scala_dependencies</arg>
+ </args>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <artifactId>maven-jar-plugin</artifactId>
+ <version>3.3.0</version>
+ <configuration>
+ <archive>
+ <manifest>
+ <mainClass>Main</mainClass>
+ </manifest>
+ </archive>
+ </configuration>
+ </plugin>
+ <plugin>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <version>3.1</version>
+ <configuration>
+ <source>1.8</source>
+ <target>1.8</target>
+ </configuration>
+ </plugin>
+ <plugin>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>2.1</version>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <transformers>
+ <transformer />
+ <transformer>
+ <resource>reference.conf</resource>
+ </transformer>
+ <transformer>
+ <manifestEntries>
+ <Specification-Title>Java Advanced Imaging Image I/O Tools</Specification-Title>
+ <Specification-Version>1.1</Specification-Version>
+ <Specification-Vendor>Sun Microsystems, Inc.</Specification-Vendor>
+ <Implementation-Title>com.sun.media.imageio</Implementation-Title>
+ <Implementation-Version>1.1</Implementation-Version>
+ <Implementation-Vendor>Sun Microsystems, Inc.</Implementation-Vendor>
+ <Extension-Name>com.sun.media.imageio</Extension-Name>
+ </manifestEntries>
+ </transformer>
+ </transformers>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ <repositories>
+ <repository>
+ <releases />
+ <snapshots>
+ <enabled>false</enabled>
+ </snapshots>
+ <id>maven-central</id>
+ <name>Maven Central Repository</name>
+ <url>https://repo.maven.apache.org/maven2/</url>
+ </repository>
+ <repository>
+ <id>maven2-repository.dev.java.net</id>
+ <name>Java.net repository</name>
+ <url>https://download.java.net/maven/2</url>
+ </repository>
+ <repository>
+ <releases />
+ <snapshots>
+ <enabled>false</enabled>
+ </snapshots>
+ <id>osgeo</id>
+ <name>OSGeo Release Repository</name>
+ <url>https://repo.osgeo.org/repository/release/</url>
+ </repository>
+ </repositories>
+ <dependencies>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.12</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <artifactId>hamcrest-core</artifactId>
+ <groupId>org.hamcrest</groupId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+ <properties>
+ <spark.compat.version>3.4</spark.compat.version>
+ <geotools.version>28.2</geotools.version>
+ <sedona.scope>compile</sedona.scope>
+ <geotools.scope>compile</geotools.scope>
+ <jt-jiffle.version>1.1.24</jt-jiffle.version>
+ <sedona.version>1.5.0</sedona.version>
+ <dependency.scope>compile</dependency.scope>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <hadoop.version>3.3.4</hadoop.version>
+ <scala.compat.version>2.12</scala.compat.version>
+ <spark.version>3.4.0</spark.version>
+ </properties>
+</project>
+
diff --git a/examples/spark-sql/pom.xml b/examples/spark-sql/pom.xml
new file mode 100644
index 000000000..b40417470
--- /dev/null
+++ b/examples/spark-sql/pom.xml
@@ -0,0 +1,324 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.apache.sedona</groupId>
+ <artifactId>sedona-spark-example</artifactId>
+ <version>1.0.0</version>
+
+ <name>${project.groupId}:${project.artifactId}</name>
+ <description>Maven Example for SedonaDB</description>
+ <packaging>jar</packaging>
+
+ <properties>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <dependency.scope>compile</dependency.scope>
+ <sedona.scope>compile</sedona.scope>
+ <geotools.scope>compile</geotools.scope>
+
+ <geotools.version>28.2</geotools.version>
+ <jt-jiffle.version>1.1.24</jt-jiffle.version>
+ <spark.version>3.4.0</spark.version>
+ <spark.compat.version>3.4</spark.compat.version>
+ <scala.compat.version>2.12</scala.compat.version>
+ <sedona.version>1.5.0</sedona.version>
+ <hadoop.version>3.3.4</hadoop.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-core_${scala.compat.version}</artifactId>
+ <version>${spark.version}</version>
+ <scope>${dependency.scope}</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql_${scala.compat.version}</artifactId>
+ <version>${spark.version}</version>
+ <scope>${dependency.scope}</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.sedona</groupId>
+ <artifactId>sedona-spark-${spark.compat.version}_${scala.compat.version}</artifactId>
+ <version>${sedona.version}</version>
+ <scope>${sedona.scope}</scope>
+ </dependency>
+ <!--The following GeoTools dependencies use GNU Lesser General Public License and thus are excluded from the binary distribution-->
+ <!-- Users have to include them by themselves manually -->
+ <!-- See https://www.apache.org/legal/resolved.html#category-x -->
+ <!-- See https://github.com/geotools/geotools#license -->
+ <!--for CRS transformation-->
+ <dependency>
+ <groupId>org.geotools</groupId>
+ <artifactId>gt-main</artifactId>
+ <version>${geotools.version}</version>
+ <scope>${geotools.scope}</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.locationtech.jts</groupId>
+ <artifactId>jts-core</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <!--for CRS transformation-->
+ <dependency>
+ <groupId>org.geotools</groupId>
+ <artifactId>gt-referencing</artifactId>
+ <version>${geotools.version}</version>
+ <scope>${geotools.scope}</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>net.sf.geographiclib</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <!--for CRS transformation-->
+ <dependency>
+ <groupId>org.geotools</groupId>
+ <artifactId>gt-epsg-hsql</artifactId>
+ <version>${geotools.version}</version>
+ <scope>${geotools.scope}</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.geotools</groupId>
+ <artifactId>gt-shapefile</artifactId>
+ <version>${geotools.version}</version>
+ <scope>${geotools.scope}</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <!--for GeoTiff Reader-->
+ <dependency>
+ <groupId>org.geotools</groupId>
+ <artifactId>gt-geotiff</artifactId>
+ <version>${geotools.version}</version>
+ <scope>${geotools.scope}</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.geotools</groupId>
+ <artifactId>gt-process-feature</artifactId>
+ <version>${geotools.version}</version>
+ <scope>${geotools.scope}</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.geotools</groupId>
+ <artifactId>gt-arcgrid</artifactId>
+ <version>${geotools.version}</version>
+ <scope>${geotools.scope}</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.geotools</groupId>
+ <artifactId>gt-coverage</artifactId>
+ <version>${geotools.version}</version>
+ <scope>${geotools.scope}</scope>
+ </dependency>
+ <dependency>
+ <groupId>it.geosolutions.jaiext.jiffle</groupId>
+ <artifactId>jt-jiffle-language</artifactId>
+ <version>${jt-jiffle.version}</version>
+ <scope>${geotools.scope}</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <!-- for AWS S3a reader / writer -->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-aws</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${dependency.scope}</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.12</version>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+ <repositories>
+ <repository>
+ <id>maven-central</id>
+ <name>Maven Central Repository</name>
+ <url>https://repo.maven.apache.org/maven2/</url>
+ <snapshots>
+ <enabled>false</enabled>
+ </snapshots>
+ <releases>
+ <enabled>true</enabled>
+ </releases>
+ </repository>
+ <repository>
+ <id>maven2-repository.dev.java.net</id>
+ <name>Java.net repository</name>
+ <url>https://download.java.net/maven/2</url>
+ </repository>
+ <repository>
+ <id>osgeo</id>
+ <name>OSGeo Release Repository</name>
+ <url>https://repo.osgeo.org/repository/release/</url>
+ <snapshots>
+ <enabled>false</enabled>
+ </snapshots>
+ <releases>
+ <enabled>true</enabled>
+ </releases>
+ </repository>
+
+ </repositories>
+ <build>
+ <sourceDirectory>src/main/scala</sourceDirectory>
+ <plugins>
+ <plugin>
+ <groupId>net.alchim31.maven</groupId>
+ <artifactId>scala-maven-plugin</artifactId>
+ <version>3.2.1</version>
+ <executions>
+ <execution>
+ <id>scala-compile-first</id>
+ <phase>process-resources</phase>
+ <goals>
+ <goal>add-source</goal>
+ <goal>compile</goal>
+ </goals>
+ <configuration>
+ <args>
+ <arg>-dependencyfile</arg>
+ <arg>${project.build.directory}/.scala_dependencies</arg>
+ </args>
+ </configuration>
+ </execution>
+ <execution>
+ <id>scala-test-compile</id>
+ <phase>process-test-resources</phase>
+ <goals>
+ <goal>testCompile</goal>
+ </goals>
+ <configuration>
+ <args>
+ <arg>-dependencyfile</arg>
+ <arg>${project.build.directory}/.scala_dependencies</arg>
+ </args>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <!-- Build an executable JAR -->
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <version>3.3.0</version>
+ <configuration>
+ <archive>
+ <manifest>
+ <!-- <addClasspath>true</addClasspath> -->
+ <mainClass>Main</mainClass>
+ </manifest>
+ </archive>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <version>3.1</version>
+ <configuration>
+ <source>1.8</source>
+ <target>1.8</target>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>2.1</version>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <transformers>
+ <!-- use transformer to handle merge of META-INF/services - see http://java.net/jira/browse/JERSEY-440?focusedCommentId=14822&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_14822 -->
+ <transformer
+ implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
+ <transformer
+ implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
+ <resource>reference.conf</resource>
+ </transformer>
+ <transformer
+ implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+ <manifestEntries>
+ <Specification-Title>Java Advanced Imaging Image I/O Tools</Specification-Title>
+ <Specification-Version>1.1</Specification-Version>
+ <Specification-Vendor>Sun Microsystems, Inc.</Specification-Vendor>
+ <Implementation-Title>com.sun.media.imageio</Implementation-Title>
+ <Implementation-Version>1.1</Implementation-Version>
+ <Implementation-Vendor>Sun Microsystems, Inc.</Implementation-Vendor>
+ <Extension-Name>com.sun.media.imageio</Extension-Name>
+ </manifestEntries>
+ </transformer>
+ </transformers>
+ <filters>
+ <!-- filter to address "Invalid signature file" issue - see http://stackoverflow.com/a/6743609/589215-->
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ <resources>
+ <resource>
+ <directory>src/test/resources/</directory>
+ </resource>
+ </resources>
+ </build>
+</project>
+
diff --git a/examples/spark-sql/project/.gitignore b/examples/spark-sql/project/.gitignore
deleted file mode 100644
index 6a60ed534..000000000
--- a/examples/spark-sql/project/.gitignore
+++ /dev/null
@@ -1,2 +0,0 @@
-/target/
-/project/
diff --git a/examples/spark-sql/project/build.properties b/examples/spark-sql/project/build.properties
deleted file mode 100644
index 215ddd2b3..000000000
--- a/examples/spark-sql/project/build.properties
+++ /dev/null
@@ -1 +0,0 @@
-sbt.version = 1.5.5
\ No newline at end of file
diff --git a/examples/spark-sql/project/plugins.sbt b/examples/spark-sql/project/plugins.sbt
deleted file mode 100644
index 415991121..000000000
--- a/examples/spark-sql/project/plugins.sbt
+++ /dev/null
@@ -1 +0,0 @@
-addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "1.1.0")
\ No newline at end of file
diff --git a/examples/spark-sql/src/main/scala/Main.scala b/examples/spark-sql/src/main/scala/Main.scala
new file mode 100644
index 000000000..cd3e4c67a
--- /dev/null
+++ b/examples/spark-sql/src/main/scala/Main.scala
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import RddExample.{calculateSpatialColocation, visualizeSpatialColocation}
+import SqlExample._
+import VizExample._
+import org.apache.log4j.{Level, Logger}
+import org.apache.sedona.spark.SedonaContext
+import org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
+import org.apache.sedona.viz.sql.utils.SedonaVizRegistrator
+
+
+object Main extends App {
+ Logger.getRootLogger().setLevel(Level.WARN)
+
+ val config = SedonaContext.builder().appName("SedonaSQL-demo")
+ .master("local[*]") // Please comment out this when use it on a cluster
+ .config("spark.kryo.registrator", classOf[SedonaVizKryoRegistrator].getName)
+ .getOrCreate()
+ val sedona = SedonaContext.create(config)
+
+ SedonaVizRegistrator.registerAll(sedona)
+
+ val resourceFolder = System.getProperty("user.dir")+"/src/test/resources/"
+
+ testPredicatePushdownAndRangeJonQuery(sedona)
+ testDistanceJoinQuery(sedona)
+ testAggregateFunction(sedona)
+ testShapefileConstructor(sedona)
+ testRasterIOAndMapAlgebra(sedona)
+
+ visualizeSpatialColocation(sedona)
+ calculateSpatialColocation(sedona)
+
+ buildScatterPlot(sedona)
+ buildHeatMap(sedona)
+ buildChoroplethMap(sedona)
+ parallelFilterRenderNoStitch(sedona)
+ sqlApiVisualization(sedona)
+ System.out.println("All SedonaSQL DEMOs passed!")
+
+}
diff --git a/examples/spark-rdd-colocation-mining/src/main/scala/ScalaExample.scala b/examples/spark-sql/src/main/scala/RddExample.scala
similarity index 74%
rename from examples/spark-rdd-colocation-mining/src/main/scala/ScalaExample.scala
rename to examples/spark-sql/src/main/scala/RddExample.scala
index 504924251..d51133ff5 100644
--- a/examples/spark-rdd-colocation-mining/src/main/scala/ScalaExample.scala
+++ b/examples/spark-sql/src/main/scala/RddExample.scala
@@ -1,41 +1,34 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
+/**
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
*
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
*/
-import org.apache.log4j.{Level, Logger}
+import Main.resourceFolder
import org.apache.sedona.core.enums.{GridType, IndexType}
import org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader
import org.apache.sedona.core.spatialOperator.JoinQuery
import org.apache.sedona.core.spatialRDD.{CircleRDD, SpatialRDD}
-import org.apache.sedona.spark.SedonaContext
import org.apache.sedona.sql.utils.Adapter
-import org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
import org.apache.sedona.viz.core.{ImageGenerator, RasterOverlayOperator}
import org.apache.sedona.viz.extension.visualizationEffect.{HeatMap, ScatterPlot}
import org.apache.sedona.viz.utils.ImageType
+import org.apache.spark.sql.SparkSession
import org.locationtech.jts.geom.Geometry
import java.awt.Color
-object ScalaExample extends App{
-
- val resourceFolder = System.getProperty("user.dir")+"/src/test/resources/"
+object RddExample {
// Data link (in shapefile): https://geo.nyu.edu/catalog/nyu_2451_34514
val nycArealandmarkShapefileLocation = resourceFolder+"nyc-area-landmark-shapefile"
@@ -45,23 +38,7 @@ object ScalaExample extends App{
val colocationMapLocation = System.getProperty("user.dir")+"/colocationMap"
- Logger.getLogger("org").setLevel(Level.WARN)
- Logger.getLogger("akka").setLevel(Level.WARN)
-
- val config = SedonaContext.builder()
- .config("spark.kryo.registrator", classOf[SedonaVizKryoRegistrator].getName) // org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
- .master("local[*]") // Delete this if run in cluster mode
- .appName("sedona-analysis") // Change this to a proper name
- .getOrCreate()
- val sedona = SedonaContext.create(config)
-
- visualizeSpatialColocation()
- calculateSpatialColocation()
-
- System.out.println("Finished Sedona Spatial Analysis Example")
-
-
- def visualizeSpatialColocation(): Unit =
+ def visualizeSpatialColocation(sedona: SparkSession): Unit =
{
// Prepare NYC area landmarks which includes airports, museums, colleges, hospitals
var arealmRDD = ShapefileReader.readToPolygonRDD(sedona.sparkContext, nycArealandmarkShapefileLocation)
@@ -98,7 +75,7 @@ object ScalaExample extends App{
imageGenerator.SaveRasterImageAsLocalFile(overlayOperator.backRasterImage, colocationMapLocation, ImageType.PNG)
}
- def calculateSpatialColocation(): Unit =
+ def calculateSpatialColocation(sedona: SparkSession): Unit =
{
// Prepare NYC area landmarks which includes airports, museums, colleges, hospitals
@@ -154,13 +131,13 @@ object ScalaExample extends App{
var bufferedArealmRDD = new CircleRDD(arealmRDD,currentDistance)
bufferedArealmRDD.spatialPartitioning(tripRDD.getPartitioner)
-// Run Sedona Distance Join Query
+ // Run Sedona Distance Join Query
var adjacentMatrix = JoinQuery.DistanceJoinQueryFlat(tripRDD, bufferedArealmRDD,true,true)
-// Uncomment the following two lines if you want to see what the join result looks like in SparkSQL
-// import scala.collection.JavaConversions._
-// var adjacentMatrixDf = Adapter.toDf(adjacentMatrix, arealmRDD.fieldNames, tripRDD.fieldNames, sparkSession)
-// adjacentMatrixDf.show()
+ // Uncomment the following two lines if you want to see what the join result looks like in SparkSQL
+ // import scala.collection.JavaConversions._
+ // var adjacentMatrixDf = Adapter.toDf(adjacentMatrix, arealmRDD.fieldNames, tripRDD.fieldNames, sparkSession)
+ // adjacentMatrixDf.show()
var observedK = adjacentMatrix.count()*area*1.0/(arealmRDD.approximateTotalCount*tripRDD.approximateTotalCount)
var observedL = Math.sqrt(observedK/Math.PI)
@@ -171,5 +148,4 @@ object ScalaExample extends App{
println(s"""$currentDistance,$observedL,$colocationDifference,$colocationStatus""")
}
}
-
}
diff --git a/examples/spark-sql/src/main/scala/ScalaExample.scala b/examples/spark-sql/src/main/scala/ScalaExample.scala
deleted file mode 100644
index 9f23304f6..000000000
--- a/examples/spark-sql/src/main/scala/ScalaExample.scala
+++ /dev/null
@@ -1,153 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-import org.apache.log4j.{Level, Logger}
-import org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader
-import org.apache.sedona.core.spatialRDD.SpatialRDD
-import org.apache.sedona.core.utils.SedonaConf
-import org.apache.sedona.sql.utils.{Adapter, SedonaSQLRegistrator}
-import org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
-import org.apache.sedona.viz.sql.utils.SedonaVizRegistrator
-import org.apache.spark.serializer.KryoSerializer
-import org.apache.spark.sql.SparkSession
-import org.locationtech.jts.geom.{Coordinate, Geometry, GeometryFactory}
-
-
-object ScalaExample extends App{
- Logger.getLogger("org").setLevel(Level.WARN)
- Logger.getLogger("akka").setLevel(Level.WARN)
-
- var sparkSession:SparkSession = SparkSession.builder().config("spark.serializer",classOf[KryoSerializer].getName).
- config("spark.kryo.registrator", classOf[SedonaVizKryoRegistrator].getName).
- master("local[*]").appName("SedonaSQL-demo").getOrCreate()
-
- SedonaSQLRegistrator.registerAll(sparkSession)
- SedonaVizRegistrator.registerAll(sparkSession)
-
- val resourceFolder = System.getProperty("user.dir")+"/src/test/resources/"
-
- val csvPolygonInputLocation = resourceFolder + "testenvelope.csv"
- val csvPointInputLocation = resourceFolder + "testpoint.csv"
- val shapefileInputLocation = resourceFolder + "shapefiles/dbf"
- val rasterdatalocation = resourceFolder + "raster/"
-
- testPredicatePushdownAndRangeJonQuery()
- testDistanceJoinQuery()
- testAggregateFunction()
- testShapefileConstructor()
- testRasterIOAndMapAlgebra()
-
- System.out.println("All SedonaSQL DEMOs passed!")
-
- def testPredicatePushdownAndRangeJonQuery():Unit =
- {
- val sedonaConf = new SedonaConf(sparkSession.conf)
- println(sedonaConf)
-
- var polygonCsvDf = sparkSession.read.format("csv").option("delimiter",",").option("header","false").load(csvPolygonInputLocation)
- polygonCsvDf.createOrReplaceTempView("polygontable")
- polygonCsvDf.show()
- var polygonDf = sparkSession.sql("select ST_PolygonFromEnvelope(cast(polygontable._c0 as Decimal(24,20)),cast(polygontable._c1 as Decimal(24,20)), cast(polygontable._c2 as Decimal(24,20)), cast(polygontable._c3 as Decimal(24,20))) as polygonshape from polygontable")
- polygonDf.createOrReplaceTempView("polygondf")
- polygonDf.show()
-
- var pointCsvDF = sparkSession.read.format("csv").option("delimiter",",").option("header","false").load(csvPointInputLocation)
- pointCsvDF.createOrReplaceTempView("pointtable")
- pointCsvDF.show()
- var pointDf = sparkSession.sql("select ST_Point(cast(pointtable._c0 as Decimal(24,20)),cast(pointtable._c1 as Decimal(24,20))) as pointshape from pointtable")
- pointDf.createOrReplaceTempView("pointdf")
- pointDf.show()
-
- var rangeJoinDf = sparkSession.sql("select * from polygondf, pointdf where ST_Contains(polygondf.polygonshape,pointdf.pointshape) " +
- "and ST_Contains(ST_PolygonFromEnvelope(1.0,101.0,501.0,601.0), polygondf.polygonshape)")
-
- rangeJoinDf.explain()
- rangeJoinDf.show(3)
- assert (rangeJoinDf.count()==500)
- }
-
- def testDistanceJoinQuery(): Unit =
- {
- val sedonaConf = new SedonaConf(sparkSession.conf)
- println(sedonaConf)
-
- var pointCsvDF1 = sparkSession.read.format("csv").option("delimiter",",").option("header","false").load(csvPointInputLocation)
- pointCsvDF1.createOrReplaceTempView("pointtable")
- pointCsvDF1.show()
- var pointDf1 = sparkSession.sql("select ST_Point(cast(pointtable._c0 as Decimal(24,20)),cast(pointtable._c1 as Decimal(24,20))) as pointshape1 from pointtable")
- pointDf1.createOrReplaceTempView("pointdf1")
- pointDf1.show()
-
- var pointCsvDF2 = sparkSession.read.format("csv").option("delimiter",",").option("header","false").load(csvPointInputLocation)
- pointCsvDF2.createOrReplaceTempView("pointtable")
- pointCsvDF2.show()
- var pointDf2 = sparkSession.sql("select ST_Point(cast(pointtable._c0 as Decimal(24,20)),cast(pointtable._c1 as Decimal(24,20))) as pointshape2 from pointtable")
- pointDf2.createOrReplaceTempView("pointdf2")
- pointDf2.show()
-
- var distanceJoinDf = sparkSession.sql("select * from pointdf1, pointdf2 where ST_Distance(pointdf1.pointshape1,pointdf2.pointshape2) < 2")
- distanceJoinDf.explain()
- distanceJoinDf.show(10)
- assert (distanceJoinDf.count()==2998)
- }
-
- def testAggregateFunction(): Unit =
- {
- val sedonaConf = new SedonaConf(sparkSession.conf)
- println(sedonaConf)
-
- var pointCsvDF = sparkSession.read.format("csv").option("delimiter",",").option("header","false").load(csvPointInputLocation)
- pointCsvDF.createOrReplaceTempView("pointtable")
- var pointDf = sparkSession.sql("select ST_Point(cast(pointtable._c0 as Decimal(24,20)), cast(pointtable._c1 as Decimal(24,20))) as arealandmark from pointtable")
- pointDf.createOrReplaceTempView("pointdf")
- var boundary = sparkSession.sql("select ST_Envelope_Aggr(pointdf.arealandmark) from pointdf")
- val coordinates:Array[Coordinate] = new Array[Coordinate](5)
- coordinates(0) = new Coordinate(1.1,101.1)
- coordinates(1) = new Coordinate(1.1,1100.1)
- coordinates(2) = new Coordinate(1000.1,1100.1)
- coordinates(3) = new Coordinate(1000.1,101.1)
- coordinates(4) = coordinates(0)
- val geometryFactory = new GeometryFactory()
- geometryFactory.createPolygon(coordinates)
- assert(boundary.take(1)(0).get(0)==geometryFactory.createPolygon(coordinates))
- }
-
- def testShapefileConstructor(): Unit =
- {
- var spatialRDD = new SpatialRDD[Geometry]
- spatialRDD = ShapefileReader.readToGeometryRDD(sparkSession.sparkContext, shapefileInputLocation)
- var rawSpatialDf = Adapter.toDf(spatialRDD,sparkSession)
- rawSpatialDf.createOrReplaceTempView("rawSpatialDf")
- var spatialDf = sparkSession.sql("""
- | SELECT geometry, STATEFP, COUNTYFP
- | FROM rawSpatialDf
- """.stripMargin)
- spatialDf.show()
- spatialDf.printSchema()
- }
-
- def testRasterIOAndMapAlgebra(): Unit = {
- var df = sparkSession.read.format("geotiff").option("dropInvalid", true).load(rasterdatalocation)
- df.printSchema()
- df.selectExpr("image.origin as origin","ST_GeomFromWkt(image.geometry) as Geom", "image.height as height", "image.width as width", "image.data as data", "image.nBands as numBands").show()
- df = df.selectExpr(" image.data as data", "image.nBands as numBands")
- df = df.selectExpr("RS_GetBand(data, 1, numBands) as targetBand")
- df.selectExpr("RS_MultiplyFactor(targetBand, 3) as multiply").show()
- }
-}
diff --git a/examples/spark-sql/src/main/scala/SqlExample.scala b/examples/spark-sql/src/main/scala/SqlExample.scala
new file mode 100644
index 000000000..1da9248cd
--- /dev/null
+++ b/examples/spark-sql/src/main/scala/SqlExample.scala
@@ -0,0 +1,128 @@
+/**
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import Main.resourceFolder
+import org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader
+import org.apache.sedona.core.spatialRDD.SpatialRDD
+import org.apache.sedona.core.utils.SedonaConf
+import org.apache.sedona.sql.utils.Adapter
+import org.apache.spark.sql.SparkSession
+import org.locationtech.jts.geom.{Coordinate, Geometry, GeometryFactory}
+
+
+object SqlExample {
+
+ val csvPolygonInputLocation = resourceFolder + "testenvelope.csv"
+ val csvPointInputLocation = resourceFolder + "testpoint.csv"
+ val shapefileInputLocation = resourceFolder + "shapefiles/dbf"
+ val rasterdatalocation = resourceFolder + "raster/"
+
+ def testPredicatePushdownAndRangeJonQuery(sedona: SparkSession):Unit =
+ {
+ val sedonaConf = new SedonaConf(sedona.conf)
+ println(sedonaConf)
+
+ var polygonCsvDf = sedona.read.format("csv").option("delimiter",",").option("header","false").load(csvPolygonInputLocation)
+ polygonCsvDf.createOrReplaceTempView("polygontable")
+ polygonCsvDf.show()
+ var polygonDf = sedona.sql("select ST_PolygonFromEnvelope(cast(polygontable._c0 as Decimal(24,20)),cast(polygontable._c1 as Decimal(24,20)), cast(polygontable._c2 as Decimal(24,20)), cast(polygontable._c3 as Decimal(24,20))) as polygonshape from polygontable")
+ polygonDf.createOrReplaceTempView("polygondf")
+ polygonDf.show()
+
+ var pointCsvDF = sedona.read.format("csv").option("delimiter",",").option("header","false").load(csvPointInputLocation)
+ pointCsvDF.createOrReplaceTempView("pointtable")
+ pointCsvDF.show()
+ var pointDf = sedona.sql("select ST_Point(cast(pointtable._c0 as Decimal(24,20)),cast(pointtable._c1 as Decimal(24,20))) as pointshape from pointtable")
+ pointDf.createOrReplaceTempView("pointdf")
+ pointDf.show()
+
+ var rangeJoinDf = sedona.sql("select * from polygondf, pointdf where ST_Contains(polygondf.polygonshape,pointdf.pointshape) " +
+ "and ST_Contains(ST_PolygonFromEnvelope(1.0,101.0,501.0,601.0), polygondf.polygonshape)")
+
+ // Write result to GeoParquet file
+ rangeJoinDf.write.format("geoparquet").mode("overwrite").save("target/test-classes/output/join.parquet")
+ rangeJoinDf.explain()
+ rangeJoinDf.show(3)
+ assert (rangeJoinDf.count()==500)
+ }
+
+ def testDistanceJoinQuery(sedona: SparkSession): Unit =
+ {
+ val sedonaConf = new SedonaConf(sedona.conf)
+ println(sedonaConf)
+
+ var pointCsvDF1 = sedona.read.format("csv").option("delimiter",",").option("header","false").load(csvPointInputLocation)
+ pointCsvDF1.createOrReplaceTempView("pointtable")
+ pointCsvDF1.show()
+ var pointDf1 = sedona.sql("select ST_Point(cast(pointtable._c0 as Decimal(24,20)),cast(pointtable._c1 as Decimal(24,20))) as pointshape1 from pointtable")
+ pointDf1.createOrReplaceTempView("pointdf1")
+ pointDf1.show()
+
+ var pointCsvDF2 = sedona.read.format("csv").option("delimiter",",").option("header","false").load(csvPointInputLocation)
+ pointCsvDF2.createOrReplaceTempView("pointtable")
+ pointCsvDF2.show()
+ var pointDf2 = sedona.sql("select ST_Point(cast(pointtable._c0 as Decimal(24,20)),cast(pointtable._c1 as Decimal(24,20))) as pointshape2 from pointtable")
+ pointDf2.createOrReplaceTempView("pointdf2")
+ pointDf2.show()
+
+ var distanceJoinDf = sedona.sql("select * from pointdf1, pointdf2 where ST_Distance(pointdf1.pointshape1,pointdf2.pointshape2) < 2")
+ distanceJoinDf.explain()
+ distanceJoinDf.show(10)
+ assert (distanceJoinDf.count()==2998)
+ }
+
+ def testAggregateFunction(sedona: SparkSession): Unit =
+ {
+ val sedonaConf = new SedonaConf(sedona.conf)
+ println(sedonaConf)
+
+ var pointCsvDF = sedona.read.format("csv").option("delimiter",",").option("header","false").load(csvPointInputLocation)
+ pointCsvDF.createOrReplaceTempView("pointtable")
+ var pointDf = sedona.sql("select ST_Point(cast(pointtable._c0 as Decimal(24,20)), cast(pointtable._c1 as Decimal(24,20))) as arealandmark from pointtable")
+ pointDf.createOrReplaceTempView("pointdf")
+ var boundary = sedona.sql("select ST_Envelope_Aggr(pointdf.arealandmark) from pointdf")
+ val coordinates:Array[Coordinate] = new Array[Coordinate](5)
+ coordinates(0) = new Coordinate(1.1,101.1)
+ coordinates(1) = new Coordinate(1.1,1100.1)
+ coordinates(2) = new Coordinate(1000.1,1100.1)
+ coordinates(3) = new Coordinate(1000.1,101.1)
+ coordinates(4) = coordinates(0)
+ val geometryFactory = new GeometryFactory()
+ geometryFactory.createPolygon(coordinates)
+ assert(boundary.take(1)(0).get(0)==geometryFactory.createPolygon(coordinates))
+ }
+
+ def testShapefileConstructor(sedona: SparkSession): Unit =
+ {
+ var spatialRDD = new SpatialRDD[Geometry]
+ spatialRDD = ShapefileReader.readToGeometryRDD(sedona.sparkContext, shapefileInputLocation)
+ var rawSpatialDf = Adapter.toDf(spatialRDD,sedona)
+ rawSpatialDf.createOrReplaceTempView("rawSpatialDf")
+ var spatialDf = sedona.sql("""
+ | SELECT geometry, STATEFP, COUNTYFP
+ | FROM rawSpatialDf
+ """.stripMargin)
+ spatialDf.show()
+ spatialDf.printSchema()
+ }
+
+ def testRasterIOAndMapAlgebra(sedona: SparkSession): Unit = {
+ var df = sedona.read.format("binaryFile").option("dropInvalid", true).load(rasterdatalocation).selectExpr("RS_FromGeoTiff(content) as raster", "path")
+ df.printSchema()
+ df.show()
+ df.selectExpr("RS_Metadata(raster) as metadata", "RS_GeoReference(raster) as georef", "RS_NumBands(raster) as numBands").show(false)
+ df.selectExpr("RS_AddBand(raster, raster, 1) as raster_extraband").show()
+ }
+}
diff --git a/examples/spark-sql/src/main/scala/VizExample.scala b/examples/spark-sql/src/main/scala/VizExample.scala
new file mode 100644
index 000000000..d0e1e498d
--- /dev/null
+++ b/examples/spark-sql/src/main/scala/VizExample.scala
@@ -0,0 +1,139 @@
+/**
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import Main.resourceFolder
+import org.apache.sedona.common.enums.FileDataSplitter
+import org.apache.sedona.core.enums.{GridType, IndexType}
+import org.apache.sedona.core.spatialOperator.JoinQuery
+import org.apache.sedona.core.spatialRDD.{PointRDD, PolygonRDD, RectangleRDD}
+import org.apache.sedona.viz.core.{ImageGenerator, ImageSerializableWrapper, RasterOverlayOperator}
+import org.apache.sedona.viz.extension.visualizationEffect.{ChoroplethMap, HeatMap, ScatterPlot}
+import org.apache.sedona.viz.utils.ImageType
+import org.apache.spark.sql.SparkSession
+import org.locationtech.jts.geom.Envelope
+
+import java.awt.Color
+
+
+object VizExample {
+
+ val demoOutputPath = "target/demo"
+
+ val scatterPlotOutputPath = System.getProperty("user.dir") + "/" + demoOutputPath + "/scatterplot"
+ val heatMapOutputPath = System.getProperty("user.dir") + "/" + demoOutputPath + "/heatmap"
+ val choroplethMapOutputPath = System.getProperty("user.dir") + "/" + demoOutputPath + "/choroplethmap"
+ val parallelFilterRenderOutputPath = System.getProperty("user.dir") + "/" + demoOutputPath + "/parallelfilterrender-heatmap"
+ val sqlApiOutputPath = System.getProperty("user.dir") + "/" + demoOutputPath + "/sql-heatmap"
+
+ val PointInputLocation = resourceFolder + "arealm.csv"
+ val PointOffset = 0
+ val PointSplitter = FileDataSplitter.CSV
+ val PointNumPartitions = 5
+
+ val RectangleInputLocation = resourceFolder + "zcta510.csv"
+ val RectangleSplitter = FileDataSplitter.CSV
+ val RectangleNumPartitions = 5
+
+ val PolygonInputLocation = resourceFolder + "primaryroads-polygon.csv"
+ val PolygonSplitter = FileDataSplitter.CSV
+ val PolygonNumPartitions = 5
+ val USMainLandBoundary = new Envelope(-126.790180, -64.630926, 24.863836, 50.000)
+
+ def buildScatterPlot(sedona: SparkSession): Boolean = {
+ val spatialRDD = new PolygonRDD(sedona.sparkContext, PolygonInputLocation, PolygonSplitter, false, PolygonNumPartitions)
+ var visualizationOperator = new ScatterPlot(1000, 600, USMainLandBoundary, false)
+ visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.GREEN, true)
+ visualizationOperator.Visualize(sedona.sparkContext, spatialRDD)
+ var imageGenerator = new ImageGenerator
+ imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.rasterImage, scatterPlotOutputPath, ImageType.PNG)
+ true
+ }
+
+ def buildHeatMap(sedona: SparkSession): Boolean = {
+ val spatialRDD = new RectangleRDD(sedona.sparkContext, RectangleInputLocation, RectangleSplitter, false, RectangleNumPartitions)
+ val visualizationOperator = new HeatMap(1000, 600, USMainLandBoundary, false, 2)
+ visualizationOperator.Visualize(sedona.sparkContext, spatialRDD)
+ val imageGenerator = new ImageGenerator
+ imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.rasterImage, heatMapOutputPath, ImageType.PNG)
+ true
+ }
+
+
+ def buildChoroplethMap(sedona: SparkSession): Boolean = {
+ val spatialRDD = new PointRDD(sedona.sparkContext, PointInputLocation, PointOffset, PointSplitter, false, PointNumPartitions)
+ val queryRDD = new PolygonRDD(sedona.sparkContext, PolygonInputLocation, PolygonSplitter, false, PolygonNumPartitions)
+ spatialRDD.spatialPartitioning(GridType.KDBTREE)
+ queryRDD.spatialPartitioning(spatialRDD.getPartitioner)
+ spatialRDD.buildIndex(IndexType.QUADTREE, true)
+ val joinResult = JoinQuery.SpatialJoinQueryCountByKey(spatialRDD, queryRDD, true, false)
+ val visualizationOperator = new ChoroplethMap(1000, 600, USMainLandBoundary, false)
+ visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.RED, true)
+ visualizationOperator.Visualize(sedona.sparkContext, joinResult)
+ val frontImage = new ScatterPlot(1000, 600, USMainLandBoundary, false)
+ frontImage.CustomizeColor(0, 0, 0, 255, Color.GREEN, true)
+ frontImage.Visualize(sedona.sparkContext, queryRDD)
+ val overlayOperator = new RasterOverlayOperator(visualizationOperator.rasterImage)
+ overlayOperator.JoinImage(frontImage.rasterImage)
+ val imageGenerator = new ImageGenerator
+ imageGenerator.SaveRasterImageAsLocalFile(overlayOperator.backRasterImage, choroplethMapOutputPath, ImageType.PNG)
+ true
+ }
+
+ def parallelFilterRenderNoStitch(sedona: SparkSession): Boolean = {
+ val spatialRDD = new RectangleRDD(sedona.sparkContext, RectangleInputLocation, RectangleSplitter, false, RectangleNumPartitions)
+ val visualizationOperator = new HeatMap(1000, 600, USMainLandBoundary, false, 2, 4, 4, true, true)
+ visualizationOperator.Visualize(sedona.sparkContext, spatialRDD)
+ val imageGenerator = new ImageGenerator
+ imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.distributedRasterImage, parallelFilterRenderOutputPath, ImageType.PNG)
+ true
+ }
+
+ def sqlApiVisualization(sedona: SparkSession): Boolean = {
+ var pointDf = sedona.read.format("csv").option("delimiter", ",").option("header", "false").load(PointInputLocation)
+ pointDf.selectExpr("ST_Point(cast(_c0 as Decimal(24,20)),cast(_c1 as Decimal(24,20))) as shape")
+ .filter("ST_Contains(ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000),shape)").createOrReplaceTempView("pointtable")
+ sedona.sql(
+ """
+ |CREATE OR REPLACE TEMP VIEW pixels AS
+ |SELECT pixel, shape FROM pointtable
+ |LATERAL VIEW Explode(ST_Pixelize(shape, 256, 256, ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000))) AS pixel
+ """.stripMargin)
+ sedona.sql(
+ """
+ |CREATE OR REPLACE TEMP VIEW pixelaggregates AS
+ |SELECT pixel, count(*) as weight
+ |FROM pixels
+ |GROUP BY pixel
+ """.stripMargin)
+ sedona.sql(
+ """
+ |CREATE OR REPLACE TEMP VIEW images AS
+ |SELECT ST_Render(pixel, ST_Colorize(weight, (SELECT max(weight) FROM pixelaggregates), 'red')) AS image
+ |FROM pixelaggregates
+ """.stripMargin)
+ var image = sedona.table("images").take(1)(0)(0).asInstanceOf[ImageSerializableWrapper].getImage
+ var imageGenerator = new ImageGenerator
+ imageGenerator.SaveRasterImageAsLocalFile(image, sqlApiOutputPath, ImageType.PNG)
+ sedona.sql(
+ """
+ |CREATE OR REPLACE TEMP VIEW imagestring AS
+ |SELECT ST_EncodeImage(image)
+ |FROM images
+ """.stripMargin)
+ sedona.table("imagestring").show()
+ true
+ }
+
+}
diff --git a/examples/spark-viz/src/test/resources/arealm.csv b/examples/spark-sql/src/test/resources/arealm.csv
similarity index 100%
rename from examples/spark-viz/src/test/resources/arealm.csv
rename to examples/spark-sql/src/test/resources/arealm.csv
diff --git a/examples/spark-rdd-colocation-mining/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.cst b/examples/spark-sql/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.cst
similarity index 100%
rename from examples/spark-rdd-colocation-mining/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.cst
rename to examples/spark-sql/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.cst
diff --git a/examples/spark-rdd-colocation-mining/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.dbf b/examples/spark-sql/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.dbf
similarity index 100%
rename from examples/spark-rdd-colocation-mining/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.dbf
rename to examples/spark-sql/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.dbf
diff --git a/examples/spark-rdd-colocation-mining/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.prj b/examples/spark-sql/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.prj
similarity index 100%
rename from examples/spark-rdd-colocation-mining/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.prj
rename to examples/spark-sql/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.prj
diff --git a/examples/spark-rdd-colocation-mining/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.shp b/examples/spark-sql/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.shp
similarity index 100%
rename from examples/spark-rdd-colocation-mining/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.shp
rename to examples/spark-sql/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.shp
diff --git a/examples/spark-rdd-colocation-mining/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.shx b/examples/spark-sql/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.shx
similarity index 100%
rename from examples/spark-rdd-colocation-mining/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.shx
rename to examples/spark-sql/src/test/resources/nyc-area-landmark-shapefile/nyu_2451_34514.shx
diff --git a/examples/spark-rdd-colocation-mining/src/test/resources/nyc-area-landmark-shapefile/wfsrequest.txt b/examples/spark-sql/src/test/resources/nyc-area-landmark-shapefile/wfsrequest.txt
similarity index 100%
rename from examples/spark-rdd-colocation-mining/src/test/resources/nyc-area-landmark-shapefile/wfsrequest.txt
rename to examples/spark-sql/src/test/resources/nyc-area-landmark-shapefile/wfsrequest.txt
diff --git a/examples/spark-viz/src/test/resources/primaryroads-polygon.csv b/examples/spark-sql/src/test/resources/primaryroads-polygon.csv
similarity index 100%
rename from examples/spark-viz/src/test/resources/primaryroads-polygon.csv
rename to examples/spark-sql/src/test/resources/primaryroads-polygon.csv
diff --git a/examples/spark-sql/src/test/resources/raster/FAA_UTM18N_NAD83.tif b/examples/spark-sql/src/test/resources/raster/FAA_UTM18N_NAD83.tif
new file mode 100644
index 000000000..a0a38aa62
Binary files /dev/null and b/examples/spark-sql/src/test/resources/raster/FAA_UTM18N_NAD83.tif differ
diff --git a/examples/spark-sql/src/test/resources/raster/test2.tiff b/examples/spark-sql/src/test/resources/raster/test2.tiff
deleted file mode 100644
index bebd68232..000000000
Binary files a/examples/spark-sql/src/test/resources/raster/test2.tiff and /dev/null differ
diff --git a/examples/spark-rdd-colocation-mining/src/test/resources/yellow_tripdata_2009-01-subset.csv b/examples/spark-sql/src/test/resources/yellow_tripdata_2009-01-subset.csv
similarity index 100%
rename from examples/spark-rdd-colocation-mining/src/test/resources/yellow_tripdata_2009-01-subset.csv
rename to examples/spark-sql/src/test/resources/yellow_tripdata_2009-01-subset.csv
diff --git a/examples/spark-viz/src/test/resources/zcta510.csv b/examples/spark-sql/src/test/resources/zcta510.csv
similarity index 100%
rename from examples/spark-viz/src/test/resources/zcta510.csv
rename to examples/spark-sql/src/test/resources/zcta510.csv
diff --git a/examples/spark-viz/.gitignore b/examples/spark-viz/.gitignore
deleted file mode 100644
index 981c582aa..000000000
--- a/examples/spark-viz/.gitignore
+++ /dev/null
@@ -1,2 +0,0 @@
-/.idea/
-/target/
\ No newline at end of file
diff --git a/examples/spark-viz/build.sbt b/examples/spark-viz/build.sbt
deleted file mode 100644
index 7f6d15eb9..000000000
--- a/examples/spark-viz/build.sbt
+++ /dev/null
@@ -1,85 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-import sbt.Keys.{libraryDependencies, version}
-
-
-lazy val root = (project in file(".")).
- settings(
- name := "SedonaVizTemplate",
-
- version := "0.1.0",
-
- scalaVersion := "2.12.11",
-
- organization := "org.apache.sedona",
-
- publishMavenStyle := true
- )
-
-val SparkVersion = "3.4.1"
-
-val SparkCompatibleVersion = "3.4"
-
-val HadoopVersion = "2.7.2"
-
-val SedonaVersion = "1.5.0"
-
-val ScalaCompatibleVersion = "2.12"
-
-// Change the dependency scope to "provided" when you run "sbt assembly"
-val dependencyScope = "compile"
-
-val geotoolsVersion = "1.5.0-28.2"
-
-//val jacksonVersion = "2.10.0"
-
-logLevel := Level.Warn
-
-logLevel in assembly := Level.Error
-
-libraryDependencies ++= Seq(
-// "com.fasterxml.jackson.core" % "jackson-databind" % jacksonVersion % dependencyScope,
-// "com.fasterxml.jackson.core" % "jackson-core" % jacksonVersion % dependencyScope,
- "org.apache.spark" %% "spark-core" % SparkVersion % dependencyScope exclude("org.apache.hadoop", "*"),
- "org.apache.spark" %% "spark-sql" % SparkVersion % dependencyScope exclude("org.apache.hadoop", "*"),
- "org.apache.hadoop" % "hadoop-mapreduce-client-core" % HadoopVersion % dependencyScope,
- "org.apache.hadoop" % "hadoop-common" % HadoopVersion % dependencyScope,
- "org.apache.hadoop" % "hadoop-hdfs" % HadoopVersion % dependencyScope,
- "org.apache.sedona" % "sedona-spark-".concat(SparkCompatibleVersion).concat("_").concat(ScalaCompatibleVersion) % SedonaVersion changing(),
- // The following GeoTools packages are only required if you need CRS transformation. Under GNU Lesser General Public License (LGPL) license
- "org.datasyslab" % "geotools-wrapper" % geotoolsVersion % "compile"
-)
-
-assemblyMergeStrategy in assembly := {
- case PathList("org.apache.sedona", "sedona-core", xs@_*) => MergeStrategy.first
- case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
- case PathList("META-INF","services",xs @ _*) => MergeStrategy.filterDistinctLines
- case path if path.endsWith(".SF") => MergeStrategy.discard
- case path if path.endsWith(".DSA") => MergeStrategy.discard
- case path if path.endsWith(".RSA") => MergeStrategy.discard
- case _ => MergeStrategy.first
-}
-
-resolvers ++= Seq(
- "Maven Central" at "https://repo.maven.apache.org/maven2/",
- "Open Source Geospatial Foundation Repository" at "https://repo.osgeo.org/repository/release/",
- "Apache Software Foundation Snapshots" at "https://repository.apache.org/content/groups/snapshots",
- "Java.net repository" at "https://download.java.net/maven/2"
-)
\ No newline at end of file
diff --git a/examples/spark-viz/project/.gitignore b/examples/spark-viz/project/.gitignore
deleted file mode 100644
index 6a60ed534..000000000
--- a/examples/spark-viz/project/.gitignore
+++ /dev/null
@@ -1,2 +0,0 @@
-/target/
-/project/
diff --git a/examples/spark-viz/project/build.properties b/examples/spark-viz/project/build.properties
deleted file mode 100644
index 215ddd2b3..000000000
--- a/examples/spark-viz/project/build.properties
+++ /dev/null
@@ -1 +0,0 @@
-sbt.version = 1.5.5
\ No newline at end of file
diff --git a/examples/spark-viz/project/plugins.sbt b/examples/spark-viz/project/plugins.sbt
deleted file mode 100644
index 415991121..000000000
--- a/examples/spark-viz/project/plugins.sbt
+++ /dev/null
@@ -1 +0,0 @@
-addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "1.1.0")
\ No newline at end of file
diff --git a/examples/spark-viz/src/main/scala/ScalaExample.scala b/examples/spark-viz/src/main/scala/ScalaExample.scala
deleted file mode 100644
index 4c78428b2..000000000
--- a/examples/spark-viz/src/main/scala/ScalaExample.scala
+++ /dev/null
@@ -1,223 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-import org.apache.log4j.{Level, Logger}
-import org.apache.sedona.core.enums.{FileDataSplitter, GridType, IndexType}
-import org.apache.sedona.core.spatialOperator.JoinQuery
-import org.apache.sedona.core.spatialRDD.{PointRDD, PolygonRDD, RectangleRDD}
-import org.apache.sedona.sql.utils.SedonaSQLRegistrator
-import org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
-import org.apache.sedona.viz.core.{ImageGenerator, ImageSerializableWrapper, RasterOverlayOperator}
-import org.apache.sedona.viz.extension.visualizationEffect.{ChoroplethMap, HeatMap, ScatterPlot}
-import org.apache.sedona.viz.sql.utils.SedonaVizRegistrator
-import org.apache.sedona.viz.utils.{ColorizeOption, ImageType}
-import org.apache.spark.serializer.KryoSerializer
-import org.apache.spark.sql.SQLContext
-import org.apache.spark.storage.StorageLevel
-import org.apache.spark.{SparkConf, SparkContext}
-import org.locationtech.jts.geom.Envelope
-
-import java.awt.Color
-import java.io.FileInputStream
-import java.util.Properties
-
-
-object ScalaExample extends App{
- Logger.getLogger("org").setLevel(Level.WARN)
- Logger.getLogger("akka").setLevel(Level.WARN)
-
- val sparkConf = new SparkConf().setAppName("SedonaVizDemo").set("spark.serializer", classOf[KryoSerializer].getName)
- .set("spark.kryo.registrator", classOf[SedonaVizKryoRegistrator].getName)
- .setMaster("local[*]")
- val sparkContext = new SparkContext(sparkConf)
-
- val prop = new Properties()
- val resourcePath = "src/test/resources/"
- val demoOutputPath = "target/demo"
- var ConfFile = new FileInputStream(resourcePath + "babylon.point.properties")
- prop.load(ConfFile)
- val scatterPlotOutputPath = System.getProperty("user.dir") + "/" + demoOutputPath + "/scatterplot"
- val heatMapOutputPath = System.getProperty("user.dir") + "/" + demoOutputPath + "/heatmap"
- val choroplethMapOutputPath = System.getProperty("user.dir") + "/" + demoOutputPath + "/choroplethmap"
- val parallelFilterRenderOutputPath = System.getProperty("user.dir") + "/" + demoOutputPath + "/parallelfilterrender-heatmap"
- val earthdataScatterPlotOutputPath = System.getProperty("user.dir") + "/" + demoOutputPath + "/earthdatascatterplot"
- val sqlApiOutputPath = System.getProperty("user.dir") + "/" + demoOutputPath + "/sql-heatmap"
-
- val PointInputLocation = "file://" + System.getProperty("user.dir") + "/" + resourcePath + prop.getProperty("inputLocation")
- val PointOffset = prop.getProperty("offset").toInt
- val PointSplitter = FileDataSplitter.getFileDataSplitter(prop.getProperty("splitter"))
- val PointNumPartitions = prop.getProperty("numPartitions").toInt
- ConfFile = new FileInputStream(resourcePath + "babylon.rectangle.properties")
- prop.load(ConfFile)
- val RectangleInputLocation = "file://" + System.getProperty("user.dir") + "/" + resourcePath + prop.getProperty("inputLocation")
- val RectangleOffset = prop.getProperty("offset").toInt
- val RectangleSplitter = FileDataSplitter.getFileDataSplitter(prop.getProperty("splitter"))
- val RectangleNumPartitions = prop.getProperty("numPartitions").toInt
- ConfFile = new FileInputStream(resourcePath + "babylon.polygon.properties")
- prop.load(ConfFile)
- val PolygonInputLocation = "file://" + System.getProperty("user.dir") + "/" + resourcePath + prop.getProperty("inputLocation")
- val PolygonOffset = prop.getProperty("offset").toInt
- val PolygonSplitter = FileDataSplitter.getFileDataSplitter(prop.getProperty("splitter"))
- val PolygonNumPartitions = prop.getProperty("numPartitions").toInt
- val USMainLandBoundary = new Envelope(-126.790180, -64.630926, 24.863836, 50.000)
-// val earthdataInputLocation = System.getProperty("user.dir") + "/src/test/resources/modis/modis.csv"
-// val earthdataNumPartitions = 5
-// val HDFIncrement = 5
-// val HDFOffset = 2
-// val HDFRootGroupName = "MOD_Swath_LST"
-// val HDFDataVariableName = "LST"
-// val HDFDataVariableList = Array("LST", "QC", "Error_LST", "Emis_31", "Emis_32")
-// val HDFswitchXY = true
-// val urlPrefix = System.getProperty("user.dir") + "/src/test/resources/modis/"
-
- if (buildScatterPlot(scatterPlotOutputPath)
- && buildHeatMap(heatMapOutputPath)
- && buildChoroplethMap(choroplethMapOutputPath)
- && parallelFilterRenderNoStitch(parallelFilterRenderOutputPath)
-// && earthdataVisualization(earthdataScatterPlotOutputPath)
- && sqlApiVisualization(sqlApiOutputPath))
- System.out.println("All SedonaViz Demos have passed.")
- else System.out.println("SedonaViz Demos failed.")
-
- /**
- * Builds the scatter plot.
- *
- * @param outputPath the output path
- * @return true, if successful
- */
- def buildScatterPlot(outputPath: String): Boolean = {
- val spatialRDD = new PolygonRDD(sparkContext, PolygonInputLocation, PolygonSplitter, false, PolygonNumPartitions)
- var visualizationOperator = new ScatterPlot(1000, 600, USMainLandBoundary, false)
- visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.GREEN, true)
- visualizationOperator.Visualize(sparkContext, spatialRDD)
- var imageGenerator = new ImageGenerator
- imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.rasterImage, outputPath, ImageType.PNG)
- true
- }
-
- /**
- * Builds the heat map.
- *
- * @param outputPath the output path
- * @return true, if successful
- */
- def buildHeatMap(outputPath: String): Boolean = {
- val spatialRDD = new RectangleRDD(sparkContext, RectangleInputLocation, RectangleSplitter, false, RectangleNumPartitions, StorageLevel.MEMORY_ONLY)
- val visualizationOperator = new HeatMap(1000, 600, USMainLandBoundary, false, 2)
- visualizationOperator.Visualize(sparkContext, spatialRDD)
- val imageGenerator = new ImageGenerator
- imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.rasterImage, outputPath, ImageType.PNG)
- true
- }
-
- /**
- * Builds the choropleth map.
- *
- * @param outputPath the output path
- * @return true, if successful
- */
- def buildChoroplethMap(outputPath: String): Boolean = {
- val spatialRDD = new PointRDD(sparkContext, PointInputLocation, PointOffset, PointSplitter, false, PointNumPartitions, StorageLevel.MEMORY_ONLY)
- val queryRDD = new PolygonRDD(sparkContext, PolygonInputLocation, PolygonSplitter, false, PolygonNumPartitions, StorageLevel.MEMORY_ONLY)
- spatialRDD.spatialPartitioning(GridType.KDBTREE)
- queryRDD.spatialPartitioning(spatialRDD.getPartitioner)
- spatialRDD.buildIndex(IndexType.QUADTREE, true)
- val joinResult = JoinQuery.SpatialJoinQueryCountByKey(spatialRDD, queryRDD, true, false)
- val visualizationOperator = new ChoroplethMap(1000, 600, USMainLandBoundary, false)
- visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.RED, true)
- visualizationOperator.Visualize(sparkContext, joinResult)
- val frontImage = new ScatterPlot(1000, 600, USMainLandBoundary, false)
- frontImage.CustomizeColor(0, 0, 0, 255, Color.GREEN, true)
- frontImage.Visualize(sparkContext, queryRDD)
- val overlayOperator = new RasterOverlayOperator(visualizationOperator.rasterImage)
- overlayOperator.JoinImage(frontImage.rasterImage)
- val imageGenerator = new ImageGenerator
- imageGenerator.SaveRasterImageAsLocalFile(overlayOperator.backRasterImage, outputPath, ImageType.PNG)
- true
- }
-
- /**
- * Parallel filter render no stitch.
- *
- * @param outputPath the output path
- * @return true, if successful
- */
- def parallelFilterRenderNoStitch(outputPath: String): Boolean = {
- val spatialRDD = new RectangleRDD(sparkContext, RectangleInputLocation, RectangleSplitter, false, RectangleNumPartitions, StorageLevel.MEMORY_ONLY)
- val visualizationOperator = new HeatMap(1000, 600, USMainLandBoundary, false, 2, 4, 4, true, true)
- visualizationOperator.Visualize(sparkContext, spatialRDD)
- val imageGenerator = new ImageGenerator
- imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.distributedRasterImage, outputPath, ImageType.PNG)
- true
- }
-
-// def earthdataVisualization(outputPath: String): Boolean = {
-// val earthdataHDFPoint = new EarthdataHDFPointMapper(HDFIncrement, HDFOffset, HDFRootGroupName,
-// HDFDataVariableList, HDFDataVariableName, HDFswitchXY, urlPrefix)
-// val spatialRDD = new PointRDD(sparkContext, earthdataInputLocation, earthdataNumPartitions, earthdataHDFPoint, StorageLevel.MEMORY_ONLY)
-// val visualizationOperator = new ScatterPlot(1000, 600, spatialRDD.boundaryEnvelope, ColorizeOption.EARTHOBSERVATION, false, false)
-// visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.BLUE, true)
-// visualizationOperator.Visualize(sparkContext, spatialRDD)
-// val imageGenerator = new ImageGenerator
-// imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.rasterImage, outputPath, ImageType.PNG)
-// true
-// }
-
- def sqlApiVisualization(outputPath: String): Boolean = {
- val sqlContext = new SQLContext(sparkContext)
- val spark = sqlContext.sparkSession
- SedonaSQLRegistrator.registerAll(spark)
- SedonaVizRegistrator.registerAll(spark)
- var pointDf = spark.read.format("csv").option("delimiter", ",").option("header", "false").load(PointInputLocation)
- pointDf.selectExpr("ST_Point(cast(_c0 as Decimal(24,20)),cast(_c1 as Decimal(24,20))) as shape")
- .filter("ST_Contains(ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000),shape)").createOrReplaceTempView("pointtable")
- spark.sql(
- """
- |CREATE OR REPLACE TEMP VIEW pixels AS
- |SELECT pixel, shape FROM pointtable
- |LATERAL VIEW Explode(ST_Pixelize(shape, 256, 256, ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000))) AS pixel
- """.stripMargin)
- spark.sql(
- """
- |CREATE OR REPLACE TEMP VIEW pixelaggregates AS
- |SELECT pixel, count(*) as weight
- |FROM pixels
- |GROUP BY pixel
- """.stripMargin)
- spark.sql(
- """
- |CREATE OR REPLACE TEMP VIEW images AS
- |SELECT ST_Render(pixel, ST_Colorize(weight, (SELECT max(weight) FROM pixelaggregates), 'red')) AS image
- |FROM pixelaggregates
- """.stripMargin)
- var image = spark.table("images").take(1)(0)(0).asInstanceOf[ImageSerializableWrapper].getImage
- var imageGenerator = new ImageGenerator
- imageGenerator.SaveRasterImageAsLocalFile(image, outputPath, ImageType.PNG)
- spark.sql(
- """
- |CREATE OR REPLACE TEMP VIEW imagestring AS
- |SELECT ST_EncodeImage(image)
- |FROM images
- """.stripMargin)
- spark.table("imagestring").show()
- true
- }
-
-
-}
diff --git a/examples/spark-viz/src/test/resources/babylon.point.properties b/examples/spark-viz/src/test/resources/babylon.point.properties
deleted file mode 100644
index 3fb21a6d2..000000000
--- a/examples/spark-viz/src/test/resources/babylon.point.properties
+++ /dev/null
@@ -1,9 +0,0 @@
-inputLocation=arealm.csv
-queryWindowSet=zcta510-small.csv
-offset=0
-splitter=csv
-gridType=rtree
-indexType=rtree
-numPartitions=5
-distance=0.01
-queryPolygonSet=primaryroads-polygon.csv
\ No newline at end of file
diff --git a/examples/spark-viz/src/test/resources/babylon.polygon.properties b/examples/spark-viz/src/test/resources/babylon.polygon.properties
deleted file mode 100644
index ef7639671..000000000
--- a/examples/spark-viz/src/test/resources/babylon.polygon.properties
+++ /dev/null
@@ -1,9 +0,0 @@
-inputLocation=primaryroads-polygon.csv
-queryWindowSet=zcta510-small.csv
-offset=0
-splitter=csv
-gridType=rtree
-indexType=rtree
-numPartitions=5
-distance=0.01
-queryPolygonSet=primaryroads-polygon.csv
\ No newline at end of file
diff --git a/examples/spark-viz/src/test/resources/babylon.rectangle.properties b/examples/spark-viz/src/test/resources/babylon.rectangle.properties
deleted file mode 100644
index 54a283d0a..000000000
--- a/examples/spark-viz/src/test/resources/babylon.rectangle.properties
+++ /dev/null
@@ -1,9 +0,0 @@
-inputLocation=zcta510.csv
-queryWindowSet=zcta510-small.csv
-offset=0
-splitter=csv
-gridType=rtree
-indexType=rtree
-numPartitions=5
-distance=0.01
-queryPolygonSet=primaryroads-polygon.csv
\ No newline at end of file