You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@wayang.apache.org by al...@apache.org on 2026/05/26 11:54:37 UTC

(wayang-website) branch main updated: adds spatial blogpost and authors (#124)

This is an automated email from the ASF dual-hosted git repository.

novatechflow pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/wayang-website.git


The following commit(s) were added to refs/heads/main by this push:
     new 3404f0c4 adds spatial blogpost and authors (#124)
3404f0c4 is described below

commit 3404f0c4bcd0a51007c1d5b54ad6a664fb7ac66e
Author: treyfel <58...@users.noreply.github.com>
AuthorDate: Tue May 26 13:54:31 2026 +0200

    adds spatial blogpost and authors (#124)
    
    Co-authored-by: Maximilian Speer <Ma...@student.hpi.uni-potsdam.de>
    Co-authored-by: Anton Persitzky <An...@student.hpi.uni-potsdam.de>
---
 blog/2026-05-19-spatial-wayang.md                  | 123 +++++++++++++++++++++
 blog/authors.yml                                   |  15 +++
 .../architecture/wayang-spatial-architecture.png   | Bin 0 -> 114643 bytes
 static/img/benchmarks/spatial-bench1.png           | Bin 0 -> 21154 bytes
 static/img/benchmarks/spatial-bench2.png           | Bin 0 -> 65750 bytes
 static/img/benchmarks/spatial-bench3.png           | Bin 0 -> 47471 bytes
 static/img/benchmarks/spatial-bench4.png           | Bin 0 -> 40447 bytes
 7 files changed, 138 insertions(+)

diff --git a/blog/2026-05-19-spatial-wayang.md b/blog/2026-05-19-spatial-wayang.md
new file mode 100644
index 00000000..1454b5e3
--- /dev/null
+++ b/blog/2026-05-19-spatial-wayang.md
@@ -0,0 +1,123 @@
+---
+slug: spatial-wayang
+title: Bringing Spatial Data Processing to Apache Wayang
+authors: [max.speer, anton.persitzky, felix.treykorn]
+---
+
+# Bringing Spatial Data Processing to Apache Wayang
+
+Apache Wayang already enables a large variety of data processing workflows, ranging from basic data retrieval and filtering to complex ML tasks. However, there are additional areas that could benefit from its platform flexibility. As part of a university project, we implemented spatial operators in Apache Wayang. This enables the execution of workflows using geospatial data, including spatial join and filter operations. Since execution times can differ significantly depending on the chos [...]
+
+---
+
+## Implementation
+
+Workflows including spatial operators are not the main task in Apache Wayang. Therefore, we chose to add spatial support as a plugin that can be enabled separately.
+
+The two new operators we added are SpatialFilter and SpatialJoin, since many spatial workflows primarily rely on these two operations.
+
+To support multiple data sources and geometry formats, we introduced an internal geometry representation, SpatialGeometry, that enables translation between formats. This allows geometries stored as WKT, WKB, and GeoJSON to be read and, if necessary, converted to the type expected by the consuming operator.
+
+For the execution of spatial jobs, we currently support Java, PostgreSQL, and Spark as backends. The Spark implementation uses Apache Sedona, a well-established library for distributed processing of spatial data. The Java implementation is based on JTS, and the PostgreSQL platform uses PostGIS for spatial operations.
+
+```java
+WayangContext wayangContext = new WayangContext(configuration)
+                .withPlugin(Java.basicPlugin())
+                .withPlugin(Postgres.plugin())
+                .withPlugin(Spatial.plugin());
+
+final Collection<Long> outputcount = builder
+       .readTable(new PostgresTableSource(tableName, "ST_AsText(geom)"))
+       .spatialFilter(
+               (input -> WayangGeometry.fromStringInput(input.getString(0))),
+               SpatialPredicate.INTERSECTS,
+               queryGeometry
+       ).withSqlGeometryColumnName("geom")
+       .withTargetPlatform(Postgres.platform())
+       .count()
+       .collect();
+
+System.out.println("Spatial Filter Postgres Result Size: " + outputcount);
+```
+
+---
+
+## Why Spatial Is a Plugin, Not a Platform
+
+In the current Wayang architecture, a platform is responsible for runtime ownership: it defines a Platform implementation, provides an executor factory, defines channel conversions, and integrates cost models. This can be seen clearly in modules such as Java (JavaPlatform + JavaExecutor) and PostgreSQL (PostgresPlatform via JdbcPlatformTemplate + JdbcExecutor).
+
+The spatial extension is intentionally different. It does not introduce a new Platform subclass and does not provide its own executor. Instead, wayang-spatial is implemented as a plugin (Spatial.java) that contributes operators, mappings for those operators, and specifies required platforms.
+
+The logical spatial operators (`SpatialFilterOperator`, `SpatialJoinOperator`, `GeoJsonFileSource`) still live in `wayang-basic` and are included in the Java-Scala API. The spatial plugin extends the system by adding platform-specific execution operators and transformation rules that map Wayang operators to their platform implementations.
+
+At registration time, `WayangContext.withPlugin(...)` calls `plugin.configure(configuration)`. This whitelists the required platforms and spatial mappings. During optimization, mappings such as `SpatialFilterMapping` and `SpatialJoinMapping` rewrite logical operators into execution operators such as:
+
+- `JavaSpatialFilterOperator` / `JavaSpatialJoinOperator`
+- `SparkSpatialFilterOperator` / `SparkSpatialJoinOperator`
+- `PostgresSpatialFilterOperator` / `PostgresSpatialJoinOperator`
+
+Execution is then handled by the existing platform runtimes:
+
+- Java operators are executed by `JavaExecutor`
+- Spark operators are executed by `SparkExecutor`
+- PostgreSQL operators are executed through the JDBC runtime (`JdbcExecutor`)
+
+In practice, this is why applications have to register both a platform plugin (for general operators and channels) and a spatial plugin (for spatial mappings), for example `Java.basicPlugin()` + `Spatial.javaPlugin()` or `Spark.basicPlugin()` + `Spatial.sparkPlugin()`.
+
+This design provides three concrete advantages:
+
+- No duplicate runtime stack, since executors, channels, and cost models remain centralized in the platforms
+- Clean multi-platform support for the same spatial semantics
+- Lower maintenance overhead, because spatial development efforts can focus on operator logic and mappings rather than recreating infrastructure
+
+Spatial support in Wayang is therefore best understood as a cross-platform capability extension: it contributes spatial semantics and translation, while execution remains fully owned by the underlying platforms.
+
+<div style={{textAlign: 'center'}}>
+  <img width="90%" alt="wayang-spatial architecture" src="/img/architecture/wayang-spatial-architecture.png" />
+</div>
+
+---
+
+## Benchmarks
+
+Below we show scenarios where it is advantageous to be able to freely choose the execution depending on the use case. The following benchmarks are not intended to be exhaustive tests of our new operators but rather to highlight how using the platform independence of Apache Wayang can speed up spatial jobs. We executed the benchmarks on an HPC cluster with reproducible Spark cluster configurations.
+
+<div style={{textAlign: 'center'}}>
+  <img width="75%" alt="Spatial benchmark 1" src="/img/benchmarks/spatial-bench1.png" />
+</div>
+
+- **Job 1:** Spatial Join of parks (~44k) and lakes (~140k) in Germany with spatial predicate CONTAINS, no index on Postgres tables, Spark cluster with 4 nodes
+- **Job 2:** Spatial Join of parks (~44k) and lakes (~140k) in Germany with spatial predicate CONTAINS, spatial index on Postgres tables, Spark cluster with 4 nodes
+- **Job 3:** Spatial Join of two synthetic datasets containing boxes (100k and 1M) with spatial predicate INTERSECTS, spatial index on Postgres tables, Spark cluster with 8 nodes
+
+The above figure shows that there are use cases in which each of the currently supported platforms performs best. For small datasets, Java performs best if there is no spatial index available for Postgres; otherwise, Postgres outperforms Java. Due to the Spark overhead, Sedona running on a Spark cluster only gains an advantage in jobs with large datasets, as seen in Job 3.
+
+This is also evident in the following figure. While Java and Postgres perform better than Sedona on a Spark cluster for joins on small datasets, this trend reverses for joins of larger datasets. Especially for the join of 100k and 10M boxes (generated using [star.cs.ucr.edu](https://star.cs.ucr.edu)), the Spark cluster outperforms the single node execution of Java and Postgres. The poor performance of Sedona/Spark on a cluster with only one node indicates that this advantage is actually  [...]
+
+<div style={{textAlign: 'center'}}>
+  <img width="75%" alt="Spatial benchmark 2" src="/img/benchmarks/spatial-bench2.png" />
+</div>
+
+The performance gained by using larger cluster configurations can also be seen in the following visualization of runtime results of box joins with datasets of various sizes.
+
+<div style={{textAlign: 'center'}}>
+  <img width="75%" alt="Spatial benchmark 3" src="/img/benchmarks/spatial-bench3.png" />
+</div>
+
+Other than dataset size, selectivity and subsequently the join result size can also impact execution time significantly. The following chart shows the execution time of joining synthetic box datasets containing 1M boxes each. For each run, one of the datasets contained boxes with decreasing max edge lengths, resulting in higher selectivity for the intersection join. Execution times for Java decreased significantly with decreasing box edge lengths, while execution times for the Spark plat [...]
+
+<div style={{textAlign: 'center'}}>
+  <img width="75%" alt="Spatial benchmark 4" src="/img/benchmarks/spatial-bench4.png" />
+</div>
+
+The benchmark results show multiple scenarios in which platform choice has a significant impact on runtime. Spatial data processing can therefore benefit greatly from using the new spatial operators in Apache Wayang.
+
+---
+
+## Future Work
+
+Our spatial extension enables basic spatial workloads using filter and join operations on the Java, Spark, and Postgres platforms. However, additional operators like nearest neighbor or within-distance operations could enable even more complex scenarios.
+
+Adding platforms specifically designed for spatial data processing could improve performance even further. An interesting candidate for this could be the relatively recently introduced Apache SedonaDB database engine.
+
+Currently, the execution platform has to be chosen manually. Part of potential future work should therefore be the implementation of heuristics for platform selection.
diff --git a/blog/authors.yml b/blog/authors.yml
index 3d9d9a22..9f29aa6d 100644
--- a/blog/authors.yml
+++ b/blog/authors.yml
@@ -23,3 +23,18 @@ glauesppen:
   title: (P)PMC Apache Wayang
   url: https://github.com/glauesppen
   image_url: https://avatars.githubusercontent.com/glauesppen
+max.speer:
+  name: Maximilian Speer
+  title: HPI Student
+  url: https://github.com/MaxSpeer
+  image_url: https://avatars.githubusercontent.com/u/82819900?s=96&v=4
+anton.persitzky:
+  name: Anton Persitzky
+  title: HPI Student
+  url: https://github.com/instant-sky
+  image_url: https://avatars.githubusercontent.com/u/82519211?v=4
+felix.treykorn:
+  name: Felix Treykorn
+  title: HPI Student
+  url: https://github.com/treyfel
+  image_url: https://avatars.githubusercontent.com/u/58570309?v=4
\ No newline at end of file
diff --git a/static/img/architecture/wayang-spatial-architecture.png b/static/img/architecture/wayang-spatial-architecture.png
new file mode 100644
index 00000000..ec81b620
Binary files /dev/null and b/static/img/architecture/wayang-spatial-architecture.png differ
diff --git a/static/img/benchmarks/spatial-bench1.png b/static/img/benchmarks/spatial-bench1.png
new file mode 100644
index 00000000..bc48cae6
Binary files /dev/null and b/static/img/benchmarks/spatial-bench1.png differ
diff --git a/static/img/benchmarks/spatial-bench2.png b/static/img/benchmarks/spatial-bench2.png
new file mode 100644
index 00000000..48205050
Binary files /dev/null and b/static/img/benchmarks/spatial-bench2.png differ
diff --git a/static/img/benchmarks/spatial-bench3.png b/static/img/benchmarks/spatial-bench3.png
new file mode 100644
index 00000000..25d29caf
Binary files /dev/null and b/static/img/benchmarks/spatial-bench3.png differ
diff --git a/static/img/benchmarks/spatial-bench4.png b/static/img/benchmarks/spatial-bench4.png
new file mode 100644
index 00000000..7cf38fca
Binary files /dev/null and b/static/img/benchmarks/spatial-bench4.png differ