You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@wayang.apache.org by ju...@apache.org on 2026/09/15 06:51:01 UTC
(wayang) branch main updated: Reorganize SQL platform examples and setup documentation
This is an automated email from the ASF dual-hosted git repository.
juripetersen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/wayang.git
The following commit(s) were added to refs/heads/main by this push:
new 617de4f41 Reorganize SQL platform examples and setup documentation
617de4f41 is described below
commit 617de4f41cfc6a9dde7dfcd853862c6dc64a6fb0
Author: JunWang222 <70...@users.noreply.github.com>
AuthorDate: Tue Sep 15 02:50:57 2026 -0400
Reorganize SQL platform examples and setup documentation
---
INTEGRATION-TESTS.md | 22 +-
bigquery-setup/README.md | 321 ---------------------
bigquery-setup/data.yaml | 72 -----
bigquery-setup/demo.sh | 23 --
bigquery-setup/docker-compose.yml | 43 ---
bigquery-setup/pom.xml | 79 -----
.../apache/wayang/bigquery/BigQueryEmulatorIT.java | 202 -------------
demo-bigquery.sh | 95 ------
demo-trino.sh | 21 --
env_template_osx.sh | 41 ---
guides/cost-profiling.md | 10 +-
improvement.md | 5 +-
platforms-setup-guides/trino-setup/README.md | 282 ------------------
platforms-setup-guides/trino-setup/demo.sh | 127 --------
.../trino-setup/docker-compose.yml | 142 ---------
platforms-setup-guides/trino-setup/pom.xml | 95 ------
.../trino-setup/scripts/init.sql | 66 -----
.../trino-setup/scripts/run-init.sh | 40 ---
.../apache/wayang/trino/TrinoIntegrationTest.java | 233 ---------------
.../trino-setup/trino/catalog/iceberg.properties | 32 --
.../trino-setup/trino/catalog/tpch.properties | 20 --
.../trino-setup/trino/config.properties | 20 --
presto-setup/README.md | 248 ----------------
presto-setup/docker-compose.yml | 47 ---
presto-setup/etc/catalog/memory.properties | 26 --
trino-setup/README.md | 283 ------------------
trino-setup/demo.sh | 126 --------
trino-setup/docker-compose.yml | 141 ---------
trino-setup/pom.xml | 93 ------
trino-setup/scripts/init.sql | 66 -----
trino-setup/scripts/run-init.sh | 39 ---
.../apache/wayang/trino/TrinoIntegrationTest.java | 232 ---------------
trino-setup/trino/catalog/iceberg.properties | 32 --
trino-setup/trino/catalog/tpch.properties | 20 --
trino-setup/trino/config.properties | 20 --
wayang-applications/README.md | 4 +
wayang-applications/bigquery.md | 103 +++++++
wayang-applications/pom.xml | 25 ++
wayang-applications/presto.md | 97 +++++++
.../apache/wayang/applications/BigQueryDemo.java | 246 ++++++++++++++++
.../org/apache/wayang/applications/PrestoDemo.java | 88 ++++++
.../org/apache/wayang/applications}/TrinoDemo.java | 123 +++-----
.../wayang/applications/SqlPlatformDemoTest.java | 61 ++++
wayang-applications/trino.md | 101 +++++++
wayang-platforms/wayang-bigquery/README.md | 51 ++++
.../org/apache/wayang/bigquery/BigQueryDemo.java | 307 --------------------
wayang-platforms/wayang-presto/README.md | 47 +++
.../resources/wayang-presto-defaults.properties | 2 +-
.../apache/wayang/presto/PrestoCostPilotIT.java | 2 +-
.../apache/wayang/presto/PrestoOperatorsIT.java | 6 +-
wayang-platforms/wayang-trino/README.md | 44 +++
.../org/apache/wayang/trino/TrinoOperatorsIT.java | 4 +-
52 files changed, 922 insertions(+), 3753 deletions(-)
diff --git a/INTEGRATION-TESTS.md b/INTEGRATION-TESTS.md
index 1d7d39406..2567d7c54 100644
--- a/INTEGRATION-TESTS.md
+++ b/INTEGRATION-TESTS.md
@@ -54,29 +54,23 @@ local runs; skip them:
## Presto
-The test is self-contained: it creates and seeds its own `memory.wayang_it`
-tables in Presto's built-in **in-memory connector** (scaled to 120k rows so the
-optimizer elects SQL pushdown) and drops them afterwards — no Hive metastore or
-object storage required.
+The test creates and seeds its own `memory.wayang_it` tables in a user-managed
+Presto deployment with the memory connector enabled. It drops the fixtures
+afterward.
```bash
-# 1. start a single PrestoDB node with the in-memory connector
-cd presto-setup && docker compose up -d --wait && cd ..
-
-# 2. run the operator tests (JDK 17)
+# Run the operator tests against the configured deployment (JDK 17)
+PRESTO_HOST=presto.example.com PRESTO_PORT=8080 PRESTO_USER=wayang \
JAVA_HOME=/path/to/jdk-17 \
mvn -o test -pl wayang-platforms/wayang-presto \
- -Dtest=AllOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \
+ -Dtest=PrestoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \
-Drat.skip=true -Dlicense.skip=true -Dmaven.javadoc.skip=true -Pskip-prerequisite-check
-
-# 3. tear down when done
-cd presto-setup && docker compose down -v && cd ..
```
Expected: `Tests run: 4, Failures: 0, Errors: 0, Skipped: 0`.
-`docker compose up -d --wait` blocks on the container healthcheck, so Presto is
-query-ready when it returns. Presto listens on host port **8081** (container 8080).
+Connection settings are supplied through `PRESTO_HOST`, `PRESTO_PORT`, and
+`PRESTO_USER`.
---
diff --git a/bigquery-setup/README.md b/bigquery-setup/README.md
deleted file mode 100644
index bffd6433d..000000000
--- a/bigquery-setup/README.md
+++ /dev/null
@@ -1,321 +0,0 @@
-# BigQuery Local Setup
-
-Local BigQuery emulator and validation instructions for the Wayang BigQuery
-platform.
-
-The local validation path has two parts and does not require a GCP account:
-
-1. Build the Wayang BigQuery platform and run the shared JDBC SQL-generation tests.
-2. Run BigQuery-compatible SQL tests against the local emulator.
-
-There is also an optional real-BigQuery validation path:
-
-3. Run the Wayang BigQuery operator tests through JDBC against real BigQuery.
-
-Run the commands below from the repository root. Java 17 and Docker with Docker
-Compose are required for the emulator tests. A GCP project and service-account
-key, plus the `gcloud` SDK, are required only for the optional real-BigQuery
-operator tests. Maven is provided by the repository wrapper.
-
-## Command Conventions
-
-Use the `bash` blocks on macOS/Linux terminals. Use the `powershell` blocks on
-Windows PowerShell from the repository root. Docker Compose commands are the
-same on both platforms. The `gcloud` commands also work on Windows; either run
-each command on one line or replace Bash line-continuation backslashes with
-PowerShell backticks.
-
-## Stack
-
-| Component | Image | Port | Role |
-|-----------|-------|------|------|
-| **BigQuery Emulator** | `ghcr.io/goccy/bigquery-emulator:0.6.6` | 9050 (HTTP) / 9060 (gRPC) | BigQuery-compatible SQL engine |
-
-Single container. Data is seeded from `data.yaml` on startup and lives in memory.
-
-## Directory Layout
-
-```
-bigquery-setup/
-|-- docker-compose.yml # Emulator container
-|-- data.yaml # Seed data (test-project.sales.orders)
-|-- pom.xml # Standalone Maven project
-`-- src/test/java/.../
- `-- BigQueryEmulatorIT.java # JUnit 5 integration tests
-
-wayang-platforms/wayang-bigquery/src/test/java/.../
-`-- BigQueryOperatorsIT.java # Wayang operator tests against real BigQuery
-```
-
-## 1. Test the Wayang BigQuery Platform
-
-Build the BigQuery platform and its required modules:
-
-```bash
-./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-bigquery -am -DskipTests -Drat.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-bigquery -am -DskipTests -Drat.skip=true test
-```
-
-Then run the shared JDBC SQL-generation tests:
-
-```bash
-./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test
-```
-
-Expected result:
-
-```text
-Wayang Platform BigQuery ... SUCCESS
-Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
-```
-
-## 2. Test the Local BigQuery Emulator
-
-### 1. Start the emulator
-
-```bash
-docker compose -f bigquery-setup/docker-compose.yml up -d
-```
-
-The emulator starts in ~2 seconds. Data from `data.yaml` is loaded automatically.
-
-### 2. Run integration tests
-
-```bash
-./mvnw -f bigquery-setup/pom.xml -Dtest=BigQueryEmulatorIT test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -f bigquery-setup/pom.xml -Dtest=BigQueryEmulatorIT test
-```
-
-The successful result must show that no tests were skipped:
-
-```text
-Tests run: 7, Failures: 0, Errors: 0, Skipped: 0
-```
-
-If the emulator is unavailable, Maven can still print `BUILD SUCCESS` while
-showing `Skipped: 7`. That does not count as a successful emulator test.
-
-### 3. Manual exploration
-
-Query via curl:
-
-```bash
-curl -s -X POST \
- "http://localhost:9050/bigquery/v2/projects/test-project/queries" \
- -H "Content-Type: application/json" \
- -d '{"query": "SELECT * FROM sales.orders LIMIT 5", "useLegacySql": false}' \
- | python3 -m json.tool
-```
-
-### 4. Tear down
-
-```bash
-docker compose -f bigquery-setup/docker-compose.yml down
-```
-
-## 3. Optional: Test the Wayang Operators Against Real BigQuery
-
-`BigQueryOperatorsIT` uses the BigQuery JDBC driver and cannot run against the
-local emulator. It requires a real GCP project and a service-account JSON key.
-
-The test setup creates its own fixture tables in a configurable dataset
-(`wayang_it` by default): `orders`, `regions`, and `operator_result`. The tests
-issue `SELECT`, `CREATE TABLE AS`, and `DROP` statements, then remove those
-tables during cleanup.
-
-### 1. Enable BigQuery and create a service account
-
-Replace `YOUR_PROJECT_ID` in the following commands:
-
-```bash
-gcloud auth login
-gcloud config set project YOUR_PROJECT_ID
-gcloud services enable bigquery.googleapis.com
-
-gcloud iam service-accounts create wayang-bq \
- --display-name="Wayang BigQuery IT"
-
-gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
- --member="serviceAccount:wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
- --role="roles/bigquery.jobUser"
-
-gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
- --member="serviceAccount:wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
- --role="roles/bigquery.dataEditor"
-
-gcloud iam service-accounts keys create "$HOME/wayang-bq-key.json" \
- --iam-account="wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com"
-```
-
-On Windows PowerShell, the same setup can be run as:
-
-```powershell
-gcloud auth login
-gcloud config set project YOUR_PROJECT_ID
-gcloud services enable bigquery.googleapis.com
-gcloud iam service-accounts create wayang-bq --display-name="Wayang BigQuery IT"
-gcloud projects add-iam-policy-binding YOUR_PROJECT_ID --member="serviceAccount:wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com" --role="roles/bigquery.jobUser"
-gcloud projects add-iam-policy-binding YOUR_PROJECT_ID --member="serviceAccount:wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com" --role="roles/bigquery.dataEditor"
-gcloud iam service-accounts keys create "$HOME\wayang-bq-key.json" --iam-account="wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com"
-```
-
-The service account needs `jobUser` to run queries and `dataEditor` to create,
-read, and drop the test dataset tables.
-
-### 2. Choose the test dataset
-
-The test creates the dataset if it does not exist. By default it uses
-`wayang_it`; override it with `-Dbigquery.dataset=DATASET_ID` or
-`BIGQUERY_DATASET` if the project needs a different dataset name.
-
-### 3. Run the operator tests
-
-```bash
-./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-bigquery -am \
- -Dtest=BigQueryOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \
- -DfailIfNoTests=false \
- -Dbigquery.project=YOUR_PROJECT_ID \
- -Dbigquery.saEmail=wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com \
- -Dbigquery.keyPath="$HOME/wayang-bq-key.json" \
- -Dbigquery.location=US \
- -Dbigquery.dataset=wayang_it \
- -Drat.skip=true -Dlicense.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-bigquery -am -Dtest=BigQueryOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Dbigquery.project=YOUR_PROJECT_ID -Dbigquery.saEmail=wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com -Dbigquery.keyPath=C:\path\to\wayang-bq-key.json -Dbigquery.location=US -Dbigquery.dataset=wayang_it -Drat.skip=true -Dlicense.skip=true test
-```
-
-System properties take precedence over the equivalent environment variables:
-
-| System property | Environment variable | Default |
-|-----------------|----------------------|---------|
-| `bigquery.project` | `BIGQUERY_PROJECT` | `your-project` |
-| `bigquery.saEmail` | `BIGQUERY_SA_EMAIL` | `wayang-bq@<project>.iam.gserviceaccount.com` |
-| `bigquery.keyPath` | `BIGQUERY_KEY_PATH` | `$HOME/wayang-bq-key.json` |
-| `bigquery.location` | `BIGQUERY_LOCATION` | `US` |
-| `bigquery.dataset` | `BIGQUERY_DATASET` | `wayang_it` |
-
-Successful real-BigQuery validation must show:
-
-```text
-Tests run: 13, Failures: 0, Errors: 0, Skipped: 0
-```
-
-If the browser uses a local proxy, pass the same proxy to both CLI tools and
-the Maven test JVM. For example, with a proxy at `127.0.0.1:7890`, set
-`HTTP_PROXY`/`HTTPS_PROXY` and use `JAVA_TOOL_OPTIONS` with
-`-Dhttp.proxyHost`, `-Dhttp.proxyPort`, `-Dhttps.proxyHost`, and
-`-Dhttps.proxyPort`.
-
-On PowerShell:
-
-```powershell
-$env:HTTP_PROXY="http://127.0.0.1:7890"
-$env:HTTPS_PROXY="http://127.0.0.1:7890"
-$env:JAVA_TOOL_OPTIONS="-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=7890 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=7890"
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-bigquery -am -Dtest=BigQueryOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Dbigquery.project=YOUR_PROJECT_ID -Dbigquery.saEmail=wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com -Dbigquery.keyPath=C:\path\to\wayang-bq-key.json -Dbigquery.location=US -Dbigquery.dataset=wayang_it -Drat.skip=true -Dlicense.skip=true test
-Remove-Item Env:HTTP_PROXY, Env:HTTPS_PROXY, Env:JAVA_TOOL_OPTIONS
-```
-
-If credentials or the project configuration are missing, Maven can still print
-`BUILD SUCCESS` with `Skipped: 13`. That does not count as successful
-real-BigQuery validation.
-
-## 4. Optional: Re-run Cost Profiling
-
-Follow the shared cost-profiling guide in
-[`guides/cost-profiling.md`](../guides/cost-profiling.md). This setup guide
-only covers the BigQuery emulator and real BigQuery validation setup. BigQuery
-cost profiling uses `BigQueryCostPilotIT`, so it needs the same real-BigQuery
-credentials as the optional operator tests above.
-
-BigQuery-specific profiling values:
-
-| Item | Value |
-|------|-------|
-| Maven module | `wayang-platforms/wayang-bigquery` |
-| Profiling test | `BigQueryCostPilotIT` |
-| Property prefix | `bigquery.profile.*` |
-| Profiling dataset property | `bigquery.profile.dataset` |
-| Default output directory | `target/cost-profiling/bigquery` |
-| Learned parameters file | `wayang-platforms/wayang-bigquery/src/main/resources/wayang-bigquery-defaults.properties` |
-
-## Test Coverage
-
-### Local emulator tests
-
-| Test | What it checks |
-|------|----------------|
-| `testDatasetVisible` | `sales` dataset exists |
-| `testFullScan` | Full table scan, 10 rows |
-| `testFilterByRegion` | `WHERE region = 'APAC'` |
-| `testFilterByAmount` | `WHERE amount > 1000` |
-| `testAggregation` | `GROUP BY region` + `SUM(amount)` |
-| `testProjection` | `SELECT region, product LIMIT 5` |
-| `testCount` | `SELECT count(*)`, used by Wayang for cardinality estimation |
-
-### Real BigQuery operator tests
-
-| Test | What it checks |
-|------|----------------|
-| `tableSource` | Full table scan through Wayang into a BigQuery sink table |
-| `filter` | String filter pushdown |
-| `projection` | Multi-column projection pushdown |
-| `join` | Full Wayang join plan with normalization before the sink table |
-| `globalReduce` | Global `SUM(amount)` |
-| `reduceBy` | `SUM(amount) GROUP BY region` |
-| `sort` | BigQuery sort operator SQL-clause contract |
-| `tableSink` | `CREATE TABLE AS SELECT` and cleanup |
-| `javaPlanBuilderReadTableFilterProjection` | `readTable -> filter -> projection -> writeTable` |
-| `javaPlanBuilderReadTableFilterGlobalReduce` | `readTable -> filter -> globalReduce -> writeTable` |
-| `javaPlanBuilderReadTableReduceBySort` | `readTable -> reduceByKey -> sort -> writeTable` |
-| `javaPlanBuilderReadTableFilterProjectionTableSink` | `readTable -> filter -> projection -> writeTable` |
-| `javaPlanBuilderReadTableJoin` | `readTable + readTable -> join -> writeTable` |
-
-The combination tests use `.withTargetPlatform(BigQuery.platform())` so the
-small 10-row fixture still exercises BigQuery SQL pushdown. The join test creates
-and cleans up a temporary distinct-region lookup table.
-
-## Emulator Environment Variable
-
-```bash
-BIGQUERY_HOST=http://localhost:9050 ./mvnw -f bigquery-setup/pom.xml -Dtest=BigQueryEmulatorIT test
-```
-
-On PowerShell:
-
-```powershell
-$env:BIGQUERY_HOST="http://localhost:9050"
-.\mvnw.cmd --% -f bigquery-setup/pom.xml -Dtest=BigQueryEmulatorIT test
-Remove-Item Env:BIGQUERY_HOST
-```
-
-## Notes
-
-- Emulator tests use the `google-cloud-bigquery` client library (REST-based, no
- JDBC).
-- The emulator client connects with `NoCredentials`; no GCP account is needed.
-- The BigQuery JDBC driver (`google-cloud-bigquery-jdbc`) requires OAuth even
- against the emulator, so `BigQueryOperatorsIT` runs only against real
- BigQuery.
-- Emulator tests validate SQL compatibility, but only `BigQueryOperatorsIT`
- validates end-to-end Wayang-to-BigQuery JDBC execution.
diff --git a/bigquery-setup/data.yaml b/bigquery-setup/data.yaml
deleted file mode 100644
index c1a283285..000000000
--- a/bigquery-setup/data.yaml
+++ /dev/null
@@ -1,72 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-projects:
-- id: test-project
- datasets:
- - id: sales
- tables:
- - id: orders
- columns:
- - name: order_id
- type: INTEGER
- - name: region
- type: STRING
- - name: product
- type: STRING
- - name: amount
- type: FLOAT
- data:
- - order_id: 1
- region: APAC
- product: Widget A
- amount: 1500.0
- - order_id: 2
- region: EMEA
- product: Widget B
- amount: 800.5
- - order_id: 3
- region: AMER
- product: Widget A
- amount: 2200.0
- - order_id: 4
- region: APAC
- product: Widget C
- amount: 350.75
- - order_id: 5
- region: EMEA
- product: Widget A
- amount: 1100.0
- - order_id: 6
- region: AMER
- product: Widget B
- amount: 950.25
- - order_id: 7
- region: APAC
- product: Widget B
- amount: 1750.0
- - order_id: 8
- region: EMEA
- product: Widget C
- amount: 420.0
- - order_id: 9
- region: AMER
- product: Widget C
- amount: 680.5
- - order_id: 10
- region: APAC
- product: Widget A
- amount: 3000.0
diff --git a/bigquery-setup/demo.sh b/bigquery-setup/demo.sh
deleted file mode 100644
index 270ab869a..000000000
--- a/bigquery-setup/demo.sh
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/usr/bin/env bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-WAYANG_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
-exec "$WAYANG_ROOT/demo-bigquery.sh" "$@"
diff --git a/bigquery-setup/docker-compose.yml b/bigquery-setup/docker-compose.yml
deleted file mode 100644
index 4f3dd69e0..000000000
--- a/bigquery-setup/docker-compose.yml
+++ /dev/null
@@ -1,43 +0,0 @@
----
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Stack: BigQuery Emulator (goccy/bigquery-emulator)
-#
-# Single container — no metastore, no object storage needed.
-# Data is seeded from data.yaml on startup and lives in memory.
-#
-# Ports:
-# HTTP (REST API): http://localhost:9050
-# gRPC (Storage API): localhost:9060
-
-services:
-
- bigquery:
- image: ghcr.io/goccy/bigquery-emulator:0.6.6
- platform: linux/amd64
- container_name: bigquery-emulator
- ports:
- - "9050:9050"
- - "9060:9060"
- volumes:
- - ./data.yaml:/data.yaml
- command: --project=test-project --data-from-yaml=/data.yaml
- healthcheck:
- test: ["CMD-SHELL", "bash -c ': >/dev/tcp/localhost/9050'"]
- interval: 10s
- timeout: 5s
- retries: 5
diff --git a/bigquery-setup/pom.xml b/bigquery-setup/pom.xml
deleted file mode 100644
index 9ffd14533..000000000
--- a/bigquery-setup/pom.xml
+++ /dev/null
@@ -1,79 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
- http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <modelVersion>4.0.0</modelVersion>
-
- <groupId>org.apache.wayang</groupId>
- <artifactId>bigquery-setup</artifactId>
- <version>1.0-SNAPSHOT</version>
- <packaging>jar</packaging>
-
- <name>BigQuery Local Setup — Integration Tests</name>
- <description>
- Standalone integration tests for a local BigQuery emulator.
- Independent of the Wayang codebase.
- </description>
-
- <properties>
- <maven.compiler.source>11</maven.compiler.source>
- <maven.compiler.target>11</maven.compiler.target>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <junit.version>5.10.2</junit.version>
- <bigquery.version>2.49.0</bigquery.version>
- </properties>
-
- <dependencies>
- <!-- Google Cloud BigQuery client library (REST-based, no JDBC) -->
- <dependency>
- <groupId>com.google.cloud</groupId>
- <artifactId>google-cloud-bigquery</artifactId>
- <version>${bigquery.version}</version>
- <scope>test</scope>
- </dependency>
-
- <!-- JUnit 5 -->
- <dependency>
- <groupId>org.junit.jupiter</groupId>
- <artifactId>junit-jupiter</artifactId>
- <version>${junit.version}</version>
- <scope>test</scope>
- </dependency>
-
- <!-- Logging -->
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-simple</artifactId>
- <version>2.0.12</version>
- <scope>test</scope>
- </dependency>
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <version>3.2.5</version>
- </plugin>
- </plugins>
- </build>
-</project>
diff --git a/bigquery-setup/src/test/java/org/apache/wayang/bigquery/BigQueryEmulatorIT.java b/bigquery-setup/src/test/java/org/apache/wayang/bigquery/BigQueryEmulatorIT.java
deleted file mode 100644
index 6c03f7843..000000000
--- a/bigquery-setup/src/test/java/org/apache/wayang/bigquery/BigQueryEmulatorIT.java
+++ /dev/null
@@ -1,202 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.wayang.bigquery;
-
-import com.google.auth.oauth2.GoogleCredentials;
-import com.google.cloud.NoCredentials;
-import com.google.cloud.bigquery.*;
-import org.junit.jupiter.api.*;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import static org.junit.jupiter.api.Assertions.*;
-
-/**
- * Integration tests for the local BigQuery emulator.
- *
- * Prerequisites: run `docker-compose up -d` first.
- *
- * Run tests:
- * mvn test -Pintegration
- */
-@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
-class BigQueryEmulatorIT {
-
- private static final String EMULATOR_HOST = System.getenv().getOrDefault("BIGQUERY_HOST", "http://localhost:9050");
- private static final String PROJECT_ID = "test-project";
- private static final String DATASET = "sales";
-
- private static BigQuery bigquery;
- private static boolean emulatorAvailable = false;
-
- @BeforeAll
- static void setupClient() {
- try {
- bigquery = BigQueryOptions.newBuilder()
- .setHost(EMULATOR_HOST)
- .setLocation("US")
- .setProjectId(PROJECT_ID)
- .setCredentials(NoCredentials.getInstance())
- .build()
- .getService();
-
- // Quick connectivity check
- bigquery.getDataset(DatasetId.of(PROJECT_ID, DATASET));
- emulatorAvailable = true;
- System.out.printf("Connected to BigQuery emulator at %s%n", EMULATOR_HOST);
- } catch (Exception e) {
- System.err.println("BigQuery emulator not available: " + e.getMessage());
- }
- }
-
- private List<List<Object>> runQuery(String sql) throws InterruptedException {
- QueryJobConfiguration config = QueryJobConfiguration.newBuilder(sql)
- .setUseLegacySql(false)
- .build();
- TableResult result = bigquery.query(config);
- List<List<Object>> rows = new ArrayList<>();
- for (FieldValueList row : result.iterateAll()) {
- List<Object> r = new ArrayList<>();
- for (FieldValue val : row) {
- r.add(val.isNull() ? null : val.getValue());
- }
- rows.add(r);
- }
- return rows;
- }
-
- // ── Test 1: Dataset visible ──────────────────────────────────────────
-
- @Test
- @Order(1)
- @DisplayName("BigQuery emulator: dataset 'sales' is visible")
- void testDatasetVisible() {
- Assumptions.assumeTrue(emulatorAvailable, "Emulator not available");
-
- Dataset ds = bigquery.getDataset(DatasetId.of(PROJECT_ID, DATASET));
- assertNotNull(ds, "Dataset 'sales' should exist");
- System.out.println("[PASS] Dataset 'sales' is visible");
- }
-
- // ── Test 2: Full table scan ──────────────────────────────────────────
-
- @Test
- @Order(2)
- @DisplayName("BigQuery emulator: full table scan on orders")
- void testFullScan() throws Exception {
- Assumptions.assumeTrue(emulatorAvailable, "Emulator not available");
-
- List<List<Object>> rows = runQuery(
- "SELECT * FROM `test-project.sales.orders` ORDER BY order_id"
- );
- assertEquals(10, rows.size(), "Expected 10 rows");
- System.out.println("[PASS] Full scan: " + rows.size() + " rows");
- rows.forEach(r -> System.out.println(" " + r));
- }
-
- // ── Test 3: Filter by region ─────────────────────────────────────────
-
- @Test
- @Order(3)
- @DisplayName("BigQuery emulator: filter by region = APAC")
- void testFilterByRegion() throws Exception {
- Assumptions.assumeTrue(emulatorAvailable, "Emulator not available");
-
- List<List<Object>> rows = runQuery(
- "SELECT order_id, region, amount FROM `test-project.sales.orders` WHERE region = 'APAC' ORDER BY order_id"
- );
- assertFalse(rows.isEmpty(), "Should have APAC rows");
- rows.forEach(r -> assertEquals("APAC", r.get(1), "All rows should be APAC"));
- System.out.printf("[PASS] Filter: %d APAC rows%n", rows.size());
- }
-
- // ── Test 4: Filter by amount ─────────────────────────────────────────
-
- @Test
- @Order(4)
- @DisplayName("BigQuery emulator: filter by amount > 1000")
- void testFilterByAmount() throws Exception {
- Assumptions.assumeTrue(emulatorAvailable, "Emulator not available");
-
- List<List<Object>> rows = runQuery(
- "SELECT order_id, amount FROM `test-project.sales.orders` WHERE amount > 1000 ORDER BY amount DESC"
- );
- assertFalse(rows.isEmpty());
- rows.forEach(r -> assertTrue(
- Double.parseDouble(r.get(1).toString()) > 1000.0,
- "All amounts should be > 1000"
- ));
- System.out.printf("[PASS] Amount filter: %d rows with amount > 1000%n", rows.size());
- }
-
- // ── Test 5: Aggregation ──────────────────────────────────────────────
-
- @Test
- @Order(5)
- @DisplayName("BigQuery emulator: aggregate by region")
- void testAggregation() throws Exception {
- Assumptions.assumeTrue(emulatorAvailable, "Emulator not available");
-
- List<List<Object>> rows = runQuery(
- "SELECT region, COUNT(*) AS cnt, SUM(amount) AS total " +
- "FROM `test-project.sales.orders` GROUP BY region ORDER BY total DESC"
- );
- assertFalse(rows.isEmpty());
- System.out.println("[PASS] Aggregation by region:");
- rows.forEach(r -> System.out.printf(" region=%-5s count=%s total=%s%n",
- r.get(0), r.get(1), r.get(2)));
- }
-
- // ── Test 6: Projection ───────────────────────────────────────────────
-
- @Test
- @Order(6)
- @DisplayName("BigQuery emulator: projection (region, product)")
- void testProjection() throws Exception {
- Assumptions.assumeTrue(emulatorAvailable, "Emulator not available");
-
- List<List<Object>> rows = runQuery(
- "SELECT region, product FROM `test-project.sales.orders` LIMIT 5"
- );
- assertEquals(5, rows.size());
- rows.forEach(r -> {
- assertNotNull(r.get(0), "region should not be null");
- assertNotNull(r.get(1), "product should not be null");
- });
- System.out.println("[PASS] Projection (region, product): 5 rows");
- }
-
- // ── Test 7: COUNT(*) ─────────────────────────────────────────────────
-
- @Test
- @Order(7)
- @DisplayName("BigQuery emulator: SELECT count(*)")
- void testCount() throws Exception {
- Assumptions.assumeTrue(emulatorAvailable, "Emulator not available");
-
- List<List<Object>> rows = runQuery(
- "SELECT count(*) FROM `test-project.sales.orders`"
- );
- assertEquals(1, rows.size());
- long count = Long.parseLong(rows.get(0).get(0).toString());
- assertEquals(10, count, "Should have 10 rows");
- System.out.println("[PASS] COUNT(*) = " + count);
- }
-}
diff --git a/demo-bigquery.sh b/demo-bigquery.sh
deleted file mode 100644
index dcfe42b40..000000000
--- a/demo-bigquery.sh
+++ /dev/null
@@ -1,95 +0,0 @@
-#!/usr/bin/env bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-set -euo pipefail
-
-WAYANG_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-LIVE_MODE=false
-[[ "${1:-}" == "--live" ]] && LIVE_MODE=true
-
-BQ_PROJECT="${BQ_PROJECT:-my-project}"
-BQ_URL="${BQ_URL:-}"
-MAVEN_FLAGS="-Pskip-prerequisite-check -Drat.skip=true -Dmaven.javadoc.skip=true"
-
-banner() {
- echo
- echo "============================================================"
- printf " %s\n" "$*"
- echo "============================================================"
- echo
-}
-
-step() {
- echo
- echo "-- $*"
- echo
-}
-
-pause() {
- if [[ "${WAYANG_DEMO_AUTO:-false}" != "true" ]]; then
- echo
- read -rp "Press ENTER to continue..." _ || true
- echo
- fi
-}
-
-run_demo_class() {
- local main_class="$1"
- shift
- cd "$WAYANG_ROOT"
- "$WAYANG_ROOT/mvnw" exec:java -pl wayang-platforms/wayang-bigquery \
- -Dexec.mainClass="$main_class" \
- "$@" \
- ${MAVEN_FLAGS} -q 2>/dev/null || true
-}
-
-banner "ACT 1: BigQuery cost model"
-step "Read cost settings from wayang-bigquery-defaults.properties"
-run_demo_class "org.apache.wayang.bigquery.BigQueryDemo" \
- "-Dbigquery.mode=cost" \
- "-Dbigquery.project=${BQ_PROJECT}"
-
-pause
-
-banner "ACT 2: BigQuery filter operator"
-if [[ "$LIVE_MODE" == true && -n "$BQ_URL" ]]; then
- run_demo_class "org.apache.wayang.bigquery.BigQueryDemo" \
- "-Dbigquery.mode=filter" \
- "-Dbigquery.url=${BQ_URL}" \
- "-Dbigquery.project=${BQ_PROJECT}"
-else
- run_demo_class "org.apache.wayang.bigquery.BigQueryDemo" \
- "-Dbigquery.mode=filter" \
- "-Dbigquery.project=${BQ_PROJECT}"
-fi
-
-pause
-
-banner "ACT 3: BigQuery projection operator"
-if [[ "$LIVE_MODE" == true && -n "$BQ_URL" ]]; then
- run_demo_class "org.apache.wayang.bigquery.BigQueryDemo" \
- "-Dbigquery.mode=projection" \
- "-Dbigquery.url=${BQ_URL}" \
- "-Dbigquery.project=${BQ_PROJECT}"
-else
- run_demo_class "org.apache.wayang.bigquery.BigQueryDemo" \
- "-Dbigquery.mode=projection" \
- "-Dbigquery.project=${BQ_PROJECT}"
-fi
-
-banner "Demo complete"
diff --git a/demo-trino.sh b/demo-trino.sh
deleted file mode 100644
index 45b1fad7a..000000000
--- a/demo-trino.sh
+++ /dev/null
@@ -1,21 +0,0 @@
-#!/usr/bin/env bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-set -euo pipefail
-
-WAYANG_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-exec "$WAYANG_ROOT/trino-setup/demo.sh" "$@"
diff --git a/env_template_osx.sh b/env_template_osx.sh
deleted file mode 100755
index 6ec3cc021..000000000
--- a/env_template_osx.sh
+++ /dev/null
@@ -1,41 +0,0 @@
-#!/usr/bin/env bash
-
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-export BOOTSTRAP_SERVER= ...
-export CLUSTER_API_KEY= ...
-export CLUSTER_API_SECRET= ...
-export SR_ENDPOINT= ...
-export SR_API_KEY= ...
-export SR_API_SECRET= ...
-export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=" ... : .... "
-export SCHEMA_REGISTRY_URL="https://.... "
-
-export SPARK_HOME= ...
-export HADOOP_HOME= ...
-export PATH=$PATH:$HADOOP_HOME/bin
-export WAYANG_VERSION= ...
-export WAYANG_HOME= ...
-export WAYANG_APP_HOME= ...
-
-echo "Hadoop home : $HADOOP_HOME"
-echo "Spark home : $SPARK_HOME"
-echo "Wayang home : $WAYANG_HOME"
-echo "Wayang app : $WAYANG_APP_HOME"
-echo "Wayang version : $WAYANG_VERSION"
-
-
diff --git a/guides/cost-profiling.md b/guides/cost-profiling.md
index 8dc5295bf..ce0d8a97c 100644
--- a/guides/cost-profiling.md
+++ b/guides/cost-profiling.md
@@ -257,7 +257,7 @@ model should predict:
- If Wayang should predict user-visible runtime, fit wall-clock elapsed time.
- If the platform reports CPU time and the model uses CPU load, make sure the
conversion to Wayang cost is consistent with the resource model.
-- Parameters learned on a local Docker setup should be treated as local
+- Parameters learned on one deployment should be treated as environment-specific
reference values, not universal defaults for every deployment.
## 6. Running a Profiling Experiment
@@ -265,11 +265,11 @@ model should predict:
The exact command depends on the platform module, test class, and property
prefix.
-| Platform | Setup guide | Maven module | Test class | Property prefix | Default output directory |
+| Platform | Platform guide | Maven module | Test class | Property prefix | Default output directory |
|----------|-------------|--------------|------------|-----------------|--------------------------|
-| Trino | `trino-setup/README.md` | `wayang-platforms/wayang-trino` | `TrinoCostPilotIT` | `trino.profile.*` | `target/cost-profiling/trino` |
-| Presto | `presto-setup/README.md` | `wayang-platforms/wayang-presto` | `PrestoCostPilotIT` | `presto.profile.*` | `target/cost-profiling/presto` |
-| BigQuery | `bigquery-setup/README.md` | `wayang-platforms/wayang-bigquery` | `BigQueryCostPilotIT` | `bigquery.profile.*` | `target/cost-profiling/bigquery` |
+| Trino | `wayang-platforms/wayang-trino/README.md` | `wayang-platforms/wayang-trino` | `TrinoCostPilotIT` | `trino.profile.*` | `target/cost-profiling/trino` |
+| Presto | `wayang-platforms/wayang-presto/README.md` | `wayang-platforms/wayang-presto` | `PrestoCostPilotIT` | `presto.profile.*` | `target/cost-profiling/presto` |
+| BigQuery | `wayang-platforms/wayang-bigquery/README.md` | `wayang-platforms/wayang-bigquery` | `BigQueryCostPilotIT` | `bigquery.profile.*` | `target/cost-profiling/bigquery` |
The commands below use PowerShell. On macOS/Linux, use `./mvnw` instead of
`.\mvnw.cmd` and replace PowerShell backticks with Bash line-continuation
diff --git a/improvement.md b/improvement.md
index 5485fd929..46086e695 100644
--- a/improvement.md
+++ b/improvement.md
@@ -106,13 +106,10 @@ The high-level tests also rely on the `withSqlUdf` / `withSqlUdfs` additions to
sort builders can carry SQL implementations.
```bash
-docker compose -f presto-setup/docker-compose.yml up -d
-
+PRESTO_HOST=presto.example.com PRESTO_PORT=8080 PRESTO_USER=wayang \
JAVA_HOME=<jdk17> mvn test -pl wayang-platforms/wayang-presto -am \
-Dtest=PrestoOperatorsIT -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false \
-Drat.skip=true -Dlicense.skip=true -Pskip-prerequisite-check
-
-docker compose -f presto-setup/docker-compose.yml down
```
Expected: `Tests run: 13, Failures: 0, Errors: 0, Skipped: 0`. The suite scales
diff --git a/platforms-setup-guides/trino-setup/README.md b/platforms-setup-guides/trino-setup/README.md
deleted file mode 100644
index 8be4f7f3c..000000000
--- a/platforms-setup-guides/trino-setup/README.md
+++ /dev/null
@@ -1,282 +0,0 @@
-# Trino Local Setup
-
-Local Trino environment backed by an **Iceberg** data lake, completely containerised.
-
-The current validation has three parts:
-
-1. Build the Wayang Trino platform and run the shared JDBC SQL-generation tests.
-2. Run the Wayang Trino operator tests against the live local stack.
-3. Run standalone JDBC integration tests against the local Trino, Iceberg, and MinIO stack.
-
-Run the commands below from the repository root. Java 17 and Docker with
-Docker Compose are required; Maven is provided by the repository wrapper.
-
-The pure Trino platform branch is named `wayang-trino`:
-
-```bash
-git checkout wayang-trino
-```
-
-## Command Conventions
-
-Use the `bash` blocks on macOS/Linux terminals. Use the `powershell` blocks on
-Windows PowerShell from the repository root. Docker Compose commands are the
-same on both platforms.
-
-## Stack
-
-| Component | Image | Port | Role |
-|-----------|-------|------|------|
-| **Trino** | `trinodb/trino:435` | 8080 | SQL query engine |
-| **Hive Metastore** | `naushadh/hive-metastore:latest` | 9083 | Iceberg table catalog (Thrift) |
-| **PostgreSQL** | `postgres:15-alpine` | 5432 | HMS metadata backing store |
-| **MinIO** | `minio/minio:latest` | 9000 / 9001 | S3-compatible object storage |
-
-HMS is the battle-tested Iceberg catalog for Trino. Parquet data files are written by Trino directly to MinIO; HMS only stores schema/table metadata.
-
-## Directory Layout
-
-```
-platforms-setup-guides/
-`-- trino-setup/
- |-- docker-compose.yml # Full stack definition
- |-- demo.sh # End-to-end Trino + Wayang walkthrough
- |-- trino/
- | |-- config.properties # Trino node config
- | `-- catalog/
- | |-- iceberg.properties # Iceberg via HMS + MinIO
- | `-- tpch.properties # Built-in TPC-H (no storage needed)
- |-- scripts/
- | |-- init.sql # Creates iceberg.sales.orders + sample rows
- | `-- run-init.sh # Helper: waits for Trino then runs init.sql
- |-- pom.xml # Standalone Maven project (Java 17)
- `-- src/test/java/.../
- `-- TrinoIntegrationTest.java # JUnit 5 integration tests
-```
-
-## 1. Test the Wayang Trino Platform
-
-Build the Trino platform and its required modules:
-
-```bash
-./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am -DskipTests -Drat.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am -DskipTests -Drat.skip=true test
-```
-
-Then run the shared JDBC SQL-generation tests:
-
-```bash
-./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test
-```
-
-Expected result:
-
-```text
-Wayang Platform Trino ... SUCCESS
-Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
-```
-
-## 2. Test Against the Local Trino Stack
-
-### 1. Start the stack
-
-```bash
-docker compose -f platforms-setup-guides/trino-setup/docker-compose.yml up -d
-```
-
-Wait ~30 seconds for all services to become healthy. Check with:
-
-```bash
-docker compose -f platforms-setup-guides/trino-setup/docker-compose.yml ps
-# or watch the Trino UI at http://localhost:8080
-```
-
-### 2. Run the Wayang Trino operator tests
-
-`TrinoOperatorsIT` exercises the Wayang Trino implementation against the live
-Trino stack. It checks `TableSource`, `Filter`, `Projection`, `Join`,
-`GlobalReduce`, `ReduceBy`, `Sort`, and `TableSink`, and confirms that the
-expected SQL reached Trino. The standalone join test now runs a full Wayang
-plan and normalizes both possible join result shapes before collecting records:
-logical joins can produce `Tuple2<Record, Record>`, while pushed-down JDBC joins
-can return a flat `Record`.
-
-The suite is self-contained: it creates `iceberg.wayang_it`, scales its test
-data to 120,000 rows so the optimizer selects SQL pushdown, and drops its test
-tables afterward. It does not require `scripts/init.sql`. The suite also
-contains five JavaPlanBuilder `readTable` combination tests that cover filter,
-projection, global reduce, reduce-by plus sort, table sink, and join through
-the public API.
-
-```bash
-./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am \
- -Dtest=TrinoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \
- -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am -Dtest=TrinoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test
-```
-
-Expected result:
-
-```text
-Tests run: 13, Failures: 0, Errors: 0, Skipped: 0
-```
-
-Verified on June 18, 2026 against the local Docker stack with the full-plan
-join test and all five JavaPlanBuilder combination tests enabled.
-
-If Trino is unreachable, these tests are skipped instead of failed. A result
-with skipped tests does not confirm that the operators work.
-
-### 3. Load sample Iceberg data
-
-```bash
-bash platforms-setup-guides/trino-setup/scripts/run-init.sh
-```
-
-On PowerShell:
-
-```powershell
-Get-Content -Raw platforms-setup-guides/trino-setup/scripts/init.sql | docker exec -i trino trino --server http://localhost:8080 --user admin
-```
-
-This creates the schema `iceberg.sales` and inserts 20 sample orders into
-`iceberg.sales.orders` (Parquet files on MinIO).
-
-### 4. Run the walkthrough demo
-
-The optional demo script starts the local Trino/Iceberg stack, seeds
-`iceberg.sales.orders`, shows direct Trino CLI queries, and then runs
-`org.apache.wayang.trino.TrinoDemo` to demonstrate Wayang filter and projection
-pushdown through the Trino platform.
-
-```bash
-bash platforms-setup-guides/trino-setup/demo.sh
-```
-
-Set `WAYANG_DEMO_AUTO=true` to skip the interactive pauses.
-
-### 5. Run the standalone stack integration tests
-
-```bash
-./mvnw -f platforms-setup-guides/trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -f platforms-setup-guides/trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test
-```
-
-Tests are skipped by default (no `-Pintegration`) to avoid requiring Docker in CI.
-These tests validate the stack and direct JDBC queries independently of the
-Wayang operator implementation.
-
-Expected result:
-
-```text
-Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
-BUILD SUCCESS
-```
-
-### 6. Manual exploration
-
-Open the **Trino UI**: http://localhost:8080
-
-Or connect via the Trino CLI inside the container:
-
-```bash
-docker exec -it trino trino --catalog iceberg --schema sales
-```
-
-```sql
--- TPC-H built-in data (no init.sql needed)
-SELECT * FROM tpch.tiny.orders LIMIT 5;
-
--- Iceberg table
-SELECT region, SUM(amount) FROM iceberg.sales.orders GROUP BY region;
-
--- Iceberg file metadata
-SELECT * FROM iceberg.sales."orders$files";
-
--- Iceberg history
-SELECT * FROM iceberg.sales."orders$history";
-```
-
-**MinIO console**: http://localhost:9001 (login: `minioadmin` / `minioadmin`)
-Look for Parquet files under `warehouse/sales/orders/`.
-
-### 7. Tear down
-
-```bash
-docker compose -f platforms-setup-guides/trino-setup/docker-compose.yml down -v
-```
-
-The `-v` option removes volumes and clears the local MinIO and PostgreSQL data.
-
-## Test Coverage
-
-### Wayang operator integration tests
-
-| Test | What it checks |
-|------|----------------|
-| `tableSource` | Full table scan through `TrinoTableSource` |
-| `filter` | Wayang `FilterOperator` and SQL `WHERE` pushdown |
-| `projection` | Column projection pushed into the Trino query |
-| `join` | Full Wayang join plan with normalization before the collecting sink |
-| `globalReduce` | Global aggregation such as `SUM` |
-| `reduceBy` | Grouped aggregation and SQL `GROUP BY` |
-| `sort` | Wayang sort and SQL `ORDER BY` |
-| `tableSink` | Filtered result written with `CREATE TABLE AS` |
-| `javaPlanBuilderReadTableFilterProjection` | `readTable -> filter -> projection -> collect` |
-| `javaPlanBuilderReadTableFilterGlobalReduce` | `readTable -> filter -> globalReduce -> collect` |
-| `javaPlanBuilderReadTableReduceBySort` | `readTable -> reduceByKey -> sort -> collect` |
-| `javaPlanBuilderReadTableFilterProjectionTableSink` | `readTable -> filter -> projection -> writeTable` |
-| `javaPlanBuilderReadTableJoin` | `readTable + readTable -> join -> collect` |
-
-### Standalone stack integration tests
-
-| Test | What it checks |
-|------|----------------|
-| `testConnectivity` | `SELECT 1`, JDBC connection works |
-| `testTpchConnector` | TPC-H built-in connector, no storage needed |
-| `testTpchTopOrders` | ORDER BY + LIMIT on TPC-H |
-| `testIcebergSchemaVisible` | Schema created by `init.sql` is visible |
-| `testIcebergSelectAll` | Full table scan, 20 rows |
-| `testIcebergFilterByRegion` | WHERE pushdown on string column |
-| `testIcebergAggregate` | GROUP BY + SUM aggregation |
-| `testIcebergFilterByAmount` | WHERE pushdown on double column |
-| `testIcebergProjection` | SELECT subset of columns |
-| `testIcebergFilesMetadata` | `$files` system table, confirms Parquet on MinIO |
-
-## Environment Variables
-
-Override defaults if running Trino on a different host/port:
-
-```bash
-TRINO_HOST=my-trino-host TRINO_PORT=8080 ./mvnw -f platforms-setup-guides/trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test
-```
-
-On PowerShell:
-
-```powershell
-$env:TRINO_HOST="my-trino-host"
-$env:TRINO_PORT="8080"
-.\mvnw.cmd --% -f platforms-setup-guides/trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test
-Remove-Item Env:TRINO_HOST, Env:TRINO_PORT
-```
diff --git a/platforms-setup-guides/trino-setup/demo.sh b/platforms-setup-guides/trino-setup/demo.sh
deleted file mode 100644
index 50b2f0018..000000000
--- a/platforms-setup-guides/trino-setup/demo.sh
+++ /dev/null
@@ -1,127 +0,0 @@
-#!/usr/bin/env bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-WAYANG_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
-TRINO_SETUP="$SCRIPT_DIR"
-TRINO_CONTAINER="trino"
-MAVEN_FLAGS="-Pskip-prerequisite-check -Drat.skip=true -Dmaven.javadoc.skip=true"
-
-banner() {
- echo
- echo "============================================================"
- printf " %s\n" "$*"
- echo "============================================================"
- echo
-}
-
-step() {
- echo
- echo "-- $*"
- echo
-}
-
-pause() {
- if [[ "${WAYANG_DEMO_AUTO:-false}" != "true" ]]; then
- echo
- read -rp "Press ENTER to continue..." _ || true
- echo
- fi
-}
-
-run_wayang_demo() {
- "$WAYANG_ROOT/mvnw" exec:java -pl wayang-platforms/wayang-trino \
- -Dexec.mainClass="org.apache.wayang.trino.TrinoDemo" \
- ${MAVEN_FLAGS}
-}
-
-banner "ACT 1: Start Trino + Iceberg via Docker"
-
-step "1a. Starting the stack"
-cd "$TRINO_SETUP"
-docker compose up -d
-
-step "1b. Containers running"
-docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}" \
- | grep -E "NAMES|trino|minio|metastore|postgres"
-
-step "1c. Waiting for Trino to be ready"
-MAX_WAIT=90
-ELAPSED=0
-until docker exec "$TRINO_CONTAINER" \
- trino --execute "SELECT 1" --output-format ALIGNED >/dev/null 2>&1; do
- if [[ "$ELAPSED" -ge "$MAX_WAIT" ]]; then
- echo "Timed out waiting for Trino after ${MAX_WAIT}s"
- exit 1
- fi
- printf ". waiting (%ds elapsed)\r" "$ELAPSED"
- sleep 3
- ELAPSED=$((ELAPSED + 3))
-done
-echo "Trino is ready at http://localhost:8080"
-
-step "1d. Initialising Iceberg tables"
-docker exec -i "$TRINO_CONTAINER" trino < "$TRINO_SETUP/scripts/init.sql" 2>&1 \
- | grep -v "^WARNING\|jline\|org.jline" || true
-echo "iceberg.sales.orders seeded"
-
-step "1e. Table schema"
-docker exec "$TRINO_CONTAINER" \
- trino --execute "DESCRIBE iceberg.sales.orders" \
- --output-format ALIGNED 2>/dev/null
-
-pause
-
-banner "ACT 2: Query Iceberg directly via Trino CLI"
-
-step "2a. Full table scan"
-echo "SQL: SELECT * FROM iceberg.sales.orders"
-docker exec "$TRINO_CONTAINER" \
- trino --execute "SELECT * FROM iceberg.sales.orders ORDER BY order_id" \
- --output-format ALIGNED 2>/dev/null
-
-step "2b. Filter: region = 'AMER'"
-echo "SQL: SELECT * FROM iceberg.sales.orders WHERE region = 'AMER'"
-docker exec "$TRINO_CONTAINER" \
- trino --execute "SELECT * FROM iceberg.sales.orders WHERE region = 'AMER' ORDER BY order_id" \
- --output-format ALIGNED 2>/dev/null
-
-step "2c. Projection with filter"
-echo "SQL: SELECT region, product, amount FROM iceberg.sales.orders WHERE region = 'AMER'"
-docker exec "$TRINO_CONTAINER" \
- trino --execute \
- "SELECT region, product, amount
- FROM iceberg.sales.orders
- WHERE region = 'AMER'
- ORDER BY order_id" \
- --output-format ALIGNED 2>/dev/null
-
-pause
-
-banner "ACT 3: Wayang API filter + projection pushdown"
-cd "$WAYANG_ROOT"
-run_wayang_demo
-
-banner "Demo complete"
-echo "Trino UI: http://localhost:8080"
-echo "MinIO UI: http://localhost:9001 (minioadmin / minioadmin)"
-echo
-echo "To stop the stack:"
-echo " cd platforms-setup-guides/trino-setup && docker compose down"
diff --git a/platforms-setup-guides/trino-setup/docker-compose.yml b/platforms-setup-guides/trino-setup/docker-compose.yml
deleted file mode 100644
index cf582184e..000000000
--- a/platforms-setup-guides/trino-setup/docker-compose.yml
+++ /dev/null
@@ -1,142 +0,0 @@
----
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Stack: Trino + Hive Metastore + MinIO (S3 storage)
-#
-# This is the battle-tested Trino + Iceberg local setup.
-# Hive Metastore (HMS) stores Iceberg table metadata over Thrift on port 9083.
-# MinIO provides S3-compatible object storage for Parquet data files.
-# Trino's Iceberg connector uses HMS as catalog and writes Parquet to MinIO.
-#
-# Ports:
-# Trino: http://localhost:8080 (UI + JDBC)
-# MinIO S3: http://localhost:9000
-# MinIO UI: http://localhost:9001 (minioadmin / minioadmin)
-# HMS: localhost:9083 (Thrift, internal)
-# Postgres: localhost:5432 (HMS backing store)
-
-services:
-
- # PostgreSQL (Hive Metastore backing database)
- postgres:
- image: postgres:15-alpine
- container_name: trino-postgres
- environment:
- POSTGRES_DB: metastore
- POSTGRES_USER: hive
- POSTGRES_PASSWORD: hive
- ports:
- - "5432:5432"
- volumes:
- - postgres-data:/var/lib/postgresql/data
- healthcheck:
- test: ["CMD-SHELL", "pg_isready -U hive -d metastore"]
- interval: 10s
- timeout: 5s
- retries: 5
-
- # MinIO (S3-compatible object storage)
- minio:
- image: minio/minio:latest
- container_name: trino-minio
- environment:
- MINIO_ROOT_USER: minioadmin
- MINIO_ROOT_PASSWORD: minioadmin
- ports:
- - "9000:9000"
- - "9001:9001"
- command: server /data --console-address ":9001"
- volumes:
- - minio-data:/data
- healthcheck:
- test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
- interval: 10s
- timeout: 5s
- retries: 5
-
- # Create the warehouse bucket before HMS starts
- minio-init:
- image: minio/mc:latest
- container_name: trino-minio-init
- depends_on:
- minio:
- condition: service_healthy
- entrypoint: >
- /bin/sh -c "
- mc alias set local http://minio:9000 minioadmin minioadmin;
- mc mb local/warehouse --ignore-existing;
- echo 'bucket warehouse ready';
- exit 0;
- "
-
- # Hive Metastore
- # naushadh/hive-metastore is a minimal, pre-configured HMS image
- # that supports S3-compatible storage via env vars.
- metastore:
- image: naushadh/hive-metastore:latest
- container_name: trino-metastore
- depends_on:
- postgres:
- condition: service_healthy
- minio:
- condition: service_healthy
- minio-init:
- condition: service_completed_successfully
- ports:
- - "9083:9083"
- environment:
- DATABASE_HOST: postgres
- DATABASE_DB: metastore
- DATABASE_USER: hive
- DATABASE_PASSWORD: hive
- # S3 / MinIO
- S3_ENDPOINT_URL: http://minio:9000
- S3_BUCKET: warehouse
- S3_PREFIX: /
- AWS_ACCESS_KEY_ID: minioadmin
- AWS_SECRET_ACCESS_KEY: minioadmin
- REGION: us-east-1
- # No nc/curl in this image; use bash's /dev/tcp built-in
- healthcheck:
- test: ["CMD", "/bin/bash", "-c", "exec 3<>/dev/tcp/localhost/9083 2>/dev/null && exit 0 || exit 1"]
- interval: 15s
- timeout: 10s
- retries: 15
-
- # Trino
- trino:
- image: trinodb/trino:435
- container_name: trino
- depends_on:
- metastore:
- condition: service_healthy
- minio:
- condition: service_healthy
- ports:
- - "8080:8080"
- volumes:
- - ./trino/catalog:/etc/trino/catalog
- - ./trino/config.properties:/etc/trino/config.properties
- healthcheck:
- test: ["CMD", "curl", "-f", "http://localhost:8080/v1/info"]
- interval: 15s
- timeout: 10s
- retries: 10
-
-volumes:
- postgres-data:
- minio-data:
diff --git a/platforms-setup-guides/trino-setup/pom.xml b/platforms-setup-guides/trino-setup/pom.xml
deleted file mode 100644
index c72cc7751..000000000
--- a/platforms-setup-guides/trino-setup/pom.xml
+++ /dev/null
@@ -1,95 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
- http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <modelVersion>4.0.0</modelVersion>
-
- <groupId>org.apache.wayang</groupId>
- <artifactId>trino-setup</artifactId>
- <version>1.0-SNAPSHOT</version>
- <packaging>jar</packaging>
-
- <name>Trino Local Setup - Integration Tests</name>
- <description>
- Standalone integration tests for a local Trino stack
- (Trino + Nessie Iceberg catalog + MinIO S3 storage).
- Independent of the Wayang codebase.
- </description>
-
- <properties>
- <maven.compiler.source>17</maven.compiler.source>
- <maven.compiler.target>17</maven.compiler.target>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <trino.version>435</trino.version>
- <junit.version>5.10.2</junit.version>
- <skipIntegrationTests>true</skipIntegrationTests>
- </properties>
-
- <dependencies>
- <!-- Trino JDBC driver -->
- <dependency>
- <groupId>io.trino</groupId>
- <artifactId>trino-jdbc</artifactId>
- <version>${trino.version}</version>
- <scope>test</scope>
- </dependency>
-
- <!-- JUnit 5 -->
- <dependency>
- <groupId>org.junit.jupiter</groupId>
- <artifactId>junit-jupiter</artifactId>
- <version>${junit.version}</version>
- <scope>test</scope>
- </dependency>
-
- <!-- Logging -->
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-simple</artifactId>
- <version>2.0.12</version>
- <scope>test</scope>
- </dependency>
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <version>3.2.5</version>
- <configuration>
- <!-- Skip integration tests by default; run with -Pintegration -->
- <skipTests>${skipIntegrationTests}</skipTests>
- </configuration>
- </plugin>
- </plugins>
- </build>
-
- <profiles>
- <!-- Run integration tests: mvn test -Pintegration -->
- <profile>
- <id>integration</id>
- <properties>
- <skipIntegrationTests>false</skipIntegrationTests>
- </properties>
- </profile>
- </profiles>
-</project>
diff --git a/platforms-setup-guides/trino-setup/scripts/init.sql b/platforms-setup-guides/trino-setup/scripts/init.sql
deleted file mode 100644
index 3ce74bcf8..000000000
--- a/platforms-setup-guides/trino-setup/scripts/init.sql
+++ /dev/null
@@ -1,66 +0,0 @@
--- Licensed to the Apache Software Foundation (ASF) under one
--- or more contributor license agreements. See the NOTICE file
--- distributed with this work for additional information
--- regarding copyright ownership. The ASF licenses this file
--- to you under the Apache License, Version 2.0 (the
--- "License"); you may not use this file except in compliance
--- with the License. You may obtain a copy of the License at
---
--- http://www.apache.org/licenses/LICENSE-2.0
---
--- Unless required by applicable law or agreed to in writing, software
--- distributed under the License is distributed on an "AS IS" BASIS,
--- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
--- See the License for the specific language governing permissions and
--- limitations under the License.
-
--- Run this after the stack is up to create sample Iceberg tables.
--- Usage: ./scripts/run-init.sh
--- Or manually: docker exec -it trino trino < /scripts/init.sql
-
--- ── Schema ────────────────────────────────────────────────────────────────
-CREATE SCHEMA IF NOT EXISTS iceberg.sales;
-
--- ── Orders table (Iceberg / Parquet on MinIO) ─────────────────────────────
-CREATE TABLE IF NOT EXISTS iceberg.sales.orders (
- order_id BIGINT,
- region VARCHAR,
- product VARCHAR,
- amount DOUBLE,
- order_date DATE
-)
-WITH (format = 'PARQUET');
-
--- ── Idempotent seed: clear before inserting so re-runs don't duplicate rows ─
-DELETE FROM iceberg.sales.orders;
-
--- ── Sample data: 20 rows, 4 regions (AMER/APAC/EMEA/LATAM), 5 products ────
--- AMER rows: 3, 6, 9, 12, 16 → 5 rows for filter demo
--- Projection demo selects only: region, product, amount
-INSERT INTO iceberg.sales.orders VALUES
- (1, 'APAC', 'Widget A', 1500.00, DATE '2024-01-15'),
- (2, 'EMEA', 'Widget B', 800.50, DATE '2024-01-16'),
- (3, 'AMER', 'Widget A', 2200.00, DATE '2024-01-17'),
- (4, 'APAC', 'Widget C', 350.75, DATE '2024-01-18'),
- (5, 'EMEA', 'Widget A', 1100.00, DATE '2024-01-19'),
- (6, 'AMER', 'Widget B', 950.25, DATE '2024-01-20'),
- (7, 'APAC', 'Widget B', 1750.00, DATE '2024-01-21'),
- (8, 'EMEA', 'Widget C', 420.00, DATE '2024-01-22'),
- (9, 'AMER', 'Widget C', 680.50, DATE '2024-01-23'),
- (10, 'APAC', 'Widget A', 3000.00, DATE '2024-01-24'),
- (11, 'LATAM', 'Widget D', 560.00, DATE '2024-01-25'),
- (12, 'AMER', 'Widget D', 1320.75, DATE '2024-01-26'),
- (13, 'EMEA', 'Widget D', 990.00, DATE '2024-01-27'),
- (14, 'LATAM', 'Widget E', 2100.50, DATE '2024-01-28'),
- (15, 'APAC', 'Widget E', 4500.00, DATE '2024-01-29'),
- (16, 'AMER', 'Widget E', 3750.00, DATE '2024-01-30'),
- (17, 'EMEA', 'Widget E', 1250.00, DATE '2024-01-31'),
- (18, 'LATAM', 'Widget A', 870.25, DATE '2024-02-01'),
- (19, 'APAC', 'Widget D', 1680.00, DATE '2024-02-02'),
- (20, 'LATAM', 'Widget B', 440.50, DATE '2024-02-03');
-
--- ── Verify ────────────────────────────────────────────────────────────────
-SELECT region, COUNT(*) AS order_count, SUM(amount) AS total_amount
-FROM iceberg.sales.orders
-GROUP BY region
-ORDER BY total_amount DESC;
diff --git a/platforms-setup-guides/trino-setup/scripts/run-init.sh b/platforms-setup-guides/trino-setup/scripts/run-init.sh
deleted file mode 100644
index ebaeb337c..000000000
--- a/platforms-setup-guides/trino-setup/scripts/run-init.sh
+++ /dev/null
@@ -1,40 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Runs init.sql against the local Trino instance.
-# The stack must be fully up before running this.
-
-set -e
-
-TRINO_HOST=${TRINO_HOST:-localhost}
-TRINO_PORT=${TRINO_PORT:-8080}
-
-echo "Waiting for Trino to be ready..."
-until curl -sf "http://${TRINO_HOST}:${TRINO_PORT}/v1/info" | grep -q '"starting":false'; do
- echo " Trino not ready yet, retrying in 5s..."
- sleep 5
-done
-echo "Trino is ready."
-
-echo "Running init.sql..."
-docker exec -i trino trino \
- --server "http://${TRINO_HOST}:${TRINO_PORT}" \
- --user admin \
- < "$(dirname "$0")/init.sql"
-
-echo "Done. Sample Iceberg data loaded into iceberg.sales.orders"
diff --git a/platforms-setup-guides/trino-setup/src/test/java/org/apache/wayang/trino/TrinoIntegrationTest.java b/platforms-setup-guides/trino-setup/src/test/java/org/apache/wayang/trino/TrinoIntegrationTest.java
deleted file mode 100644
index 3c2735b82..000000000
--- a/platforms-setup-guides/trino-setup/src/test/java/org/apache/wayang/trino/TrinoIntegrationTest.java
+++ /dev/null
@@ -1,233 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.wayang.trino;
-
-import org.junit.jupiter.api.*;
-
-import java.sql.*;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Properties;
-
-import static org.junit.jupiter.api.Assertions.*;
-
-/**
- * Integration tests for the local Trino stack.
- *
- * Prerequisites: run `docker-compose up -d` and `./scripts/run-init.sh` first.
- *
- * Run tests:
- * mvn test -Pintegration
- *
- * Or skip infrastructure setup and run with a custom host:
- * TRINO_HOST=localhost TRINO_PORT=8080 mvn test -Pintegration
- */
-@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
-class TrinoIntegrationTest {
-
- private static final String TRINO_HOST = System.getenv().getOrDefault("TRINO_HOST", "localhost");
- private static final int TRINO_PORT = Integer.parseInt(System.getenv().getOrDefault("TRINO_PORT", "8080"));
- private static final String JDBC_URL = String.format("jdbc:trino://%s:%d", TRINO_HOST, TRINO_PORT);
-
- private static Connection connection;
-
- // ── Lifecycle ─────────────────────────────────────────────────────────
-
- @BeforeAll
- static void openConnection() throws Exception {
- Properties props = new Properties();
- props.setProperty("user", "admin"); // Trino requires a non-empty user
- connection = DriverManager.getConnection(JDBC_URL, props);
- System.out.printf("Connected to Trino at %s%n", JDBC_URL);
- }
-
- @AfterAll
- static void closeConnection() throws Exception {
- if (connection != null && !connection.isClosed()) {
- connection.close();
- }
- }
-
- // ── Helper ────────────────────────────────────────────────────────────
-
- private List<List<Object>> query(String sql) throws SQLException {
- List<List<Object>> rows = new ArrayList<>();
- try (Statement stmt = connection.createStatement();
- ResultSet rs = stmt.executeQuery(sql)) {
- int cols = rs.getMetaData().getColumnCount();
- while (rs.next()) {
- List<Object> row = new ArrayList<>();
- for (int i = 1; i <= cols; i++) row.add(rs.getObject(i));
- rows.add(row);
- }
- }
- return rows;
- }
-
- // ── Test 1: Basic connectivity ────────────────────────────────────────
-
- @Test
- @Order(1)
- @DisplayName("Trino responds to a simple SELECT 1")
- void testConnectivity() throws SQLException {
- List<List<Object>> rows = query("SELECT 1");
- assertEquals(1, rows.size());
- assertEquals(1L, ((Number) rows.get(0).get(0)).longValue());
- System.out.println("[PASS] Basic connectivity OK");
- }
-
- // ── Test 2: TPC-H built-in connector ─────────────────────────────────
-
- @Test
- @Order(2)
- @DisplayName("TPC-H tiny catalog: count orders")
- void testTpchConnector() throws SQLException {
- List<List<Object>> rows = query("SELECT COUNT(*) FROM tpch.tiny.orders");
- long count = ((Number) rows.get(0).get(0)).longValue();
- assertTrue(count > 0, "tpch.tiny.orders should have rows");
- System.out.printf("[PASS] TPC-H tiny.orders has %,d rows%n", count);
- }
-
- @Test
- @Order(3)
- @DisplayName("TPC-H tiny catalog: top 5 orders by total price")
- void testTpchTopOrders() throws SQLException {
- List<List<Object>> rows = query("""
- SELECT orderkey, totalprice
- FROM tpch.tiny.orders
- ORDER BY totalprice DESC
- LIMIT 5
- """);
- assertEquals(5, rows.size(), "Expected exactly 5 rows");
- System.out.println("[PASS] TPC-H top 5 orders:");
- rows.forEach(r -> System.out.printf(" orderkey=%s totalprice=%s%n", r.get(0), r.get(1)));
- }
-
- // ── Test 4: Iceberg — schema exists ──────────────────────────────────
-
- @Test
- @Order(4)
- @DisplayName("Iceberg catalog: schema 'sales' is visible")
- void testIcebergSchemaVisible() throws SQLException {
- List<List<Object>> rows = query("SHOW SCHEMAS IN iceberg LIKE 'sales'");
- assertFalse(rows.isEmpty(), "Schema 'sales' should exist in iceberg catalog. " +
- "Did you run scripts/run-init.sh?");
- System.out.println("[PASS] Iceberg schema 'sales' is visible");
- }
-
- // ── Test 5: Iceberg — full table scan ────────────────────────────────
-
- @Test
- @Order(5)
- @DisplayName("Iceberg table: select all orders")
- void testIcebergSelectAll() throws SQLException {
- List<List<Object>> rows = query("SELECT * FROM iceberg.sales.orders ORDER BY order_id");
- assertEquals(20, rows.size(), "Expected 20 rows inserted by init.sql");
- System.out.println("[PASS] Iceberg full scan: 20 rows");
- rows.forEach(r -> System.out.printf(" %s%n", r));
- }
-
- // ── Test 6: Iceberg — pushdown filter ────────────────────────────────
-
- @Test
- @Order(6)
- @DisplayName("Iceberg table: filter by region = APAC")
- void testIcebergFilterByRegion() throws SQLException {
- List<List<Object>> rows = query("""
- SELECT order_id, region, amount
- FROM iceberg.sales.orders
- WHERE region = 'APAC'
- ORDER BY order_id
- """);
- assertFalse(rows.isEmpty(), "Should have APAC orders");
- rows.forEach(r -> assertEquals("APAC", r.get(1), "All rows must be APAC"));
- System.out.printf("[PASS] Filter pushdown: %d APAC rows%n", rows.size());
- }
-
- // ── Test 7: Iceberg — aggregation ────────────────────────────────────
-
- @Test
- @Order(7)
- @DisplayName("Iceberg table: aggregate total_amount by region")
- void testIcebergAggregate() throws SQLException {
- List<List<Object>> rows = query("""
- SELECT region, COUNT(*) AS order_count, SUM(amount) AS total_amount
- FROM iceberg.sales.orders
- GROUP BY region
- ORDER BY total_amount DESC
- """);
- assertFalse(rows.isEmpty(), "Aggregation should return rows");
- System.out.println("[PASS] Aggregation by region:");
- rows.forEach(r -> System.out.printf(" region=%-5s count=%s total=%.2f%n",
- r.get(0), r.get(1), ((Number) r.get(2)).doubleValue()));
- }
-
- // ── Test 8: Iceberg — amount threshold filter ─────────────────────────
-
- @Test
- @Order(8)
- @DisplayName("Iceberg table: filter orders with amount > 1000")
- void testIcebergFilterByAmount() throws SQLException {
- List<List<Object>> rows = query("""
- SELECT order_id, amount
- FROM iceberg.sales.orders
- WHERE amount > 1000.0
- ORDER BY amount DESC
- """);
- rows.forEach(r -> assertTrue(
- ((Number) r.get(1)).doubleValue() > 1000.0,
- "All rows must have amount > 1000"
- ));
- System.out.printf("[PASS] Amount filter: %d rows with amount > 1000%n", rows.size());
- }
-
- // ── Test 9: Iceberg — projection (select subset of columns) ───────────
-
- @Test
- @Order(9)
- @DisplayName("Iceberg table: project only region and product columns")
- void testIcebergProjection() throws SQLException {
- List<List<Object>> rows = query("""
- SELECT region, product
- FROM iceberg.sales.orders
- LIMIT 5
- """);
- assertEquals(5, rows.size());
- rows.forEach(r -> {
- assertNotNull(r.get(0), "region should not be null");
- assertNotNull(r.get(1), "product should not be null");
- });
- System.out.println("[PASS] Projection (region, product): 5 rows returned");
- }
-
- // ── Test 10: Iceberg metadata — table files ──────────────────────────
-
- @Test
- @Order(10)
- @DisplayName("Iceberg metadata: $files table lists at least one Parquet file")
- void testIcebergFilesMetadata() throws SQLException {
- List<List<Object>> rows = query("""
- SELECT file_path, record_count
- FROM iceberg.sales."orders$files"
- """);
- assertFalse(rows.isEmpty(), "There should be at least one data file after inserts");
- System.out.println("[PASS] Iceberg $files metadata:");
- rows.forEach(r -> System.out.printf(" %s (records=%s)%n", r.get(0), r.get(1)));
- }
-}
diff --git a/platforms-setup-guides/trino-setup/trino/catalog/iceberg.properties b/platforms-setup-guides/trino-setup/trino/catalog/iceberg.properties
deleted file mode 100644
index ed05c180a..000000000
--- a/platforms-setup-guides/trino-setup/trino/catalog/iceberg.properties
+++ /dev/null
@@ -1,32 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Iceberg catalog backed by Hive Metastore (HMS) + MinIO (S3 storage).
-# HMS stores Iceberg table metadata; Trino writes Parquet files to MinIO via S3.
-connector.name=iceberg
-iceberg.catalog.type=hive_metastore
-hive.metastore.uri=thrift://metastore:9083
-
-# Native S3 filesystem — handles both s3:// and s3a:// (which HMS uses internally).
-# This avoids the "No FileSystem for scheme s3" error when HMS assigns locations.
-fs.native-s3.enabled=true
-s3.endpoint=http://minio:9000
-s3.path-style-access=true
-s3.aws-access-key=minioadmin
-s3.aws-secret-key=minioadmin
-s3.region=us-east-1
-
-iceberg.file-format=PARQUET
diff --git a/platforms-setup-guides/trino-setup/trino/catalog/tpch.properties b/platforms-setup-guides/trino-setup/trino/catalog/tpch.properties
deleted file mode 100644
index 064863fc7..000000000
--- a/platforms-setup-guides/trino-setup/trino/catalog/tpch.properties
+++ /dev/null
@@ -1,20 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Built-in TPC-H connector – no external dependencies.
-# Useful for testing basic Trino connectivity without any storage setup.
-# Usage: SELECT * FROM tpch.tiny.orders LIMIT 10;
-connector.name=tpch
diff --git a/platforms-setup-guides/trino-setup/trino/config.properties b/platforms-setup-guides/trino-setup/trino/config.properties
deleted file mode 100644
index 8584e4aea..000000000
--- a/platforms-setup-guides/trino-setup/trino/config.properties
+++ /dev/null
@@ -1,20 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-coordinator=true
-node-scheduler.include-coordinator=true
-http-server.http.port=8080
-discovery.uri=http://localhost:8080
diff --git a/presto-setup/README.md b/presto-setup/README.md
deleted file mode 100644
index b5e4bf4dd..000000000
--- a/presto-setup/README.md
+++ /dev/null
@@ -1,248 +0,0 @@
-# Presto Local Setup
-
-Local PrestoDB environment using the built-in **memory** connector, completely
-containerised.
-
-The current validation has two parts:
-
-1. Build the Wayang Presto platform and run the shared JDBC SQL-generation tests.
-2. Run the Wayang Presto operator tests against the live local PrestoDB.
-
-Run the commands below from the repository root. Java 17 and Docker with Docker
-Compose are required; Maven is provided by the repository wrapper.
-
-## Command Conventions
-
-Use the `bash` blocks on macOS/Linux terminals. Use the `powershell` blocks on
-Windows PowerShell from the repository root. Docker Compose commands are the
-same on both platforms.
-
-## Stack
-
-| Component | Image | Port | Role |
-|-----------|-------|------|------|
-| **PrestoDB** | `prestodb/presto:0.289` | 8081 | SQL engine and in-memory test storage |
-
-The container listens on port `8080`; Docker exposes it as `8081` to avoid
-clashing with the Trino setup. The `memory` connector needs no metastore,
-database, or object storage. All tables disappear when the container stops.
-
-## Directory Layout
-
-```text
-presto-setup/
-|-- docker-compose.yml
-|-- README.md
-`-- etc/
- `-- catalog/
- `-- memory.properties
-
-wayang-platforms/wayang-presto/src/test/java/.../
-|-- PrestoOperatorsIT.java
-`-- PrestoCostPilotIT.java
-```
-
-## 1. Test the Wayang Presto Platform
-
-Build the Presto platform and its required modules:
-
-```bash
-./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am \
- -DskipTests -Drat.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am -DskipTests -Drat.skip=true test
-```
-
-Then run the shared JDBC SQL-generation tests:
-
-```bash
-./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am \
- -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false \
- -DfailIfNoTests=false -Drat.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test
-```
-
-Expected result:
-
-```text
-Wayang Platform Presto ... SUCCESS
-Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
-```
-
-## 2. Test Against the Local Presto Stack
-
-### 1. Start Presto
-
-```bash
-docker compose -f presto-setup/docker-compose.yml up -d --wait
-```
-
-Presto can take 20-60 seconds to accept queries. Confirm that it is healthy:
-
-```bash
-docker compose -f presto-setup/docker-compose.yml ps
-```
-
-The Presto web UI is available at <http://localhost:8081>.
-
-### 2. Run the Wayang Presto operator tests
-
-`PrestoOperatorsIT` exercises the Wayang Presto implementation against the live
-container. It checks `TableSource`, `Filter`, `Projection`, `Join`,
-`GlobalReduce`, `ReduceBy`, `Sort`, and `TableSink`, and confirms that the
-expected SQL reached Presto through `system.runtime.queries`.
-
-The standalone join test now runs as a full Wayang plan:
-`PrestoTableSource + PrestoTableSource -> JoinOperator -> MapOperator -> sink`.
-The normalization map accepts both logical `Tuple2<Record, Record>` output and
-pushed-down JDBC flat `Record` output. The suite also includes five
-`JavaPlanBuilder.readTable` combination plans. Together, they cover every
-supported Presto operator through the public API.
-
-The suite is self-contained. It creates `memory.wayang_it`, generates 120,000
-rows so the optimizer selects SQL pushdown, runs the tests, and drops its tables
-afterward.
-
-```bash
-./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am \
- -Dtest=PrestoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \
- -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am -Dtest=PrestoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test
-```
-
-Successful validation must show:
-
-```text
-Tests run: 13, Failures: 0, Errors: 0, Skipped: 0
-BUILD SUCCESS
-```
-
-If Presto is unreachable, the tests are skipped instead of failed. A result
-with skipped tests does not confirm that the operators work. Errors while
-creating the test schema or tables are treated as real failures.
-
-### Verified Result
-
-On June 18, 2026, the suite completed successfully against the local PrestoDB
-0.289 container, including the full-plan join validation:
-
-```text
-[PrestoOperatorsIT] Connected to Presto at jdbc:presto://localhost:8081/memory
-Executed sql sink: CREATE TABLE memory.wayang_it.amer_orders AS SELECT ...
-Tests run: 13, Failures: 0, Errors: 0, Skipped: 0
-BUILD SUCCESS
-```
-
-This verified the complete `Wayang -> Presto JDBC -> live PrestoDB` path,
-including reads, SQL pushdown, join normalization, aggregation, sorting, and
-`CREATE TABLE AS SELECT`.
-
-### 3. Tear down
-
-```bash
-docker compose -f presto-setup/docker-compose.yml down
-```
-
-## Test Coverage
-
-| Test | What it checks |
-|------|----------------|
-| `tableSource` | Full table scan through `PrestoTableSource` |
-| `filter` | Wayang `FilterOperator` and SQL `WHERE` pushdown |
-| `projection` | Column projection pushed into the Presto query |
-| `join` | Full Wayang join plan with normalization before the sink table |
-| `globalReduce` | Global aggregation such as `SUM` |
-| `reduceBy` | Grouped aggregation and SQL `GROUP BY` |
-| `sort` | Wayang sort and SQL `ORDER BY` |
-| `tableSink` | Filtered result written with `CREATE TABLE AS` |
-| `javaPlanBuilderReadTableFilterProjection` | Public API filter and projection combination |
-| `javaPlanBuilderReadTableFilterGlobalReduce` | Public API filter and global aggregation combination |
-| `javaPlanBuilderReadTableReduceBySort` | Public API grouped aggregation and sort combination |
-| `javaPlanBuilderReadTableFilterProjectionTableSink` | Public API filtered projection written to a table |
-| `javaPlanBuilderReadTableJoin` | Public API two-table join with pushed-down record output |
-
-## Environment Variables
-
-Override the default endpoint when running against another PrestoDB:
-
-| Variable | Default |
-|----------|---------|
-| `PRESTO_HOST` | `localhost` |
-| `PRESTO_PORT` | `8081` |
-| `PRESTO_USER` | `test` |
-
-Example:
-
-```bash
-PRESTO_HOST=my-presto PRESTO_PORT=8080 PRESTO_USER=wayang \
- ./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am \
- -Dtest=PrestoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \
- -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-$env:PRESTO_HOST="my-presto"
-$env:PRESTO_PORT="8080"
-$env:PRESTO_USER="wayang"
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am -Dtest=PrestoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test
-Remove-Item Env:PRESTO_HOST, Env:PRESTO_PORT, Env:PRESTO_USER
-```
-
-## Cost Profiling
-
-Follow the shared cost-profiling guide in
-[`guides/cost-profiling.md`](../guides/cost-profiling.md). This setup guide
-only covers the Presto stack itself.
-
-Presto-specific profiling values:
-
-| Item | Value |
-|------|-------|
-| Maven module | `wayang-platforms/wayang-presto` |
-| Profiling test | `PrestoCostPilotIT` |
-| Property prefix | `presto.profile.*` |
-| Profiling schema | `memory.wayang_profile` |
-| Default output directory | `target/cost-profiling/presto` |
-| Learned parameters file | `wayang-platforms/wayang-presto/src/main/resources/wayang-presto-defaults.properties` |
-
-`PrestoCostPilotIT` uses the same `PRESTO_HOST`, `PRESTO_PORT`, and
-`PRESTO_USER` endpoint variables as `PrestoOperatorsIT`.
-
-## Troubleshooting
-
-### `Catalog does not exist: memory`
-
-Check that `presto-setup/etc/catalog/memory.properties` is a regular file
-before starting the container. If Docker first created the container while the
-source file was absent, it may have mounted a directory at the catalog path.
-Recreate the container after confirming the catalog file is present:
-
-```bash
-docker compose -f presto-setup/docker-compose.yml down
-docker compose -f presto-setup/docker-compose.yml up -d --force-recreate --wait
-```
-
-Confirm the mounted path inside the container is a file:
-
-```bash
-docker exec presto sh -c \
- "ls -l /opt/presto-server/etc/catalog/memory.properties"
-```
-
-Then rerun `PrestoOperatorsIT`.
diff --git a/presto-setup/docker-compose.yml b/presto-setup/docker-compose.yml
deleted file mode 100644
index 99d8e4fe1..000000000
--- a/presto-setup/docker-compose.yml
+++ /dev/null
@@ -1,47 +0,0 @@
----
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Stack: a single PrestoDB coordinator/worker with the in-memory connector.
-#
-# The `memory` connector supports CREATE SCHEMA / CREATE TABLE / INSERT / SELECT
-# entirely in-memory, so the integration test is fully self-contained — no Hive
-# metastore, object storage, or external catalog required.
-#
-# Ports:
-# Presto: http://localhost:8081 (UI + JDBC; container listens on 8080)
-#
-# The host port is 8081 to avoid clashing with the Trino stack (which uses 8080).
-
-services:
-
- presto:
- image: prestodb/presto:0.289
- container_name: presto
- ports:
- - "8081:8080"
- volumes:
- # Enable the in-memory connector by adding a catalog properties file.
- - ./etc/catalog/memory.properties:/opt/presto-server/etc/catalog/memory.properties
- # Presto needs ~20-60s before it accepts queries. Gate on /v1/info so callers
- # (and `docker compose up -d --wait`) can wait for readiness; the container
- # reports "Up" long before the coordinator is ready.
- healthcheck:
- test: ["CMD", "curl", "-f", "http://localhost:8080/v1/info"]
- interval: 10s
- timeout: 5s
- retries: 15
- start_period: 30s
diff --git a/presto-setup/etc/catalog/memory.properties b/presto-setup/etc/catalog/memory.properties
deleted file mode 100644
index 2accfc274..000000000
--- a/presto-setup/etc/catalog/memory.properties
+++ /dev/null
@@ -1,26 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# PrestoDB in-memory connector.
-# Backs the `memory` catalog used by the Wayang Presto integration tests:
-# tables created here (CREATE TABLE memory.<schema>.<table>) live in worker
-# memory and are dropped at teardown.
-connector.name=memory
-# Cap on-heap connector data per node. The image's default heap is -Xmx1G with
-# -XX:+ExitOnOutOfMemoryError, so keep this well under the heap; 256MB is ample
-# for the small, transient test tables. Raise the heap (custom jvm.config) before
-# increasing this if you reuse the stack for larger inserts.
-memory.max-data-per-node=256MB
diff --git a/trino-setup/README.md b/trino-setup/README.md
deleted file mode 100644
index 56dcca59b..000000000
--- a/trino-setup/README.md
+++ /dev/null
@@ -1,283 +0,0 @@
-# Trino Local Setup
-
-Local Trino environment backed by an **Iceberg** data lake, completely containerised.
-
-The current validation has three parts:
-
-1. Build the Wayang Trino platform and run the shared JDBC SQL-generation tests.
-2. Run the Wayang Trino operator tests against the live local stack.
-3. Run standalone JDBC integration tests against the local Trino, Iceberg, and MinIO stack.
-
-Run the commands below from the repository root. Java 17 and Docker with
-Docker Compose are required; Maven is provided by the repository wrapper.
-
-The Trino cost-profiling branch is named `feature/trino-cost-profiling`:
-
-```bash
-git checkout feature/trino-cost-profiling
-```
-
-## Command Conventions
-
-Use the `bash` blocks on macOS/Linux terminals. Use the `powershell` blocks on
-Windows PowerShell from the repository root. Docker Compose commands are the
-same on both platforms.
-
-## Stack
-
-| Component | Image | Port | Role |
-|-----------|-------|------|------|
-| **Trino** | `trinodb/trino:435` | 8080 | SQL query engine |
-| **Hive Metastore** | `naushadh/hive-metastore:latest` | 9083 | Iceberg table catalog (Thrift) |
-| **PostgreSQL** | `postgres:15-alpine` | 5432 | HMS metadata backing store |
-| **MinIO** | `minio/minio:latest` | 9000 / 9001 | S3-compatible object storage |
-
-HMS is the battle-tested Iceberg catalog for Trino. Parquet data files are written by Trino directly to MinIO; HMS only stores schema/table metadata.
-
-## Directory Layout
-
-```
-trino-setup/
-|-- docker-compose.yml # Full stack definition
-|-- trino/
-| |-- config.properties # Trino node config
-| `-- catalog/
-| |-- iceberg.properties # Iceberg via HMS + MinIO
-| `-- tpch.properties # Built-in TPC-H (no storage needed)
-|-- scripts/
-| |-- init.sql # Creates iceberg.sales.orders + sample rows
-| `-- run-init.sh # Helper: waits for Trino then runs init.sql
-|-- pom.xml # Standalone Maven project (Java 17)
-`-- src/test/java/.../
- `-- TrinoIntegrationTest.java # JUnit 5 integration tests
-```
-
-## 1. Test the Wayang Trino Platform
-
-Build the Trino platform and its required modules:
-
-```bash
-./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am -DskipTests -Drat.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am -DskipTests -Drat.skip=true test
-```
-
-Then run the shared JDBC SQL-generation tests:
-
-```bash
-./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test
-```
-
-Expected result:
-
-```text
-Wayang Platform Trino ... SUCCESS
-Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
-```
-
-## 2. Test Against the Local Trino Stack
-
-### 1. Start the stack
-
-```bash
-docker compose -f trino-setup/docker-compose.yml up -d
-```
-
-Wait ~30 seconds for all services to become healthy. Check with:
-
-```bash
-docker compose -f trino-setup/docker-compose.yml ps
-# or watch the Trino UI at http://localhost:8080
-```
-
-### 2. Run the Wayang Trino operator tests
-
-`TrinoOperatorsIT` exercises the Wayang Trino implementation against the live
-Trino stack. It checks `TableSource`, `Filter`, `Projection`, `Join`,
-`GlobalReduce`, `ReduceBy`, `Sort`, and `TableSink`, and confirms that the
-expected SQL reached Trino. The standalone join test now runs a full Wayang
-plan and normalizes both possible join result shapes before collecting records:
-logical joins can produce `Tuple2<Record, Record>`, while pushed-down JDBC joins
-can return a flat `Record`.
-
-The suite is self-contained: it creates `iceberg.wayang_it`, scales its test
-data to 120,000 rows so the optimizer selects SQL pushdown, and drops its test
-tables afterward. It does not require `scripts/init.sql`. The suite also
-contains five JavaPlanBuilder `readTable` combination tests that cover filter,
-projection, global reduce, reduce-by plus sort, table sink, and join through
-the public API.
-
-```bash
-./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am \
- -Dtest=TrinoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \
- -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am -Dtest=TrinoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test
-```
-
-Expected result:
-
-```text
-Tests run: 13, Failures: 0, Errors: 0, Skipped: 0
-```
-
-Verified on June 18, 2026 against the local Docker stack with the full-plan
-join test and all five JavaPlanBuilder combination tests enabled.
-
-If Trino is unreachable, these tests are skipped instead of failed. A result
-with skipped tests does not confirm that the operators work.
-
-### 3. Load sample Iceberg data
-
-```bash
-bash trino-setup/scripts/run-init.sh
-```
-
-On PowerShell:
-
-```powershell
-Get-Content -Raw trino-setup/scripts/init.sql | docker exec -i trino trino --server http://localhost:8080 --user admin
-```
-
-This creates the schema `iceberg.sales` and inserts 20 sample orders into
-`iceberg.sales.orders` (Parquet files on MinIO).
-
-### 4. Run the standalone stack integration tests
-
-```bash
-./mvnw -f trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test
-```
-
-On PowerShell:
-
-```powershell
-.\mvnw.cmd --% -f trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test
-```
-
-Tests are skipped by default (no `-Pintegration`) to avoid requiring Docker in CI.
-These tests validate the stack and direct JDBC queries independently of the
-Wayang operator implementation.
-
-Expected result:
-
-```text
-Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
-BUILD SUCCESS
-```
-
-### 5. Manual exploration
-
-Open the **Trino UI**: http://localhost:8080
-
-Or connect via the Trino CLI inside the container:
-
-```bash
-docker exec -it trino trino --catalog iceberg --schema sales
-```
-
-```sql
--- TPC-H built-in data (no init.sql needed)
-SELECT * FROM tpch.tiny.orders LIMIT 5;
-
--- Iceberg table
-SELECT region, SUM(amount) FROM iceberg.sales.orders GROUP BY region;
-
--- Iceberg file metadata
-SELECT * FROM iceberg.sales."orders$files";
-
--- Iceberg history
-SELECT * FROM iceberg.sales."orders$history";
-```
-
-**MinIO console**: http://localhost:9001 (login: `minioadmin` / `minioadmin`)
-Look for Parquet files under `warehouse/sales/orders/`.
-
-### 6. Tear down
-
-```bash
-docker compose -f trino-setup/docker-compose.yml down -v
-```
-
-The `-v` option removes volumes and clears the local MinIO and PostgreSQL data.
-
-## Test Coverage
-
-### Wayang operator integration tests
-
-| Test | What it checks |
-|------|----------------|
-| `tableSource` | Full table scan through `TrinoTableSource` |
-| `filter` | Wayang `FilterOperator` and SQL `WHERE` pushdown |
-| `projection` | Column projection pushed into the Trino query |
-| `join` | Full Wayang join plan with normalization before the collecting sink |
-| `globalReduce` | Global aggregation such as `SUM` |
-| `reduceBy` | Grouped aggregation and SQL `GROUP BY` |
-| `sort` | Wayang sort and SQL `ORDER BY` |
-| `tableSink` | Filtered result written with `CREATE TABLE AS` |
-| `javaPlanBuilderReadTableFilterProjection` | `readTable -> filter -> projection -> collect` |
-| `javaPlanBuilderReadTableFilterGlobalReduce` | `readTable -> filter -> globalReduce -> collect` |
-| `javaPlanBuilderReadTableReduceBySort` | `readTable -> reduceByKey -> sort -> collect` |
-| `javaPlanBuilderReadTableFilterProjectionTableSink` | `readTable -> filter -> projection -> writeTable` |
-| `javaPlanBuilderReadTableJoin` | `readTable + readTable -> join -> collect` |
-
-### Standalone stack integration tests
-
-| Test | What it checks |
-|------|----------------|
-| `testConnectivity` | `SELECT 1`, JDBC connection works |
-| `testTpchConnector` | TPC-H built-in connector, no storage needed |
-| `testTpchTopOrders` | ORDER BY + LIMIT on TPC-H |
-| `testIcebergSchemaVisible` | Schema created by `init.sql` is visible |
-| `testIcebergSelectAll` | Full table scan, 20 rows |
-| `testIcebergFilterByRegion` | WHERE pushdown on string column |
-| `testIcebergAggregate` | GROUP BY + SUM aggregation |
-| `testIcebergFilterByAmount` | WHERE pushdown on double column |
-| `testIcebergProjection` | SELECT subset of columns |
-| `testIcebergFilesMetadata` | `$files` system table, confirms Parquet on MinIO |
-
-## Cost Profiling
-
-Follow the shared cost-profiling guide in
-[`guides/cost-profiling.md`](../guides/cost-profiling.md). This setup guide
-only covers the Trino stack itself.
-
-Trino-specific profiling values:
-
-| Item | Value |
-|------|-------|
-| Maven module | `wayang-platforms/wayang-trino` |
-| Profiling test | `TrinoCostPilotIT` |
-| Property prefix | `trino.profile.*` |
-| Default output directory | `target/cost-profiling/trino` |
-| Learned parameters file | `wayang-platforms/wayang-trino/src/main/resources/wayang-trino-defaults.properties` |
-
-## Environment Variables
-
-Override defaults if running Trino on a different host/port:
-
-```bash
-TRINO_HOST=my-trino-host TRINO_PORT=8080 ./mvnw -f trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test
-```
-
-On PowerShell:
-
-```powershell
-$env:TRINO_HOST="my-trino-host"
-$env:TRINO_PORT="8080"
-.\mvnw.cmd --% -f trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test
-Remove-Item Env:TRINO_HOST, Env:TRINO_PORT
-```
diff --git a/trino-setup/demo.sh b/trino-setup/demo.sh
deleted file mode 100644
index ea3df7892..000000000
--- a/trino-setup/demo.sh
+++ /dev/null
@@ -1,126 +0,0 @@
-#!/usr/bin/env bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-WAYANG_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
-TRINO_SETUP="$SCRIPT_DIR"
-TRINO_CONTAINER="trino"
-MAVEN_FLAGS="-Pskip-prerequisite-check -Drat.skip=true -Dmaven.javadoc.skip=true"
-
-banner() {
- echo
- echo "============================================================"
- printf " %s\n" "$*"
- echo "============================================================"
- echo
-}
-
-step() {
- echo
- echo "-- $*"
- echo
-}
-
-pause() {
- if [[ "${WAYANG_DEMO_AUTO:-false}" != "true" ]]; then
- echo
- read -rp "Press ENTER to continue..." _ || true
- echo
- fi
-}
-
-run_wayang_demo() {
- "$WAYANG_ROOT/mvnw" exec:java -pl wayang-platforms/wayang-trino \
- -Dexec.mainClass="org.apache.wayang.trino.TrinoDemo" \
- ${MAVEN_FLAGS}
-}
-
-banner "ACT 1: Start Trino + Iceberg via Docker"
-
-step "1a. Starting the stack"
-cd "$TRINO_SETUP"
-docker compose up -d
-
-step "1b. Containers running"
-docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}" \
- | grep -E "NAMES|trino|minio|metastore|postgres"
-
-step "1c. Waiting for Trino to be ready"
-MAX_WAIT=90
-ELAPSED=0
-until docker exec "$TRINO_CONTAINER" \
- trino --execute "SELECT 1" --output-format ALIGNED >/dev/null 2>&1; do
- if [[ "$ELAPSED" -ge "$MAX_WAIT" ]]; then
- echo "Timed out waiting for Trino after ${MAX_WAIT}s"
- exit 1
- fi
- printf ". waiting (%ds elapsed)\r" "$ELAPSED"
- sleep 3
- ELAPSED=$((ELAPSED + 3))
-done
-echo "Trino is ready at http://localhost:8080"
-
-step "1d. Initialising Iceberg tables"
-docker exec -i "$TRINO_CONTAINER" trino < "$TRINO_SETUP/scripts/init.sql" 2>&1 \
- | grep -v "^WARNING\|jline\|org.jline" || true
-echo "iceberg.sales.orders seeded"
-
-step "1e. Table schema"
-docker exec "$TRINO_CONTAINER" \
- trino --execute "DESCRIBE iceberg.sales.orders" \
- --output-format ALIGNED 2>/dev/null
-
-pause
-
-banner "ACT 2: Query Iceberg directly via Trino CLI"
-
-step "2a. Full table scan"
-echo "SQL: SELECT * FROM iceberg.sales.orders"
-docker exec "$TRINO_CONTAINER" \
- trino --execute "SELECT * FROM iceberg.sales.orders ORDER BY order_id" \
- --output-format ALIGNED 2>/dev/null
-
-step "2b. Filter: region = 'AMER'"
-echo "SQL: SELECT * FROM iceberg.sales.orders WHERE region = 'AMER'"
-docker exec "$TRINO_CONTAINER" \
- trino --execute "SELECT * FROM iceberg.sales.orders WHERE region = 'AMER' ORDER BY order_id" \
- --output-format ALIGNED 2>/dev/null
-
-step "2c. Projection with filter"
-echo "SQL: SELECT region, product, amount FROM iceberg.sales.orders WHERE region = 'AMER'"
-docker exec "$TRINO_CONTAINER" \
- trino --execute \
- "SELECT region, product, amount
- FROM iceberg.sales.orders
- WHERE region = 'AMER'
- ORDER BY order_id" \
- --output-format ALIGNED 2>/dev/null
-
-pause
-
-banner "ACT 3: Wayang API filter + projection pushdown"
-cd "$WAYANG_ROOT"
-run_wayang_demo
-
-banner "Demo complete"
-echo "Trino UI: http://localhost:8080"
-echo "MinIO UI: http://localhost:9001 (minioadmin / minioadmin)"
-echo
-echo "To stop the stack:"
-echo " cd trino-setup && docker compose down"
diff --git a/trino-setup/docker-compose.yml b/trino-setup/docker-compose.yml
deleted file mode 100644
index 0c057540e..000000000
--- a/trino-setup/docker-compose.yml
+++ /dev/null
@@ -1,141 +0,0 @@
----
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Stack: Trino + Hive Metastore + MinIO (S3 storage)
-#
-# This is the battle-tested Trino + Iceberg local setup.
-# Hive Metastore (HMS) stores Iceberg table metadata over Thrift on port 9083.
-# MinIO provides S3-compatible object storage for Parquet data files.
-# Trino's Iceberg connector uses HMS as catalog and writes Parquet to MinIO.
-#
-# Ports:
-# Trino: http://localhost:8080 (UI + JDBC)
-# MinIO S3: http://localhost:9000
-# MinIO UI: http://localhost:9001 (minioadmin / minioadmin)
-# HMS: localhost:9083 (Thrift, internal)
-# Postgres: localhost:5432 (HMS backing store)
-
-services:
-
- # ── PostgreSQL (Hive Metastore backing database) ───────────────────────────
- postgres:
- image: postgres:15-alpine
- container_name: trino-postgres
- environment:
- POSTGRES_DB: metastore
- POSTGRES_USER: hive
- POSTGRES_PASSWORD: hive
- ports:
- - "5432:5432"
- volumes:
- - postgres-data:/var/lib/postgresql/data
- healthcheck:
- test: ["CMD-SHELL", "pg_isready -U hive -d metastore"]
- interval: 10s
- timeout: 5s
- retries: 5
-
- # ── MinIO (S3-compatible object storage) ──────────────────────────────────
- minio:
- image: minio/minio:latest
- container_name: trino-minio
- environment:
- MINIO_ROOT_USER: minioadmin
- MINIO_ROOT_PASSWORD: minioadmin
- ports:
- - "9000:9000"
- - "9001:9001"
- command: server /data --console-address ":9001"
- volumes:
- - minio-data:/data
- healthcheck:
- test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
- interval: 10s
- timeout: 5s
- retries: 5
-
- # Create the warehouse bucket before HMS starts
- minio-init:
- image: minio/mc:latest
- container_name: trino-minio-init
- depends_on:
- minio:
- condition: service_healthy
- entrypoint: >
- /bin/sh -c "
- mc alias set local http://minio:9000 minioadmin minioadmin;
- mc mb local/warehouse --ignore-existing;
- echo 'bucket warehouse ready';
- exit 0;
- "
-
- # ── Hive Metastore ────────────────────────────────────────────────────────
- # naushadh/hive-metastore is a minimal, pre-configured HMS image
- # that supports S3-compatible storage via env vars.
- metastore:
- image: naushadh/hive-metastore:latest
- container_name: trino-metastore
- depends_on:
- postgres:
- condition: service_healthy
- minio:
- condition: service_healthy
- minio-init:
- condition: service_completed_successfully
- ports:
- - "9083:9083"
- environment:
- DATABASE_HOST: postgres
- DATABASE_DB: metastore
- DATABASE_USER: hive
- DATABASE_PASSWORD: hive
- # S3 / MinIO
- S3_ENDPOINT_URL: http://minio:9000
- S3_BUCKET: warehouse
- S3_PREFIX: /
- AWS_ACCESS_KEY_ID: minioadmin
- AWS_SECRET_ACCESS_KEY: minioadmin
- REGION: us-east-1
- # No nc/curl in this image; use bash's /dev/tcp built-in
- healthcheck:
- test: ["CMD", "/bin/bash", "-c", "exec 3<>/dev/tcp/localhost/9083 2>/dev/null && exit 0 || exit 1"]
- interval: 15s
- timeout: 10s
- retries: 15
-
- # ── Trino ─────────────────────────────────────────────────────────────────
- trino:
- image: trinodb/trino:435
- container_name: trino
- depends_on:
- metastore:
- condition: service_healthy
- minio:
- condition: service_healthy
- ports:
- - "8080:8080"
- volumes:
- - ./trino/catalog:/etc/trino/catalog
- - ./trino/config.properties:/etc/trino/config.properties
- healthcheck:
- test: ["CMD", "curl", "-f", "http://localhost:8080/v1/info"]
- interval: 15s
- timeout: 10s
- retries: 10
-
-volumes:
- postgres-data:
- minio-data:
diff --git a/trino-setup/pom.xml b/trino-setup/pom.xml
deleted file mode 100644
index f04955203..000000000
--- a/trino-setup/pom.xml
+++ /dev/null
@@ -1,93 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
- http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <modelVersion>4.0.0</modelVersion>
-
- <groupId>org.apache.wayang</groupId>
- <artifactId>trino-setup</artifactId>
- <version>1.0-SNAPSHOT</version>
- <packaging>jar</packaging>
-
- <name>Trino Local Setup — Integration Tests</name>
- <description>
- Standalone integration tests for a local Trino stack
- (Trino + Nessie Iceberg catalog + MinIO S3 storage).
- Independent of the Wayang codebase.
- </description>
-
- <properties>
- <maven.compiler.source>17</maven.compiler.source>
- <maven.compiler.target>17</maven.compiler.target>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <trino.version>435</trino.version>
- <junit.version>5.10.2</junit.version>
- </properties>
-
- <dependencies>
- <!-- Trino JDBC driver -->
- <dependency>
- <groupId>io.trino</groupId>
- <artifactId>trino-jdbc</artifactId>
- <version>${trino.version}</version>
- <scope>test</scope>
- </dependency>
-
- <!-- JUnit 5 -->
- <dependency>
- <groupId>org.junit.jupiter</groupId>
- <artifactId>junit-jupiter</artifactId>
- <version>${junit.version}</version>
- <scope>test</scope>
- </dependency>
-
- <!-- Logging -->
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-simple</artifactId>
- <version>2.0.12</version>
- <scope>test</scope>
- </dependency>
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <version>3.2.5</version>
- <configuration>
- <!-- Skip integration tests by default; run with -Pintegration -->
- <skipTests>${skipIntegrationTests}</skipTests>
- </configuration>
- </plugin>
- </plugins>
- </build>
-
- <profiles>
- <!-- Run integration tests: mvn test -Pintegration -->
- <profile>
- <id>integration</id>
- <properties>
- <skipIntegrationTests>false</skipIntegrationTests>
- </properties>
- </profile>
- </profiles>
-</project>
diff --git a/trino-setup/scripts/init.sql b/trino-setup/scripts/init.sql
deleted file mode 100644
index 245ffbce3..000000000
--- a/trino-setup/scripts/init.sql
+++ /dev/null
@@ -1,66 +0,0 @@
--- Licensed to the Apache Software Foundation (ASF) under one
--- or more contributor license agreements. See the NOTICE file
--- distributed with this work for additional information
--- regarding copyright ownership. The ASF licenses this file
--- to you under the Apache License, Version 2.0 (the
--- "License"); you may not use this file except in compliance
--- with the License. You may obtain a copy of the License at
---
--- http://www.apache.org/licenses/LICENSE-2.0
---
--- Unless required by applicable law or agreed to in writing, software
--- distributed under the License is distributed on an "AS IS" BASIS,
--- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
--- See the License for the specific language governing permissions and
--- limitations under the License.
-
--- Run this after the stack is up to create sample Iceberg tables.
--- Usage: ./scripts/run-init.sh
--- Or manually: docker exec -it trino trino < /scripts/init.sql
-
--- Schema
-CREATE SCHEMA IF NOT EXISTS iceberg.sales;
-
--- Orders table (Iceberg / Parquet on MinIO)
-CREATE TABLE IF NOT EXISTS iceberg.sales.orders (
- order_id BIGINT,
- region VARCHAR,
- product VARCHAR,
- amount DOUBLE,
- order_date DATE
-)
-WITH (format = 'PARQUET');
-
--- Idempotent seed: clear before inserting so re-runs do not duplicate rows.
-DELETE FROM iceberg.sales.orders;
-
--- Sample data: 20 rows, 4 regions (AMER/APAC/EMEA/LATAM), 5 products.
--- AMER rows: 3, 6, 9, 12, 16 -> 5 rows for filter demo
--- Projection demo selects only: region, product, amount
-INSERT INTO iceberg.sales.orders VALUES
- (1, 'APAC', 'Widget A', 1500.00, DATE '2024-01-15'),
- (2, 'EMEA', 'Widget B', 800.50, DATE '2024-01-16'),
- (3, 'AMER', 'Widget A', 2200.00, DATE '2024-01-17'),
- (4, 'APAC', 'Widget C', 350.75, DATE '2024-01-18'),
- (5, 'EMEA', 'Widget A', 1100.00, DATE '2024-01-19'),
- (6, 'AMER', 'Widget B', 950.25, DATE '2024-01-20'),
- (7, 'APAC', 'Widget B', 1750.00, DATE '2024-01-21'),
- (8, 'EMEA', 'Widget C', 420.00, DATE '2024-01-22'),
- (9, 'AMER', 'Widget C', 680.50, DATE '2024-01-23'),
- (10, 'APAC', 'Widget A', 3000.00, DATE '2024-01-24'),
- (11, 'LATAM', 'Widget D', 560.00, DATE '2024-01-25'),
- (12, 'AMER', 'Widget D', 1320.75, DATE '2024-01-26'),
- (13, 'EMEA', 'Widget D', 990.00, DATE '2024-01-27'),
- (14, 'LATAM', 'Widget E', 2100.50, DATE '2024-01-28'),
- (15, 'APAC', 'Widget E', 4500.00, DATE '2024-01-29'),
- (16, 'AMER', 'Widget E', 3750.00, DATE '2024-01-30'),
- (17, 'EMEA', 'Widget E', 1250.00, DATE '2024-01-31'),
- (18, 'LATAM', 'Widget A', 870.25, DATE '2024-02-01'),
- (19, 'APAC', 'Widget D', 1680.00, DATE '2024-02-02'),
- (20, 'LATAM', 'Widget B', 440.50, DATE '2024-02-03');
-
--- Verify
-SELECT region, COUNT(*) AS order_count, SUM(amount) AS total_amount
-FROM iceberg.sales.orders
-GROUP BY region
-ORDER BY total_amount DESC;
diff --git a/trino-setup/scripts/run-init.sh b/trino-setup/scripts/run-init.sh
deleted file mode 100644
index 91d279192..000000000
--- a/trino-setup/scripts/run-init.sh
+++ /dev/null
@@ -1,39 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Runs init.sql against the local Trino instance.
-# The stack must be fully up before running this.
-
-set -e
-
-TRINO_HOST=${TRINO_HOST:-localhost}
-TRINO_PORT=${TRINO_PORT:-8080}
-
-echo "Waiting for Trino to be ready..."
-until curl -sf "http://${TRINO_HOST}:${TRINO_PORT}/v1/info" | grep -q '"starting":false'; do
- echo " Trino not ready yet, retrying in 5s..."
- sleep 5
-done
-echo "Trino is ready."
-
-echo "Running init.sql..."
-docker exec -i trino trino \
- --server "http://${TRINO_HOST}:${TRINO_PORT}" \
- --user admin \
- < "$(dirname "$0")/init.sql"
-
-echo "Done. Sample Iceberg data loaded into iceberg.sales.orders"
diff --git a/trino-setup/src/test/java/org/apache/wayang/trino/TrinoIntegrationTest.java b/trino-setup/src/test/java/org/apache/wayang/trino/TrinoIntegrationTest.java
deleted file mode 100644
index 081beea99..000000000
--- a/trino-setup/src/test/java/org/apache/wayang/trino/TrinoIntegrationTest.java
+++ /dev/null
@@ -1,232 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.wayang.trino;
-
-import org.junit.jupiter.api.*;
-
-import java.sql.*;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Properties;
-
-import static org.junit.jupiter.api.Assertions.*;
-
-/**
- * Integration tests for the local Trino stack.
- *
- * Prerequisites: run `docker-compose up -d` and `./scripts/run-init.sh` first.
- *
- * Run tests:
- * mvn test -Pintegration
- *
- * Or skip infrastructure setup and run with a custom host:
- * TRINO_HOST=localhost TRINO_PORT=8080 mvn test -Pintegration
- */
-@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
-class TrinoIntegrationTest {
-
- private static final String TRINO_HOST = System.getenv().getOrDefault("TRINO_HOST", "localhost");
- private static final int TRINO_PORT = Integer.parseInt(System.getenv().getOrDefault("TRINO_PORT", "8080"));
- private static final String JDBC_URL = String.format("jdbc:trino://%s:%d", TRINO_HOST, TRINO_PORT);
-
- private static Connection connection;
-
- // ── Lifecycle ─────────────────────────────────────────────────────────
-
- @BeforeAll
- static void openConnection() throws Exception {
- Properties props = new Properties();
- props.setProperty("user", "admin"); // Trino requires a non-empty user
- connection = DriverManager.getConnection(JDBC_URL, props);
- System.out.printf("Connected to Trino at %s%n", JDBC_URL);
- }
-
- @AfterAll
- static void closeConnection() throws Exception {
- if (connection != null && !connection.isClosed()) {
- connection.close();
- }
- }
-
- // ── Helper ────────────────────────────────────────────────────────────
-
- private List<List<Object>> query(String sql) throws SQLException {
- List<List<Object>> rows = new ArrayList<>();
- try (Statement stmt = connection.createStatement();
- ResultSet rs = stmt.executeQuery(sql)) {
- int cols = rs.getMetaData().getColumnCount();
- while (rs.next()) {
- List<Object> row = new ArrayList<>();
- for (int i = 1; i <= cols; i++) row.add(rs.getObject(i));
- rows.add(row);
- }
- }
- return rows;
- }
-
- // ── Test 1: Basic connectivity ────────────────────────────────────────
-
- @Test
- @Order(1)
- @DisplayName("Trino responds to a simple SELECT 1")
- void testConnectivity() throws SQLException {
- List<List<Object>> rows = query("SELECT 1");
- assertEquals(1, rows.size());
- assertEquals(1L, ((Number) rows.get(0).get(0)).longValue());
- System.out.println("[PASS] Basic connectivity OK");
- }
-
- // ── Test 2: TPC-H built-in connector ─────────────────────────────────
-
- @Test
- @Order(2)
- @DisplayName("TPC-H tiny catalog: count orders")
- void testTpchConnector() throws SQLException {
- List<List<Object>> rows = query("SELECT COUNT(*) FROM tpch.tiny.orders");
- long count = ((Number) rows.get(0).get(0)).longValue();
- assertTrue(count > 0, "tpch.tiny.orders should have rows");
- System.out.printf("[PASS] TPC-H tiny.orders has %,d rows%n", count);
- }
-
- @Test
- @Order(3)
- @DisplayName("TPC-H tiny catalog: top 5 orders by total price")
- void testTpchTopOrders() throws SQLException {
- List<List<Object>> rows = query("""
- SELECT orderkey, totalprice
- FROM tpch.tiny.orders
- ORDER BY totalprice DESC
- LIMIT 5
- """);
- assertEquals(5, rows.size(), "Expected exactly 5 rows");
- System.out.println("[PASS] TPC-H top 5 orders:");
- rows.forEach(r -> System.out.printf(" orderkey=%s totalprice=%s%n", r.get(0), r.get(1)));
- }
-
- // ── Test 4: Iceberg — schema exists ──────────────────────────────────
-
- @Test
- @Order(4)
- @DisplayName("Iceberg catalog: schema 'sales' is visible")
- void testIcebergSchemaVisible() throws SQLException {
- List<List<Object>> rows = query("SHOW SCHEMAS IN iceberg LIKE 'sales'");
- assertFalse(rows.isEmpty(), "Schema 'sales' should exist in iceberg catalog. " +
- "Did you run scripts/run-init.sh?");
- System.out.println("[PASS] Iceberg schema 'sales' is visible");
- }
-
- // ── Test 5: Iceberg — full table scan ────────────────────────────────
-
- @Test
- @Order(5)
- @DisplayName("Iceberg table: select all orders")
- void testIcebergSelectAll() throws SQLException {
- List<List<Object>> rows = query("SELECT * FROM iceberg.sales.orders ORDER BY order_id");
- assertEquals(20, rows.size(), "Expected 20 rows inserted by init.sql");
- System.out.println("[PASS] Iceberg full scan: 20 rows");
- rows.forEach(r -> System.out.printf(" %s%n", r));
- }
-
- // ── Test 6: Iceberg — pushdown filter ────────────────────────────────
-
- @Test
- @Order(6)
- @DisplayName("Iceberg table: filter by region = APAC")
- void testIcebergFilterByRegion() throws SQLException {
- List<List<Object>> rows = query("""
- SELECT order_id, region, amount
- FROM iceberg.sales.orders
- WHERE region = 'APAC'
- ORDER BY order_id
- """);
- assertFalse(rows.isEmpty(), "Should have APAC orders");
- rows.forEach(r -> assertEquals("APAC", r.get(1), "All rows must be APAC"));
- System.out.printf("[PASS] Filter pushdown: %d APAC rows%n", rows.size());
- }
-
- // ── Test 7: Iceberg — aggregation ────────────────────────────────────
-
- @Test
- @Order(7)
- @DisplayName("Iceberg table: aggregate total_amount by region")
- void testIcebergAggregate() throws SQLException {
- List<List<Object>> rows = query("""
- SELECT region, COUNT(*) AS order_count, SUM(amount) AS total_amount
- FROM iceberg.sales.orders
- GROUP BY region
- ORDER BY total_amount DESC
- """);
- assertFalse(rows.isEmpty(), "Aggregation should return rows");
- System.out.println("[PASS] Aggregation by region:");
- rows.forEach(r -> System.out.printf(" region=%-5s count=%s total=%.2f%n",
- r.get(0), r.get(1), ((Number) r.get(2)).doubleValue()));
- }
-
- // ── Test 8: Iceberg — amount threshold filter ─────────────────────────
-
- @Test
- @Order(8)
- @DisplayName("Iceberg table: filter orders with amount > 1000")
- void testIcebergFilterByAmount() throws SQLException {
- List<List<Object>> rows = query("""
- SELECT order_id, amount
- FROM iceberg.sales.orders
- WHERE amount > 1000.0
- ORDER BY amount DESC
- """);
- rows.forEach(r -> assertTrue(
- ((Number) r.get(1)).doubleValue() > 1000.0,
- "All rows must have amount > 1000"
- ));
- System.out.printf("[PASS] Amount filter: %d rows with amount > 1000%n", rows.size());
- }
-
- // ── Test 9: Iceberg — projection (select subset of columns) ───────────
-
- @Test
- @Order(9)
- @DisplayName("Iceberg table: project only region and product columns")
- void testIcebergProjection() throws SQLException {
- List<List<Object>> rows = query("""
- SELECT region, product
- FROM iceberg.sales.orders
- LIMIT 5
- """);
- assertEquals(5, rows.size());
- rows.forEach(r -> {
- assertNotNull(r.get(0), "region should not be null");
- assertNotNull(r.get(1), "product should not be null");
- });
- System.out.println("[PASS] Projection (region, product): 5 rows returned");
- }
-
- // ── Test 10: Iceberg metadata — table files ──────────────────────────
-
- @Test
- @Order(10)
- @DisplayName("Iceberg metadata: $files table lists at least one Parquet file")
- void testIcebergFilesMetadata() throws SQLException {
- List<List<Object>> rows = query("""
- SELECT file_path, record_count
- FROM iceberg.sales."orders$files"
- """);
- assertFalse(rows.isEmpty(), "There should be at least one data file after inserts");
- System.out.println("[PASS] Iceberg $files metadata:");
- rows.forEach(r -> System.out.printf(" %s (records=%s)%n", r.get(0), r.get(1)));
- }
-}
diff --git a/trino-setup/trino/catalog/iceberg.properties b/trino-setup/trino/catalog/iceberg.properties
deleted file mode 100644
index 5aabe5275..000000000
--- a/trino-setup/trino/catalog/iceberg.properties
+++ /dev/null
@@ -1,32 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Iceberg catalog backed by Hive Metastore (HMS) and MinIO (S3 storage).
-# HMS stores Iceberg table metadata; Trino writes Parquet files to MinIO via S3.
-connector.name=iceberg
-iceberg.catalog.type=hive_metastore
-hive.metastore.uri=thrift://metastore:9083
-
-# Native S3 filesystem handles both s3:// and s3a:// (which HMS uses internally).
-# This avoids the "No FileSystem for scheme s3" error when HMS assigns locations.
-fs.native-s3.enabled=true
-s3.endpoint=http://minio:9000
-s3.path-style-access=true
-s3.aws-access-key=minioadmin
-s3.aws-secret-key=minioadmin
-s3.region=us-east-1
-
-iceberg.file-format=PARQUET
diff --git a/trino-setup/trino/catalog/tpch.properties b/trino-setup/trino/catalog/tpch.properties
deleted file mode 100644
index ee4f23783..000000000
--- a/trino-setup/trino/catalog/tpch.properties
+++ /dev/null
@@ -1,20 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Built-in TPC-H connector with no external dependencies.
-# Useful for testing basic Trino connectivity without any storage setup.
-# Usage: SELECT * FROM tpch.tiny.orders LIMIT 10;
-connector.name=tpch
diff --git a/trino-setup/trino/config.properties b/trino-setup/trino/config.properties
deleted file mode 100644
index 8584e4aea..000000000
--- a/trino-setup/trino/config.properties
+++ /dev/null
@@ -1,20 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-coordinator=true
-node-scheduler.include-coordinator=true
-http-server.http.port=8080
-discovery.uri=http://localhost:8080
diff --git a/wayang-applications/README.md b/wayang-applications/README.md
index 3332b3caa..fa61185b2 100644
--- a/wayang-applications/README.md
+++ b/wayang-applications/README.md
@@ -76,4 +76,8 @@ The file _env.demo1.sh_ contains additional properties, need in a particlar demo
In this file we will never see cluster or user specific details, only properties which are specific to the
particular application are listed here.
+## SQL Platform Examples
+- [BigQuery](bigquery.md)
+- [Presto](presto.md)
+- [Trino](trino.md)
diff --git a/wayang-applications/bigquery.md b/wayang-applications/bigquery.md
new file mode 100644
index 000000000..a63327c54
--- /dev/null
+++ b/wayang-applications/bigquery.md
@@ -0,0 +1,103 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# BigQuery Example
+
+This example demonstrates the BigQuery cost model and executes filter/projection
+pushdown against an existing BigQuery table.
+
+## Prerequisites
+
+- Java 17
+- A Google Cloud project with the BigQuery API enabled
+- A dataset and a service account that can run queries and read the example table
+- A service-account JSON key, or another authentication mode supported by the
+ bundled Google BigQuery JDBC driver
+
+Wayang does not provision a BigQuery emulator or Google Cloud resources.
+
+## Prepare the table
+
+Run this query in the BigQuery console or with your preferred BigQuery client,
+after replacing `my-project` with your project ID:
+
+```sql
+CREATE SCHEMA IF NOT EXISTS `my-project.sales`;
+
+CREATE OR REPLACE TABLE `my-project.sales.orders` AS
+SELECT 1 AS order_id, 'AMER' AS region, 'book' AS product,
+ 25.50 AS amount, '2026-01-10' AS order_date
+UNION ALL
+SELECT 2, 'EMEA', 'desk', 300.00, '2026-01-11'
+UNION ALL
+SELECT 3, 'AMER', 'chair', 85.25, '2026-01-12';
+```
+
+You can instead use an existing table with `order_id`, `region`, `product`,
+`amount`, and `order_date` columns in that order.
+
+## Configure Wayang
+
+Create a properties file such as `/tmp/bigquery-example.properties`:
+
+```properties
+wayang.bigquery.jdbc.url = jdbc:bigquery://https://www.googleapis.com/bigquery/v2;ProjectId=my-project;OAuthType=0;OAuthServiceAcctEmail=service-account@example.com;OAuthPvtKeyPath=/path/to/key.json
+wayang.bigquery.jdbc.user =
+wayang.bigquery.jdbc.password =
+wayang.bigquery.demo.table = `my-project.sales.orders`
+```
+
+Authentication is configured through the JDBC URL supported by the BigQuery
+JDBC driver. Keep the backticks around the fully qualified table name because
+BigQuery project IDs can contain hyphens. Do not commit the service account key
+or a properties file containing credentials.
+
+## Build and run
+
+Build the application and its dependencies from the repository root:
+
+```bash
+./mvnw -Pskip-prerequisite-check -pl wayang-applications -am \
+ -DskipTests -Dpython.worker.tests.skip=true install
+```
+
+Run all segments against the configured table:
+
+```bash
+./mvnw -Pskip-prerequisite-check -pl wayang-applications exec:java \
+ -Dexec.mainClass=org.apache.wayang.applications.BigQueryDemo \
+ "-Dexec.args=file:///tmp/bigquery-example.properties all"
+```
+
+The optional second argument is `cost`, `filter`, `projection`, or `all`.
+`filter`, `projection`, and `all` execute against the configured BigQuery table
+and require the JDBC URL. The `cost` mode does not connect to BigQuery, so the
+JDBC URL can be omitted when running that mode alone.
+
+On Windows, replace `./mvnw` with `.\mvnw.cmd` and use a file URL such as
+`file:///C:/Temp/bigquery-example.properties`.
+
+## Troubleshooting
+
+- **Project or dataset not found:** check `ProjectId` in the JDBC URL and the
+ fully qualified table name.
+- **Permission denied:** grant the identity permission to create query jobs and
+ read the table data.
+- **Private key error:** use an absolute path in `OAuthPvtKeyPath` and confirm
+ that the Wayang process can read the JSON file.
+- **Only testing configuration:** run the `cost` mode; it does not contact
+ BigQuery or require credentials.
diff --git a/wayang-applications/pom.xml b/wayang-applications/pom.xml
index 28fba545d..0afa1d7af 100644
--- a/wayang-applications/pom.xml
+++ b/wayang-applications/pom.xml
@@ -56,6 +56,26 @@
</properties>
<dependencies>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-bigquery</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-presto</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-trino</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.antlr</groupId>
+ <artifactId>antlr4-runtime</artifactId>
+ <version>4.13.1</version>
+ </dependency>
<dependency>
<groupId>org.apache.wayang</groupId>
<artifactId>wayang-core</artifactId>
@@ -104,6 +124,11 @@
<version>3.9.2</version> <!-- Use the latest version available -->
</dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ <version>2.18.8</version>
+ </dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
diff --git a/wayang-applications/presto.md b/wayang-applications/presto.md
new file mode 100644
index 000000000..c418131f0
--- /dev/null
+++ b/wayang-applications/presto.md
@@ -0,0 +1,97 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Presto Example
+
+This example reads an existing PrestoDB table, filters rows whose `region` is
+`AMER`, and returns only the `region`, `product`, and `amount` columns. Wayang
+pushes the filter and projection into the SQL query executed by Presto.
+
+## Prerequisites
+
+- Java 17
+- A PrestoDB deployment reachable from the machine running Wayang
+- A catalog and schema in which you can create or read the example table
+- A Presto user with `SELECT` permission on that table
+
+Wayang does not start or configure PrestoDB. The endpoint can be a local,
+shared, or hosted deployment.
+
+## Prepare the table
+
+Run the following with your Presto CLI or SQL client. Replace
+`memory.default.orders` with a table name supported by your catalog when the
+memory connector is unavailable.
+
+```sql
+CREATE TABLE memory.default.orders (
+ order_id BIGINT,
+ region VARCHAR,
+ product VARCHAR,
+ amount DOUBLE,
+ order_date VARCHAR
+);
+
+INSERT INTO memory.default.orders VALUES
+ (1, 'AMER', 'book', 25.50, '2026-01-10'),
+ (2, 'EMEA', 'desk', 300.00, '2026-01-11'),
+ (3, 'AMER', 'chair', 85.25, '2026-01-12');
+```
+
+The example accepts any table with these five columns in this order:
+`order_id`, `region`, `product`, `amount`, and `order_date`.
+
+## Configure Wayang
+
+Create `/tmp/presto-example.properties`:
+
+```properties
+wayang.presto.jdbc.url = jdbc:presto://presto.example.com:8080/memory/default
+wayang.presto.jdbc.user = wayang
+wayang.presto.demo.table = memory.default.orders
+```
+
+Change the URL, credentials, and fully qualified table name for your deployment.
+Add `wayang.presto.jdbc.password` only when the deployment requires it; the
+Presto driver rejects an empty password on a non-TLS connection. TLS and other
+driver options can be included in the JDBC URL.
+
+## Build and run
+
+From the repository root:
+
+```bash
+./mvnw -Pskip-prerequisite-check -pl wayang-applications -am \
+ -DskipTests -Dpython.worker.tests.skip=true install
+
+./mvnw -Pskip-prerequisite-check -pl wayang-applications exec:java \
+ -Dexec.mainClass=org.apache.wayang.applications.PrestoDemo \
+ "-Dexec.args=file:///tmp/presto-example.properties"
+```
+
+The output should contain the two `AMER` rows and only three columns. On
+Windows, use `.\mvnw.cmd` and a URL such as
+`file:///C:/Temp/presto-example.properties`.
+
+## Troubleshooting
+
+- **Connection refused:** check the host, port, and network access from the
+ Wayang machine.
+- **Catalog, schema, or table not found:** use a fully qualified table name and
+ confirm it with `SELECT * FROM catalog.schema.table` in a Presto client.
+- **Authentication failed:** set the user, password, and any TLS or access-token
+ options required by your Presto JDBC endpoint.
diff --git a/wayang-applications/src/main/java/org/apache/wayang/applications/BigQueryDemo.java b/wayang-applications/src/main/java/org/apache/wayang/applications/BigQueryDemo.java
new file mode 100644
index 000000000..c709fc1b5
--- /dev/null
+++ b/wayang-applications/src/main/java/org/apache/wayang/applications/BigQueryDemo.java
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.wayang.applications;
+
+import org.apache.wayang.basic.data.Record;
+import org.apache.wayang.basic.function.ProjectionDescriptor;
+import org.apache.wayang.basic.operators.FilterOperator;
+import org.apache.wayang.basic.operators.LocalCallbackSink;
+import org.apache.wayang.basic.operators.MapOperator;
+import org.apache.wayang.basic.types.RecordType;
+import org.apache.wayang.bigquery.BigQuery;
+import org.apache.wayang.bigquery.operators.BigQueryTableSource;
+import org.apache.wayang.bigquery.platform.BigQueryPlatform;
+import org.apache.wayang.core.api.Configuration;
+import org.apache.wayang.core.api.WayangContext;
+import org.apache.wayang.core.function.PredicateDescriptor;
+import org.apache.wayang.core.plan.wayangplan.WayangPlan;
+import org.apache.wayang.core.types.DataSetType;
+import org.apache.wayang.java.Java;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Configurable example for the Wayang BigQuery connector.
+ *
+ * <p>The optional second argument selects a mode:
+ * <ul>
+ * <li>{@code cost}: cost model output (no credentials needed)</li>
+ * <li>{@code filter}: filter operator pushdown</li>
+ * <li>{@code projection}: projection and filter operator pushdown</li>
+ * </ul>
+ *
+ * <p>See {@code wayang-applications/bigquery.md} for configuration and usage.
+ */
+public class BigQueryDemo {
+
+ private static Configuration configuration;
+ private static String jdbcUrl;
+ private static String sourceTable;
+
+ public static void main(String[] args) {
+ if (args.length < 1 || args.length > 2) {
+ throw new IllegalArgumentException(
+ "Usage: BigQueryDemo <configuration URL> [cost|filter|projection|all]"
+ );
+ }
+ configuration = new Configuration(args[0]);
+ jdbcUrl = configuration.getStringProperty("wayang.bigquery.jdbc.url", "");
+ sourceTable = configuration.getStringProperty(
+ "wayang.bigquery.demo.table", "`my-project.sales.orders`"
+ );
+ String mode = args.length == 2 ? args[1] : "all";
+ switch (mode) {
+ case "cost": costModel(); break;
+ case "filter": filterDemo(); break;
+ case "projection": projectionDemo(); break;
+ case "all":
+ costModel();
+ filterDemo();
+ projectionDemo();
+ break;
+ default:
+ throw new IllegalArgumentException("Unknown BigQuery demo mode: " + mode);
+ }
+ }
+
+ static void costModel() {
+ BigQueryPlatform.getInstance().configureDefaults(configuration);
+
+ long mhz = configuration.getLongProperty("wayang.bigquery.cpu.mhz", 0);
+ long cores = configuration.getLongProperty("wayang.bigquery.cores", 0);
+ double fix = configuration.getDoubleProperty("wayang.bigquery.costs.fix", 0);
+ double perMs = configuration.getDoubleProperty("wayang.bigquery.costs.per-ms", 1);
+
+ long rows = 10;
+ long alpha = 5;
+ long beta = 2_000_000;
+ long cpuCycles = alpha * rows + beta;
+ double timeMs = cpuCycles / (cores * mhz * 1000.0);
+ double cost = fix + perMs * timeMs;
+
+ System.out.println();
+ System.out.println("BigQuery cost model");
+ System.out.println();
+ System.out.println(" Layer 1: cost formula (wayang-bigquery-defaults.properties)");
+ System.out.printf(" tablesource : %s%n", configuration.getStringProperty("wayang.bigquery.tablesource.load", null));
+ System.out.printf(" filter : %s%n", configuration.getStringProperty("wayang.bigquery.filter.load", null));
+ System.out.println();
+ System.out.println(" Layer 2: hardware profile (CPU cycles to wall-clock time)");
+ System.out.printf(" cpu.mhz = %d cores = %d%n", mhz, cores);
+ System.out.println();
+ System.out.println(" Layer 3: time to abstract cost");
+ System.out.printf(" costs.fix = %.1f costs.per-ms = %.1f%n", fix, perMs);
+ System.out.println();
+ System.out.println(" -- Worked example: 10-row table scan --");
+ System.out.printf(" alpha = %d (per-row, serverless columnar)%n", alpha);
+ System.out.printf(" beta = %,d (cold-start / slot reservation)%n", beta);
+ System.out.printf(" cpu cycles = %d * %d + %,d = %,d%n", alpha, rows, beta, cpuCycles);
+ System.out.printf(" time = %,d / (%d * %d * 1000) = %.4f ms%n", cpuCycles, cores, mhz, timeMs);
+ System.out.printf(" cost = %.1f + %.1f * %.4f = %.4f%n", fix, perMs, timeMs, cost);
+ System.out.println();
+ System.out.println();
+ }
+
+ static void filterDemo() {
+ requireJdbcUrl();
+ String table = sourceTable;
+
+ System.out.println();
+ System.out.println("BigQuery filter pushdown");
+ System.out.println();
+ System.out.println(" Operator: FilterOperator -> BigQueryFilterOperator");
+ System.out.printf(" SQL sent: SELECT * FROM %s%n", table);
+ System.out.println(" WHERE region = 'AMER'");
+ System.out.println();
+
+ runLiveFilter(table);
+
+ System.out.println();
+ }
+
+ private static void runLiveFilter(String table) {
+ WayangContext wayang = buildWayang();
+ List<Record> results = new ArrayList<>();
+
+ BigQueryTableSource source = new BigQueryTableSource(
+ table, "order_id", "region", "product", "amount", "order_date"
+ );
+ FilterOperator<Record> filter = new FilterOperator<>(
+ new PredicateDescriptor<>(
+ r -> "AMER".equals(r.getField(1)), Record.class
+ ).withSqlImplementation("region = 'AMER'")
+ );
+ LocalCallbackSink<Record> sink = LocalCallbackSink.createCollectingSink(results, Record.class);
+ source.connectTo(0, filter, 0);
+ filter.connectTo(0, sink, 0);
+ wayang.execute("BigQuery-Filter-Demo", new WayangPlan(sink));
+
+ System.out.println(" Results returned by Wayang:");
+ System.out.printf(" %-10s %-6s %-10s %10s %-12s%n",
+ "order_id", "region", "product", "amount", "order_date");
+ System.out.println(" " + repeat('-', 54));
+ for (Record r : results) {
+ System.out.printf(" %-10s %-6s %-10s %10s %-12s%n",
+ r.getField(0), r.getField(1), r.getField(2), r.getField(3), r.getField(4));
+ }
+ System.out.println();
+ System.out.printf(" %d AMER rows returned.%n%n", results.size());
+ }
+
+ static void projectionDemo() {
+ requireJdbcUrl();
+ String table = sourceTable;
+
+ System.out.println();
+ System.out.println("BigQuery projection pushdown");
+ System.out.println();
+ System.out.println(" Operators: FilterOperator -> BigQueryFilterOperator");
+ System.out.println(" MapOperator -> BigQueryProjectionOperator");
+ System.out.printf(" SQL sent: SELECT region, product, amount%n");
+ System.out.printf(" FROM %s%n", table);
+ System.out.println(" WHERE region = 'AMER'");
+ System.out.println();
+ System.out.println(" Both operators are combined in one SQL query; only 3 of 5");
+ System.out.println(" columns transferred; order_id + order_date never leave BQ.");
+ System.out.println();
+
+ runLiveProjection(table);
+
+ System.out.println();
+ }
+
+ private static void runLiveProjection(String table) {
+ WayangContext wayang = buildWayang();
+ List<Record> results = new ArrayList<>();
+
+ BigQueryTableSource source = new BigQueryTableSource(
+ table, "order_id", "region", "product", "amount", "order_date"
+ );
+ FilterOperator<Record> filter = new FilterOperator<>(
+ new PredicateDescriptor<>(
+ r -> "AMER".equals(r.getField(1)), Record.class
+ ).withSqlImplementation("region = 'AMER'")
+ );
+ // Use the Record-specific descriptor for a projection with multiple fields.
+ MapOperator<Record, Record> projection = new MapOperator<>(
+ ProjectionDescriptor.createForRecords(
+ new RecordType("order_id", "region", "product", "amount", "order_date"),
+ "region", "product", "amount"),
+ DataSetType.createDefault(Record.class),
+ DataSetType.createDefault(Record.class)
+ );
+ LocalCallbackSink<Record> sink = LocalCallbackSink.createCollectingSink(results, Record.class);
+ source.connectTo(0, filter, 0);
+ filter.connectTo(0, projection, 0);
+ projection.connectTo(0, sink, 0);
+ wayang.execute("BigQuery-Projection-Demo", new WayangPlan(sink));
+
+ System.out.println(" Results returned by Wayang (projected columns only):");
+ System.out.printf(" %-6s %-10s %10s%n", "region", "product", "amount");
+ System.out.println(" " + repeat('-', 30));
+ for (Record r : results) {
+ System.out.printf(" %-6s %-10s %10s%n", r.getField(0), r.getField(1), r.getField(2));
+ }
+ System.out.println();
+ System.out.printf(" %d AMER rows returned with 3 columns.%n%n",
+ results.size());
+ }
+
+ private static WayangContext buildWayang() {
+ return new WayangContext(configuration)
+ .withPlugin(Java.basicPlugin())
+ .withPlugin(BigQuery.plugin());
+ }
+
+ private static void requireJdbcUrl() {
+ if (jdbcUrl.isEmpty()) {
+ throw new IllegalArgumentException(
+ "wayang.bigquery.jdbc.url is required for filter and projection modes."
+ );
+ }
+ }
+
+ private static String repeat(char c, int n) {
+ StringBuilder sb = new StringBuilder(n);
+ for (int i = 0; i < n; i++) sb.append(c);
+ return sb.toString();
+ }
+}
diff --git a/wayang-applications/src/main/java/org/apache/wayang/applications/PrestoDemo.java b/wayang-applications/src/main/java/org/apache/wayang/applications/PrestoDemo.java
new file mode 100644
index 000000000..c9590df54
--- /dev/null
+++ b/wayang-applications/src/main/java/org/apache/wayang/applications/PrestoDemo.java
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.wayang.applications;
+
+import org.apache.wayang.basic.data.Record;
+import org.apache.wayang.basic.function.ProjectionDescriptor;
+import org.apache.wayang.basic.operators.FilterOperator;
+import org.apache.wayang.basic.operators.LocalCallbackSink;
+import org.apache.wayang.basic.operators.MapOperator;
+import org.apache.wayang.basic.types.RecordType;
+import org.apache.wayang.core.api.Configuration;
+import org.apache.wayang.core.api.WayangContext;
+import org.apache.wayang.core.function.PredicateDescriptor;
+import org.apache.wayang.core.plan.wayangplan.WayangPlan;
+import org.apache.wayang.core.types.DataSetType;
+import org.apache.wayang.java.Java;
+import org.apache.wayang.presto.Presto;
+import org.apache.wayang.presto.operators.PrestoTableSource;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/** Configurable filter and projection example for the Wayang Presto connector. */
+public class PrestoDemo {
+
+ public static void main(String[] args) {
+ if (args.length != 1) {
+ throw new IllegalArgumentException("Usage: PrestoDemo <configuration URL>");
+ }
+
+ Configuration configuration = new Configuration(args[0]);
+ configuration.getStringProperty("wayang.presto.jdbc.url");
+ String table = configuration.getStringProperty(
+ "wayang.presto.demo.table", "memory.default.orders"
+ );
+
+ WayangContext wayang = new WayangContext(configuration)
+ .withPlugin(Java.basicPlugin())
+ .withPlugin(Presto.plugin());
+ List<Record> results = new ArrayList<>();
+
+ PrestoTableSource source = new PrestoTableSource(
+ table, "order_id", "region", "product", "amount", "order_date"
+ );
+ FilterOperator<Record> filter = new FilterOperator<>(
+ new PredicateDescriptor<>(
+ record -> "AMER".equals(record.getField(1)), Record.class
+ ).withSqlImplementation("region = 'AMER'")
+ );
+ MapOperator<Record, Record> projection = new MapOperator<>(
+ ProjectionDescriptor.createForRecords(
+ new RecordType("order_id", "region", "product", "amount", "order_date"),
+ "region", "product", "amount"
+ ),
+ DataSetType.createDefault(Record.class),
+ DataSetType.createDefault(Record.class)
+ );
+ LocalCallbackSink<Record> sink = LocalCallbackSink.createCollectingSink(results, Record.class);
+
+ source.connectTo(0, filter, 0);
+ filter.connectTo(0, projection, 0);
+ projection.connectTo(0, sink, 0);
+ wayang.execute("Presto-Filter-Projection-Demo", new WayangPlan(sink));
+
+ System.out.printf("%-8s %-16s %10s%n", "region", "product", "amount");
+ for (Record result : results) {
+ System.out.printf("%-8s %-16s %10s%n",
+ result.getField(0), result.getField(1), result.getField(2));
+ }
+ System.out.printf("%n%d AMER rows returned from %s.%n", results.size(), table);
+ }
+}
diff --git a/wayang-platforms/wayang-trino/src/main/java/org/apache/wayang/trino/TrinoDemo.java b/wayang-applications/src/main/java/org/apache/wayang/applications/TrinoDemo.java
similarity index 50%
rename from wayang-platforms/wayang-trino/src/main/java/org/apache/wayang/trino/TrinoDemo.java
rename to wayang-applications/src/main/java/org/apache/wayang/applications/TrinoDemo.java
index 9becd061d..c0017e9e3 100644
--- a/wayang-platforms/wayang-trino/src/main/java/org/apache/wayang/trino/TrinoDemo.java
+++ b/wayang-applications/src/main/java/org/apache/wayang/applications/TrinoDemo.java
@@ -16,7 +16,7 @@
* limitations under the License.
*/
-package org.apache.wayang.trino;
+package org.apache.wayang.applications;
import org.apache.wayang.basic.data.Record;
import org.apache.wayang.basic.function.ProjectionDescriptor;
@@ -30,62 +30,55 @@ import org.apache.wayang.basic.types.RecordType;
import org.apache.wayang.core.plan.wayangplan.WayangPlan;
import org.apache.wayang.core.types.DataSetType;
import org.apache.wayang.java.Java;
+import org.apache.wayang.trino.Trino;
import org.apache.wayang.trino.operators.TrinoTableSource;
-import java.sql.Connection;
-import java.sql.DriverManager;
-import java.sql.ResultSet;
import java.util.ArrayList;
import java.util.List;
-import java.util.Properties;
/**
- * Standalone demo for the Wayang Trino connector.
+ * Configurable example for the Wayang Trino connector.
*
* <p>Demonstrates two Trino operator types:
* <ol>
- * <li>Seg 3 — Filter pushdown: WHERE region = 'AMER'.</li>
- * <li>Seg 4 — Projection + Filter pushdown:
+ * <li>Filter pushdown: WHERE region = 'AMER'.</li>
+ * <li>Projection and filter pushdown:
* SELECT region, product, amount ... WHERE region = 'AMER'.</li>
* </ol>
*
- * <p>Run with:
- * <pre>
- * cd /path/to/wayang
- * mvn exec:java -pl wayang-platforms/wayang-trino \
- * -Dexec.mainClass=org.apache.wayang.trino.TrinoDemo \
- * -Pskip-prerequisite-check -Drat.skip=true \
- * [-Dtrino.url=jdbc:trino://localhost:8080] [-Dtrino.user=admin]
- * </pre>
+ * <p>See {@code wayang-applications/trino.md} for configuration and usage.
*/
public class TrinoDemo {
- private static final String JDBC_URL = System.getProperty("trino.url", "jdbc:trino://localhost:8080");
- private static final String JDBC_USER = System.getProperty("trino.user", "admin");
+ private static Configuration configuration;
+ private static String sourceTable;
public static void main(String[] args) throws Exception {
- seg3Filter();
- seg4Projection();
+ if (args.length != 1) {
+ throw new IllegalArgumentException("Usage: TrinoDemo <configuration URL>");
+ }
+ configuration = new Configuration(args[0]);
+ configuration.getStringProperty("wayang.trino.jdbc.url");
+ sourceTable = configuration.getStringProperty(
+ "wayang.trino.demo.table", "iceberg.sales.orders"
+ );
+ filterDemo();
+ projectionDemo();
}
- // ── Seg 3 — Filter pushdown ───────────────────────────────────────────────
-
- static void seg3Filter() throws Exception {
- System.out.println("══════════════════════════════════════════════════════");
- System.out.println(" Seg 3 — Filter Operator Pushdown");
- System.out.println("══════════════════════════════════════════════════════");
+ static void filterDemo() throws Exception {
+ System.out.println("Trino filter pushdown");
System.out.println();
System.out.println(" Operator: FilterOperator -> TrinoFilterOperator");
- System.out.println(" SQL sent: SELECT * FROM iceberg.sales.orders");
+ System.out.println(" SQL sent: SELECT * FROM " + sourceTable);
System.out.println(" WHERE region = 'AMER'");
System.out.println();
- Configuration config = buildConfig();
- WayangContext wayang = buildWayang(config);
+ WayangContext wayang = buildWayang();
List<Record> results = new ArrayList<>();
TrinoTableSource source = new TrinoTableSource(
- "iceberg.sales.orders", "order_id", "region", "product", "amount", "order_date"
+ sourceTable, "order_id", "region", "product", "amount", "order_date"
);
FilterOperator<Record> filter = new FilterOperator<>(
new PredicateDescriptor<>(
@@ -107,48 +100,35 @@ public class TrinoDemo {
r.getField(0), r.getField(1), r.getField(2), r.getField(3), r.getField(4));
}
System.out.println();
- System.out.printf(" ✓ %d AMER rows — filter pushed to Trino as SQL WHERE clause%n", results.size());
-
- verifyInQueryHistory("iceberg.sales.orders");
-
- System.out.println("══════════════════════════════════════════════════════");
+ System.out.printf(" %d AMER rows returned.%n", results.size());
System.out.println();
}
- // ── Seg 4 — Projection + Filter pushdown ─────────────────────────────────
-
- static void seg4Projection() throws Exception {
- System.out.println("══════════════════════════════════════════════════════");
- System.out.println(" Seg 4 — Projection Operator Pushdown");
- System.out.println("══════════════════════════════════════════════════════");
+ static void projectionDemo() throws Exception {
+ System.out.println("Trino projection pushdown");
System.out.println();
System.out.println(" Operators: FilterOperator -> TrinoFilterOperator");
System.out.println(" MapOperator -> TrinoProjectionOperator");
System.out.println(" SQL sent: SELECT region, product, amount");
- System.out.println(" FROM iceberg.sales.orders");
+ System.out.println(" FROM " + sourceTable);
System.out.println(" WHERE region = 'AMER'");
System.out.println();
- System.out.println(" Both operators get pushed into a single SQL query —");
+ System.out.println(" Both operators get pushed into a single SQL query;");
System.out.println(" no unnecessary columns are transferred over the network.");
System.out.println();
- Configuration config = buildConfig();
- WayangContext wayang = buildWayang(config);
+ WayangContext wayang = buildWayang();
List<Record> results = new ArrayList<>();
TrinoTableSource source = new TrinoTableSource(
- "iceberg.sales.orders", "order_id", "region", "product", "amount", "order_date"
+ sourceTable, "order_id", "region", "product", "amount", "order_date"
);
FilterOperator<Record> filter = new FilterOperator<>(
new PredicateDescriptor<>(
r -> "AMER".equals(r.getField(1)), Record.class
).withSqlImplementation("region = 'AMER'")
);
- // Use the Record-aware projection (multi-field). The plain
- // ProjectionDescriptor(Class, Class, fields...) builds a POJO projection
- // whose Java implementation only supports a single field; createForRecords
- // yields a multi-field Record implementation that also works if Wayang
- // executes the projection on the Java side instead of pushing it to Trino.
+ // Use the Record-specific descriptor for a projection with multiple fields.
MapOperator<Record, Record> projection = new MapOperator<>(
ProjectionDescriptor.createForRecords(
new RecordType("order_id", "region", "product", "amount", "order_date"),
@@ -170,52 +150,17 @@ public class TrinoDemo {
System.out.printf(" %-6s %-10s %10s%n", r.getField(0), r.getField(1), r.getField(2));
}
System.out.println();
- System.out.printf(" ✓ %d AMER rows — only 3 of 5 columns fetched (projection pushed to SQL)%n",
+ System.out.printf(" %d AMER rows returned with 3 columns.%n",
results.size());
-
- verifyInQueryHistory("iceberg.sales.orders");
-
- System.out.println("══════════════════════════════════════════════════════");
System.out.println();
}
- // ── Shared helpers ────────────────────────────────────────────────────────
-
- private static Configuration buildConfig() {
- Configuration config = new Configuration();
- config.setProperty("wayang.trino.jdbc.url", JDBC_URL);
- config.setProperty("wayang.trino.jdbc.user", JDBC_USER);
- config.setProperty("wayang.trino.jdbc.password", "");
- return config;
- }
-
- private static WayangContext buildWayang(Configuration config) {
- return new WayangContext(config)
+ private static WayangContext buildWayang() {
+ return new WayangContext(configuration)
.withPlugin(Java.basicPlugin())
.withPlugin(Trino.plugin());
}
- private static void verifyInQueryHistory(String tableHint) throws Exception {
- System.out.println();
- System.out.println(" Checking Trino's system.runtime.queries for proof...");
- Properties props = new Properties();
- props.setProperty("user", JDBC_USER);
- try (Connection conn = DriverManager.getConnection(JDBC_URL, props)) {
- ResultSet rs = conn.createStatement().executeQuery(
- "SELECT query FROM system.runtime.queries " +
- "WHERE state = 'FINISHED' AND query LIKE '%" + tableHint + "%' " +
- "ORDER BY created DESC LIMIT 2"
- );
- System.out.println();
- System.out.println(" Last SQL Trino executed:");
- while (rs.next()) {
- System.out.println(" > " + rs.getString(1).replaceAll("\\s+", " "));
- }
- }
- System.out.println();
- System.out.println(" ✓ Wayang-assembled SQL confirmed in Trino query history.");
- }
-
private static String repeat(char c, int n) {
StringBuilder sb = new StringBuilder(n);
for (int i = 0; i < n; i++) sb.append(c);
diff --git a/wayang-applications/src/test/java/org/apache/wayang/applications/SqlPlatformDemoTest.java b/wayang-applications/src/test/java/org/apache/wayang/applications/SqlPlatformDemoTest.java
new file mode 100644
index 000000000..3394639f1
--- /dev/null
+++ b/wayang-applications/src/test/java/org/apache/wayang/applications/SqlPlatformDemoTest.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.wayang.applications;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+class SqlPlatformDemoTest {
+
+ @TempDir
+ Path directory;
+
+ @Test
+ void bigQueryCostModeLoadsWayangConfiguration() throws Exception {
+ Path configuration = directory.resolve("bigquery.properties");
+ Files.writeString(configuration, "wayang.bigquery.demo.table = `project.dataset.orders`\n");
+
+ assertDoesNotThrow(() -> BigQueryDemo.main(
+ new String[]{configuration.toUri().toString(), "cost"}
+ ));
+ }
+
+ @Test
+ void demosRequireConfigurationUrls() {
+ assertThrows(IllegalArgumentException.class, () -> BigQueryDemo.main(new String[0]));
+ assertThrows(IllegalArgumentException.class, () -> PrestoDemo.main(new String[0]));
+ assertThrows(IllegalArgumentException.class, () -> TrinoDemo.main(new String[0]));
+ }
+
+ @Test
+ void bigQueryOperatorModesRequireJdbcConfiguration() throws Exception {
+ Path configuration = directory.resolve("bigquery-no-jdbc.properties");
+ Files.writeString(configuration, "wayang.bigquery.demo.table = `project.dataset.orders`\n");
+
+ assertThrows(IllegalArgumentException.class, () -> BigQueryDemo.main(
+ new String[]{configuration.toUri().toString(), "filter"}
+ ));
+ }
+}
diff --git a/wayang-applications/trino.md b/wayang-applications/trino.md
new file mode 100644
index 000000000..7da14376a
--- /dev/null
+++ b/wayang-applications/trino.md
@@ -0,0 +1,101 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Trino Example
+
+This example runs filter and projection plans against an existing Trino table.
+Wayang pushes `region = 'AMER'` and the selected columns into SQL executed by
+Trino.
+
+## Prerequisites
+
+- Java 17
+- A Trino deployment reachable from the machine running Wayang
+- A writable catalog for preparing the sample data, or an existing readable table
+- A Trino user with `SELECT` permission on the example table
+
+Wayang does not start Trino, its catalog, or its storage services.
+
+## Prepare the table
+
+Using the Trino CLI or another SQL client, replace `iceberg.sales` with a catalog
+and schema available in your deployment, then run:
+
+```sql
+CREATE SCHEMA IF NOT EXISTS iceberg.sales;
+
+CREATE TABLE iceberg.sales.orders (
+ order_id BIGINT,
+ region VARCHAR,
+ product VARCHAR,
+ amount DOUBLE,
+ order_date VARCHAR
+);
+
+INSERT INTO iceberg.sales.orders VALUES
+ (1, 'AMER', 'book', 25.50, '2026-01-10'),
+ (2, 'EMEA', 'desk', 300.00, '2026-01-11'),
+ (3, 'AMER', 'chair', 85.25, '2026-01-12');
+```
+
+If the catalog is read-only, point the example at an existing table with these
+five columns in this order.
+
+## Configure Wayang
+
+Create a properties file such as `/tmp/trino-example.properties`:
+
+```properties
+wayang.trino.jdbc.url = jdbc:trino://trino.example.com:8080/iceberg/sales
+wayang.trino.jdbc.user = wayang
+wayang.trino.jdbc.password =
+wayang.trino.demo.table = iceberg.sales.orders
+```
+
+Change the endpoint, credentials, catalog, schema, and table for your deployment.
+Trino JDBC options such as SSL can be added to the JDBC URL.
+
+## Build and run
+
+Build the application and its dependencies from the repository root:
+
+```bash
+./mvnw -Pskip-prerequisite-check -pl wayang-applications -am \
+ -DskipTests -Dpython.worker.tests.skip=true install
+```
+
+Run the example:
+
+```bash
+./mvnw -Pskip-prerequisite-check -pl wayang-applications exec:java \
+ -Dexec.mainClass=org.apache.wayang.applications.TrinoDemo \
+ "-Dexec.args=file:///tmp/trino-example.properties"
+```
+
+Both sections should return the two `AMER` rows. The projection section returns
+only `region`, `product`, and `amount`. On Windows, replace `./mvnw` with
+`.\mvnw.cmd` and use a URL such as `file:///C:/Temp/trino-example.properties`.
+
+## Troubleshooting
+
+- **Connection refused:** verify the host, port, and network access from Wayang.
+- **Catalog, schema, or table not found:** check the fully qualified table name
+ with the Trino CLI.
+- **Authentication or TLS failure:** add the credentials and JDBC URL options
+ required by the deployment.
+- **Create table is unsupported:** prepare the five columns in a catalog that
+ supports writes, or use an existing table.
diff --git a/wayang-platforms/wayang-bigquery/README.md b/wayang-platforms/wayang-bigquery/README.md
new file mode 100644
index 000000000..9e6d479c6
--- /dev/null
+++ b/wayang-platforms/wayang-bigquery/README.md
@@ -0,0 +1,51 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0.
+ You may obtain a copy at http://www.apache.org/licenses/LICENSE-2.0.
+-->
+
+# Wayang Platform BigQuery
+
+This module connects Wayang to a user-managed Google BigQuery project through
+JDBC. Supply the project and authentication settings in the JDBC URL; the
+example format is documented in `wayang-bigquery-defaults.properties`.
+
+```properties
+wayang.bigquery.jdbc.url = jdbc:bigquery://https://www.googleapis.com/bigquery/v2;ProjectId=my-project;OAuthType=0;OAuthServiceAcctEmail=service-account@example.com;OAuthPvtKeyPath=/path/to/key.json
+wayang.bigquery.jdbc.user =
+wayang.bigquery.jdbc.password =
+```
+
+The runnable filter/projection example and its table requirements are documented
+in [`wayang-applications/bigquery.md`](../../wayang-applications/bigquery.md).
+
+## Integration tests
+
+`BigQueryOperatorsIT` is intended for connector development. The service
+account must be able to create query jobs and create, read, and delete tables in
+the selected dataset. The test creates its fixtures and removes their tables
+afterward; it leaves the dataset in place.
+
+Configure the test with environment variables and run it from the repository
+root:
+
+```bash
+BIGQUERY_PROJECT=my-project \
+BIGQUERY_SA_EMAIL=wayang-bq@my-project.iam.gserviceaccount.com \
+BIGQUERY_KEY_PATH=/absolute/path/to/key.json \
+BIGQUERY_DATASET=wayang_it \
+BIGQUERY_LOCATION=US \
+./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-bigquery -am \
+ -Dtest=BigQueryOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \
+ -DfailIfNoTests=false test
+```
+
+The equivalent system properties are `bigquery.project`, `bigquery.saEmail`,
+`bigquery.keyPath`, `bigquery.dataset`, and `bigquery.location`. These settings
+configure the integration test only; applications use the
+`wayang.bigquery.jdbc.*` properties described above.
+
+Cost calibration uses `BigQueryCostPilotIT`; see
+[`guides/cost-profiling.md`](../../guides/cost-profiling.md).
diff --git a/wayang-platforms/wayang-bigquery/src/main/java/org/apache/wayang/bigquery/BigQueryDemo.java b/wayang-platforms/wayang-bigquery/src/main/java/org/apache/wayang/bigquery/BigQueryDemo.java
deleted file mode 100644
index 18349d3db..000000000
--- a/wayang-platforms/wayang-bigquery/src/main/java/org/apache/wayang/bigquery/BigQueryDemo.java
+++ /dev/null
@@ -1,307 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.wayang.bigquery;
-
-import org.apache.wayang.basic.data.Record;
-import org.apache.wayang.basic.function.ProjectionDescriptor;
-import org.apache.wayang.basic.operators.FilterOperator;
-import org.apache.wayang.basic.operators.LocalCallbackSink;
-import org.apache.wayang.basic.operators.MapOperator;
-import org.apache.wayang.basic.types.RecordType;
-import org.apache.wayang.bigquery.operators.BigQueryTableSource;
-import org.apache.wayang.bigquery.platform.BigQueryPlatform;
-import org.apache.wayang.core.api.Configuration;
-import org.apache.wayang.core.api.WayangContext;
-import org.apache.wayang.core.function.PredicateDescriptor;
-import org.apache.wayang.core.plan.wayangplan.WayangPlan;
-import org.apache.wayang.core.types.DataSetType;
-import org.apache.wayang.java.Java;
-
-import java.util.ArrayList;
-import java.util.List;
-
-/**
- * Standalone demo for the Wayang BigQuery connector.
- *
- * <p>Controlled by {@code -Dbigquery.mode}:
- * <ul>
- * <li>{@code cost} — three-layer cost model (no credentials needed)</li>
- * <li>{@code filter} — filter operator pushdown demo</li>
- * <li>{@code projection} — projection + filter operator pushdown demo</li>
- * </ul>
- *
- * <p>Run with:
- * <pre>
- * mvn exec:java -pl wayang-platforms/wayang-bigquery \
- * -Dexec.mainClass=org.apache.wayang.bigquery.BigQueryDemo \
- * -Dbigquery.mode=cost \
- * -Pskip-prerequisite-check -Drat.skip=true
- * </pre>
- */
-public class BigQueryDemo {
-
- private static final String MODE = System.getProperty("bigquery.mode", "cost");
- private static final String JDBC_URL = System.getProperty("bigquery.url", "");
- private static final String PROJECT = System.getProperty("bigquery.project", "my-project");
-
- // 20-row dataset: 4 regions (AMER/APAC/EMEA/LATAM), 5 products (Widget A-E)
- // AMER rows: 3, 6, 9, 12, 16 → 5 rows for filter demo
- private static final String[][] SAMPLE_DATA = {
- {"1", "APAC", "Widget A", "1500.00", "2024-01-15"},
- {"2", "EMEA", "Widget B", "800.50", "2024-01-16"},
- {"3", "AMER", "Widget A", "2200.00", "2024-01-17"},
- {"4", "APAC", "Widget C", "350.75", "2024-01-18"},
- {"5", "EMEA", "Widget A", "1100.00", "2024-01-19"},
- {"6", "AMER", "Widget B", "950.25", "2024-01-20"},
- {"7", "APAC", "Widget B", "1750.00", "2024-01-21"},
- {"8", "EMEA", "Widget C", "420.00", "2024-01-22"},
- {"9", "AMER", "Widget C", "680.50", "2024-01-23"},
- {"10", "APAC", "Widget A", "3000.00", "2024-01-24"},
- {"11", "LATAM", "Widget D", "560.00", "2024-01-25"},
- {"12", "AMER", "Widget D", "1320.75", "2024-01-26"},
- {"13", "EMEA", "Widget D", "990.00", "2024-01-27"},
- {"14", "LATAM", "Widget E", "2100.50", "2024-01-28"},
- {"15", "APAC", "Widget E", "4500.00", "2024-01-29"},
- {"16", "AMER", "Widget E", "3750.00", "2024-01-30"},
- {"17", "EMEA", "Widget E", "1250.00", "2024-01-31"},
- {"18", "LATAM", "Widget A", "870.25", "2024-02-01"},
- {"19", "APAC", "Widget D", "1680.00", "2024-02-02"},
- {"20", "LATAM", "Widget B", "440.50", "2024-02-03"},
- };
-
- public static void main(String[] args) {
- switch (MODE) {
- case "cost": costModel(); break;
- case "filter": filterDemo(); break;
- case "projection": projectionDemo(); break;
- default:
- costModel();
- filterDemo();
- projectionDemo();
- }
- }
-
- // ── Cost model ────────────────────────────────────────────────────────────
-
- static void costModel() {
- Configuration config = new Configuration();
- BigQueryPlatform.getInstance().configureDefaults(config);
-
- long mhz = config.getLongProperty("wayang.bigquery.cpu.mhz", 0);
- long cores = config.getLongProperty("wayang.bigquery.cores", 0);
- double fix = config.getDoubleProperty("wayang.bigquery.costs.fix", 0);
- double perMs = config.getDoubleProperty("wayang.bigquery.costs.per-ms", 1);
-
- long rows = 10;
- long alpha = 5;
- long beta = 2_000_000;
- long cpuCycles = alpha * rows + beta;
- double timeMs = cpuCycles / (cores * mhz * 1000.0);
- double cost = fix + perMs * timeMs;
-
- System.out.println();
- System.out.println("══════════════════════════════════════════════════════");
- System.out.println(" BigQuery — Cost Model Integration");
- System.out.println("══════════════════════════════════════════════════════");
- System.out.println();
- System.out.println(" LAYER 1 — Cost formula (wayang-bigquery-defaults.properties)");
- System.out.printf(" tablesource : %s%n", config.getStringProperty("wayang.bigquery.tablesource.load", null));
- System.out.printf(" filter : %s%n", config.getStringProperty("wayang.bigquery.filter.load", null));
- System.out.println();
- System.out.println(" LAYER 2 — Hardware profile (cpu cycles -> wall-clock ms)");
- System.out.printf(" cpu.mhz = %d cores = %d%n", mhz, cores);
- System.out.println();
- System.out.println(" LAYER 3 — Time -> abstract cost");
- System.out.printf(" costs.fix = %.1f costs.per-ms = %.1f%n", fix, perMs);
- System.out.println();
- System.out.println(" -- Worked example: 10-row table scan --");
- System.out.printf(" alpha = %d (per-row, serverless columnar)%n", alpha);
- System.out.printf(" beta = %,d (cold-start / slot reservation)%n", beta);
- System.out.printf(" cpu cycles = %d * %d + %,d = %,d%n", alpha, rows, beta, cpuCycles);
- System.out.printf(" time = %,d / (%d * %d * 1000) = %.4f ms%n", cpuCycles, cores, mhz, timeMs);
- System.out.printf(" cost = %.1f + %.1f * %.4f = %.4f%n", fix, perMs, timeMs, cost);
- System.out.println();
- System.out.println("══════════════════════════════════════════════════════");
- System.out.println();
- }
-
- // ── Filter pushdown ───────────────────────────────────────────────────────
-
- static void filterDemo() {
- String table = String.format("`%s.sales.orders`", PROJECT);
-
- System.out.println();
- System.out.println("══════════════════════════════════════════════════════");
- System.out.println(" BigQuery — Filter Operator Pushdown");
- System.out.println("══════════════════════════════════════════════════════");
- System.out.println();
- System.out.println(" Operator: FilterOperator -> BigQueryFilterOperator");
- System.out.printf(" SQL sent: SELECT * FROM %s%n", table);
- System.out.println(" WHERE region = 'AMER'");
- System.out.println();
-
- if (!JDBC_URL.isEmpty()) {
- runLiveFilter(table);
- } else {
- System.out.println(" Results (20-row dataset, AMER rows only):");
- System.out.printf(" %-10s %-6s %-10s %10s %-12s%n",
- "order_id", "region", "product", "amount", "order_date");
- System.out.println(" " + repeat('-', 54));
- int count = 0;
- for (String[] row : SAMPLE_DATA) {
- if ("AMER".equals(row[1])) {
- System.out.printf(" %-10s %-6s %-10s %10s %-12s%n",
- row[0], row[1], row[2], row[3], row[4]);
- count++;
- }
- }
- System.out.println();
- System.out.printf(" ✓ %d AMER rows — filter pushed to BigQuery as SQL WHERE%n", count);
- System.out.println(" (pass -Dbigquery.url=... for live execution)");
- }
-
- System.out.println("══════════════════════════════════════════════════════");
- System.out.println();
- }
-
- private static void runLiveFilter(String table) {
- WayangContext wayang = buildWayang();
- List<Record> results = new ArrayList<>();
-
- BigQueryTableSource source = new BigQueryTableSource(
- table, "order_id", "region", "product", "amount", "order_date"
- );
- FilterOperator<Record> filter = new FilterOperator<>(
- new PredicateDescriptor<>(
- r -> "AMER".equals(r.getField(1)), Record.class
- ).withSqlImplementation("region = 'AMER'")
- );
- LocalCallbackSink<Record> sink = LocalCallbackSink.createCollectingSink(results, Record.class);
- source.connectTo(0, filter, 0);
- filter.connectTo(0, sink, 0);
- wayang.execute("BigQuery-Filter-Demo", new WayangPlan(sink));
-
- System.out.println(" Results returned by Wayang:");
- System.out.printf(" %-10s %-6s %-10s %10s %-12s%n",
- "order_id", "region", "product", "amount", "order_date");
- System.out.println(" " + repeat('-', 54));
- for (Record r : results) {
- System.out.printf(" %-10s %-6s %-10s %10s %-12s%n",
- r.getField(0), r.getField(1), r.getField(2), r.getField(3), r.getField(4));
- }
- System.out.println();
- System.out.printf(" ✓ %d AMER rows via Wayang -> BigQuery SQL pushdown%n%n", results.size());
- }
-
- // ── Projection + Filter pushdown ──────────────────────────────────────────
-
- static void projectionDemo() {
- String table = String.format("`%s.sales.orders`", PROJECT);
-
- System.out.println();
- System.out.println("══════════════════════════════════════════════════════");
- System.out.println(" BigQuery — Projection Operator Pushdown");
- System.out.println("══════════════════════════════════════════════════════");
- System.out.println();
- System.out.println(" Operators: FilterOperator -> BigQueryFilterOperator");
- System.out.println(" MapOperator -> BigQueryProjectionOperator");
- System.out.printf(" SQL sent: SELECT region, product, amount%n");
- System.out.printf(" FROM %s%n", table);
- System.out.println(" WHERE region = 'AMER'");
- System.out.println();
- System.out.println(" Both operators collapsed into one SQL — only 3 of 5");
- System.out.println(" columns transferred; order_id + order_date never leave BQ.");
- System.out.println();
-
- if (!JDBC_URL.isEmpty()) {
- runLiveProjection(table);
- } else {
- System.out.println(" Results (projected: region, product, amount — AMER only):");
- System.out.printf(" %-6s %-10s %10s%n", "region", "product", "amount");
- System.out.println(" " + repeat('-', 30));
- int count = 0;
- for (String[] row : SAMPLE_DATA) {
- if ("AMER".equals(row[1])) {
- System.out.printf(" %-6s %-10s %10s%n", row[1], row[2], row[3]);
- count++;
- }
- }
- System.out.println();
- System.out.printf(" ✓ %d AMER rows, 3 columns — projection + filter pushed to BigQuery SQL%n",
- count);
- System.out.println(" (pass -Dbigquery.url=... for live execution)");
- }
-
- System.out.println("══════════════════════════════════════════════════════");
- System.out.println();
- }
-
- private static void runLiveProjection(String table) {
- WayangContext wayang = buildWayang();
- List<Record> results = new ArrayList<>();
-
- BigQueryTableSource source = new BigQueryTableSource(
- table, "order_id", "region", "product", "amount", "order_date"
- );
- FilterOperator<Record> filter = new FilterOperator<>(
- new PredicateDescriptor<>(
- r -> "AMER".equals(r.getField(1)), Record.class
- ).withSqlImplementation("region = 'AMER'")
- );
- // Record-aware multi-field projection (see TrinoDemo for rationale).
- MapOperator<Record, Record> projection = new MapOperator<>(
- ProjectionDescriptor.createForRecords(
- new RecordType("order_id", "region", "product", "amount", "order_date"),
- "region", "product", "amount"),
- DataSetType.createDefault(Record.class),
- DataSetType.createDefault(Record.class)
- );
- LocalCallbackSink<Record> sink = LocalCallbackSink.createCollectingSink(results, Record.class);
- source.connectTo(0, filter, 0);
- filter.connectTo(0, projection, 0);
- projection.connectTo(0, sink, 0);
- wayang.execute("BigQuery-Projection-Demo", new WayangPlan(sink));
-
- System.out.println(" Results returned by Wayang (projected columns only):");
- System.out.printf(" %-6s %-10s %10s%n", "region", "product", "amount");
- System.out.println(" " + repeat('-', 30));
- for (Record r : results) {
- System.out.printf(" %-6s %-10s %10s%n", r.getField(0), r.getField(1), r.getField(2));
- }
- System.out.println();
- System.out.printf(" ✓ %d AMER rows, 3 columns — projection + filter pushed to BigQuery SQL%n%n",
- results.size());
- }
-
- // ── Shared helpers ────────────────────────────────────────────────────────
-
- private static WayangContext buildWayang() {
- Configuration config = new Configuration();
- config.setProperty("wayang.bigquery.jdbc.url", JDBC_URL);
- return new WayangContext(config)
- .withPlugin(Java.basicPlugin())
- .withPlugin(BigQuery.plugin());
- }
-
- private static String repeat(char c, int n) {
- StringBuilder sb = new StringBuilder(n);
- for (int i = 0; i < n; i++) sb.append(c);
- return sb.toString();
- }
-}
diff --git a/wayang-platforms/wayang-presto/README.md b/wayang-platforms/wayang-presto/README.md
new file mode 100644
index 000000000..268108928
--- /dev/null
+++ b/wayang-platforms/wayang-presto/README.md
@@ -0,0 +1,47 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0.
+ You may obtain a copy at http://www.apache.org/licenses/LICENSE-2.0.
+-->
+
+# Wayang Platform Presto
+
+This module connects Wayang to a user-managed PrestoDB deployment through JDBC.
+It does not require Docker or a repository-provided Presto environment.
+
+Configure the connection in a Wayang properties file:
+
+```properties
+wayang.presto.jdbc.url = jdbc:presto://presto.example.com:8080/catalog/schema
+wayang.presto.jdbc.user = wayang
+```
+
+Add `wayang.presto.jdbc.password` only when the deployment requires one.
+
+For a complete table definition, sample data, and runnable Wayang plan, see
+[`wayang-applications/presto.md`](../../wayang-applications/presto.md).
+
+## Integration tests
+
+`PrestoOperatorsIT` is intended for connector development. It requires a
+writable `memory` catalog and permission to read `system.runtime.queries`. The
+test creates the `memory.wayang_it` schema and its fixture tables, exercises the
+supported JDBC operators, and removes the fixtures afterward.
+
+Set the endpoint when it differs from the defaults shown below, then run the
+test from the repository root:
+
+```bash
+PRESTO_HOST=localhost PRESTO_PORT=8080 PRESTO_USER=test \
+./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am \
+ -Dtest=PrestoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \
+ -DfailIfNoTests=false test
+```
+
+These environment variables configure the integration test only. Applications
+use the `wayang.presto.jdbc.*` properties described above.
+
+Cost calibration uses `PrestoCostPilotIT`; see
+[`guides/cost-profiling.md`](../../guides/cost-profiling.md).
diff --git a/wayang-platforms/wayang-presto/src/main/resources/wayang-presto-defaults.properties b/wayang-platforms/wayang-presto/src/main/resources/wayang-presto-defaults.properties
index d614fa087..3c63f1621 100644
--- a/wayang-platforms/wayang-presto/src/main/resources/wayang-presto-defaults.properties
+++ b/wayang-platforms/wayang-presto/src/main/resources/wayang-presto-defaults.properties
@@ -18,7 +18,7 @@
# Connection (override per deployment in wayang.properties)
# wayang.presto.jdbc.url = jdbc:presto://localhost:8080
# wayang.presto.jdbc.user = test
-# wayang.presto.jdbc.password =
+# wayang.presto.jdbc.password = (optional; omit when no password is required)
wayang.presto.jdbc.driverName = com.facebook.presto.jdbc.PrestoDriver
# Hardware profile used by LoadProfileToTimeConverter.
diff --git a/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoCostPilotIT.java b/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoCostPilotIT.java
index 0020a77c2..1ab4fb7c9 100644
--- a/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoCostPilotIT.java
+++ b/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoCostPilotIT.java
@@ -71,7 +71,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
class PrestoCostPilotIT {
private static final String HOST = System.getenv().getOrDefault("PRESTO_HOST", "localhost");
- private static final int PORT = Integer.parseInt(System.getenv().getOrDefault("PRESTO_PORT", "8081"));
+ private static final int PORT = Integer.parseInt(System.getenv().getOrDefault("PRESTO_PORT", "8080"));
private static final String USER = System.getenv().getOrDefault("PRESTO_USER", "test");
private static final String JDBC_URL = String.format("jdbc:presto://%s:%d/memory", HOST, PORT);
diff --git a/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoOperatorsIT.java b/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoOperatorsIT.java
index d92f30cf6..2ab989a2a 100644
--- a/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoOperatorsIT.java
+++ b/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoOperatorsIT.java
@@ -85,8 +85,8 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
* {@code CREATE TABLE ... AS SELECT} ran entirely inside Presto.
*
* <p>Prerequisites: a Presto reachable at {@code PRESTO_HOST:PRESTO_PORT}
- * (defaults {@code localhost:8081}) with the {@code memory} connector enabled —
- * e.g. {@code cd presto-setup && docker compose up -d}. If Presto is not reachable
+ * (defaults {@code localhost:8080}) with the {@code memory} connector enabled.
+ * If Presto is not reachable
* the whole class is skipped (not failed).
*
* <pre>
@@ -99,7 +99,7 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
class PrestoOperatorsIT {
private static final String HOST = System.getenv().getOrDefault("PRESTO_HOST", "localhost");
- private static final int PORT = Integer.parseInt(System.getenv().getOrDefault("PRESTO_PORT", "8081"));
+ private static final int PORT = Integer.parseInt(System.getenv().getOrDefault("PRESTO_PORT", "8080"));
private static final String USER = System.getenv().getOrDefault("PRESTO_USER", "test");
private static final String JDBC_URL = String.format("jdbc:presto://%s:%d/memory", HOST, PORT);
diff --git a/wayang-platforms/wayang-trino/README.md b/wayang-platforms/wayang-trino/README.md
new file mode 100644
index 000000000..1376531e1
--- /dev/null
+++ b/wayang-platforms/wayang-trino/README.md
@@ -0,0 +1,44 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0.
+ You may obtain a copy at http://www.apache.org/licenses/LICENSE-2.0.
+-->
+
+# Wayang Platform Trino
+
+This module connects Wayang to a user-managed Trino deployment through JDBC.
+Configure the endpoint and credentials in a Wayang properties file:
+
+```properties
+wayang.trino.jdbc.url = jdbc:trino://trino.example.com:8080/catalog/schema
+wayang.trino.jdbc.user = wayang
+wayang.trino.jdbc.password =
+```
+
+The runnable filter/projection example and its table requirements are documented
+in [`wayang-applications/trino.md`](../../wayang-applications/trino.md).
+
+## Integration tests
+
+`TrinoOperatorsIT` is intended for connector development. It requires a
+writable catalog named `iceberg`, support for Parquet tables, permission to
+create the `iceberg.wayang_it` schema, and permission to read
+`system.runtime.queries`. It creates and removes its own fixture tables.
+
+Set the endpoint when it differs from the defaults shown below, then run the
+test from the repository root:
+
+```bash
+TRINO_HOST=localhost TRINO_PORT=8080 TRINO_USER=admin \
+./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am \
+ -Dtest=TrinoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \
+ -DfailIfNoTests=false test
+```
+
+These environment variables configure the integration test only. Applications
+use the `wayang.trino.jdbc.*` properties described above.
+
+Cost calibration uses `TrinoCostPilotIT`; see
+[`guides/cost-profiling.md`](../../guides/cost-profiling.md).
diff --git a/wayang-platforms/wayang-trino/src/test/java/org/apache/wayang/trino/TrinoOperatorsIT.java b/wayang-platforms/wayang-trino/src/test/java/org/apache/wayang/trino/TrinoOperatorsIT.java
index 283fefe8d..fcdb2bdca 100644
--- a/wayang-platforms/wayang-trino/src/test/java/org/apache/wayang/trino/TrinoOperatorsIT.java
+++ b/wayang-platforms/wayang-trino/src/test/java/org/apache/wayang/trino/TrinoOperatorsIT.java
@@ -80,8 +80,8 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
* execution itself does not require the Java plugin. Result assertions use
* plain JDBC only after the Wayang execution has completed.
*
- * <p>Prerequisites: a Trino reachable at {@code TRINO_HOST:TRINO_PORT}
- * (defaults {@code localhost:8080}); e.g. {@code cd platforms-setup-guides/trino-setup && docker compose up -d}.
+ * <p>Prerequisites: a user-managed Trino reachable at
+ * {@code TRINO_HOST:TRINO_PORT} (defaults {@code localhost:8080}).
* If Trino is not reachable the whole class is skipped (not failed).
*
* <p>Run: