You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@xtable.apache.org by "vinishjail97 (via GitHub)" <gi...@apache.org> on 2026/08/25 00:33:13 UTC

[PR] [855] Add the 0.4.0 release notes and a Spark runtime quickstart with a runnable demo [incubator-xtable]

vinishjail97 opened a new pull request, #913:
URL: https://github.com/apache/incubator-xtable/pull/913

   ## What
   
   Website documentation plus one runnable demo. No module sources are touched.
   
   - **0.4.0-incubating release page** (`website/releases/release-0.4.0-incubating.mdx`) and a matching entry on the **Downloads** page.
   - **New docs page** `Run a sync on Apache Spark` (`website/docs/spark-runtime.md`), under **Quick Start** in the sidebar and linked from **Installation**.
   - **New demo** `demo/spark-runtime/` — a Spark job that keeps a table readable in the other formats from inside the job that writes it.
   
   This replaces #847, which was closed so the docs could be rewritten and get a fresh review.
   
   ## The Spark quickstart
   
   `xtable-spark-runtime_2.12` `0.4.0-incubating` is on Maven Central, so the page points at the released jar rather than a source build. The module is not on `main`, so nothing here depends on it being in the tree.
   
   The page leads with the case the jar exists for. A job already writes a table, adds the jar with `--jars`, and calls `XTableSyncService` after its write:
   
   ```shell
   $SPARK_HOME/bin/spark-submit \
     --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
     --class com.example.OrdersJob \
     orders-job.jar
   ```
   
   The standalone `spark-submit` entry point and the multi-table `--datasetconfig` follow, then the CLI options, the format matrix and the Spark version table.
   
   ### On `--jars` vs `--packages`
   
   Every dependency in the published pom is `provided` or `test` scoped, so `--packages` on this jar resolves one jar and no transitives. Both flags work. `--jars` is the recommendation only because it needs no Maven Central lookup on each submit.
   
   The real hazard is adding the **engines** with `--packages`. `--packages` and `--jars` both load into a child class loader, and a second Avro or Parquet there breaks casts across the boundary. Engine jars belong on `spark.driver.extraClassPath`, which is what `ITXTableSparkRuntimeBundle` already does. The page and the demo README both say this.
   
   ## The demo
   
   `demo/spark-runtime/XTableInJobSyncDemo.scala` runs both directions and checks the row counts, so it doubles as a smoke test of a release:
   
   | Direction | The job writes | XTable adds |
   | --- | --- | --- |
   | 1 | `HUDI` | `ICEBERG`, `DELTA` |
   | 2 | `ICEBERG` | `HUDI`, `DELTA` |
   
   It needs no Docker and no Scala compiler — `spark-shell -i` compiles the script, so a reader runs one command. The jar comes from Maven Central with `curl` and goes on `--jars`.
   
   A local Spark distribution ships no engines, so the README resolves them from the downloaded pom with `dependency:build-classpath` and puts them on `extraClassPath`. Spark 3.4.x is required, because Hudi 0.14.0 publishes no Spark 3.5 bundle for the datasource direction 1 writes with; the runtime jar itself supports 3.4 and 3.5.
   
   ## Verification
   
   - Ran the demo README verbatim from a clean directory against **Spark 3.4.2** with the published `0.4.0-incubating` jar:
   
     ```
     [PASS] hudi -> iceberg        1000 rows
     [PASS] hudi -> delta          1000 rows
     [PASS] iceberg -> hudi        1000 rows
     [PASS] iceberg -> delta       1000 rows
     ALL CHECKS PASSED
     ```
   
   - `npm run build` in `website/` succeeds and both new pages generate.
   - `apache-rat:check` passes on the new files.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [855] Add the 0.4.0 release notes and a Spark runtime quickstart with a runnable demo [incubator-xtable]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #913:
URL: https://github.com/apache/incubator-xtable/pull/913#discussion_r3848765966


##########
website/docs/setup.md:
##########
@@ -23,4 +23,7 @@ For more information on the steps, follow the project's GitHub [README.md](https
 
 ## Next Steps
 See the [Quickstart](/docs/how-to) guide to learn to use Apache XTable™ (Incubating) to add interoperability between
-different table formats.
\ No newline at end of file
+different table formats.
+
+To run a sync on an Apache Spark cluster you already have, see
+[Run a sync on Apache Spark](/docs/spark-runtime).

Review Comment:
   Renamed to `how-to-spark-runtime.md` in 8248b3f6, so it lines up with `how-to` and `how-to-catalog-sync`. Updated the sidebar, this link, the release notes link and the demo README. Also retitled the page to "Run an XTable sync on Apache Spark".



##########
website/releases/release-0.4.0-incubating.mdx:
##########
@@ -0,0 +1,81 @@
+---
+title: "Release 0.4.0-incubating"
+sidebar_position: -4
+---
+
+## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+
+This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files.
+Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats.
+
+## ✨ What's Changed
+
+### 📦 Run a sync on Apache Spark
+
+This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster
+you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit
+it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi,
+Iceberg and Delta libraries your cluster already supplies.
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It
+supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime).
+
+### 🔄 New source formats
+
+* **Apache Paimon source**, with incremental sync and column statistics.
+* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync.
+
+### 🆕 Delta Kernel
+
+* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs

Review Comment:
   You're right — `DeltaKernelConversionSource` landed in #729, so the release ships both. Fixed in 8248b3f6: the bullet now reads "Added a **Delta Kernel** conversion source and target".



##########
website/releases/release-0.4.0-incubating.mdx:
##########
@@ -0,0 +1,81 @@
+---
+title: "Release 0.4.0-incubating"
+sidebar_position: -4
+---
+
+## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+
+This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files.
+Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats.
+
+## ✨ What's Changed
+
+### 📦 Run a sync on Apache Spark
+
+This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster
+you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit
+it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi,
+Iceberg and Delta libraries your cluster already supplies.
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It
+supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime).
+
+### 🔄 New source formats
+
+* **Apache Paimon source**, with incremental sync and column statistics.
+* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync.
+
+### 🆕 Delta Kernel
+
+* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs

Review Comment:
   Simplified in 8248b3f6. It's now a single bullet with no `delta-core`:
   
   > Added a **Delta Kernel** conversion source and target, which no longer depend on Spark for the conversion itself.
   
   Dropped `delta-core` from the Spark version table on the docs page too — it now says Delta Standalone and Delta Kernel.



##########
website/releases/release-0.4.0-incubating.mdx:
##########
@@ -0,0 +1,81 @@
+---
+title: "Release 0.4.0-incubating"
+sidebar_position: -4
+---
+
+## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+
+This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files.
+Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats.
+
+## ✨ What's Changed
+
+### 📦 Run a sync on Apache Spark
+
+This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster
+you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit
+it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi,
+Iceberg and Delta libraries your cluster already supplies.
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It
+supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime).
+
+### 🔄 New source formats
+
+* **Apache Paimon source**, with incremental sync and column statistics.
+* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync.
+
+### 🆕 Delta Kernel
+
+* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs
+  on Spark 3.5 and newer, where `delta-core` does not.
+
+### 🔧 Conversion improvements
+
+* Delta source: extract statistics for nested fields.
+* Hudi source: support nested partition columns when hive-style partitioning is on.
+* Fall back to Parquet footers when the metadata table has no column statistics.
+* Throw on a bad Delta checkpoint CRC by default.

Review Comment:
   Reworded in 8248b3f6 to "Fail fast on a corrupt Delta checkpoint CRC, instead of continuing with bad statistics."



##########
website/releases/release-0.4.0-incubating.mdx:
##########
@@ -0,0 +1,81 @@
+---
+title: "Release 0.4.0-incubating"
+sidebar_position: -4
+---
+
+## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+
+This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files.
+Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats.
+
+## ✨ What's Changed
+
+### 📦 Run a sync on Apache Spark
+
+This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster
+you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit
+it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi,
+Iceberg and Delta libraries your cluster already supplies.
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It
+supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime).
+
+### 🔄 New source formats
+
+* **Apache Paimon source**, with incremental sync and column statistics.
+* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync.
+
+### 🆕 Delta Kernel
+
+* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs
+  on Spark 3.5 and newer, where `delta-core` does not.
+
+### 🔧 Conversion improvements
+
+* Delta source: extract statistics for nested fields.
+* Hudi source: support nested partition columns when hive-style partitioning is on.
+* Fall back to Parquet footers when the metadata table has no column statistics.
+* Throw on a bad Delta checkpoint CRC by default.
+* Pass additional properties through to a Hudi target.

Review Comment:
   Agreed, removed in 8248b3f6.



##########
website/docs/spark-runtime.md:
##########
@@ -0,0 +1,237 @@
+---
+sidebar_position: 3
+title: "Run a sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run a sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an
+Apache Spark cluster. The sync reads the source table metadata and writes the target format
+metadata next to the data files that already exist. It rewrites no data.
+
+Add the jar to a Spark job you already run. You need no separate service and no separate cluster.
+
+## Get the jar
+
+The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`.
+Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+The [Downloads](/releases/downloads) page lists every release.
+
+## Add the sync to a Spark job
+
+This is the main use case. Your job already writes a table in one format. You want the same
+table to be readable in the other formats when the write completes.
+
+Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you
+already submit:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the
+table and pass the Hadoop configuration of the Spark session:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against the classes, add the same coordinates to your build with `provided` scope.
+
+`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls
+back to a full snapshot when an incremental sync is not safe, such as the first run. A call
+after every write is therefore safe to repeat.
+
+For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files
+there, and each target writes its metadata next to those files.
+
+:::note Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime)
+holds a complete job that runs both directions and checks the row counts.
+:::
+
+## Why `--jars` and not `--packages`
+
+The runtime jar carries only XTable code plus a few relocated helper libraries. It declares
+Spark, Hadoop, Hudi, Iceberg, Delta, Avro and Parquet as `provided`, so it uses the copies your
+cluster already supplies.
+
+Use `--jars`. It takes the file you downloaded and needs no network access at submit time.
+
+`--packages` also works. Every dependency in the published POM is `provided` or `test` scoped,
+so Ivy downloads this one jar and no transitive jars. It costs you a Maven Central lookup on
+each submit, which is the only reason to prefer `--jars`.
+
+:::danger Do not add the engines with `--packages`
+Add Hudi, Iceberg or Delta with `--packages` only when your cluster does not already supply
+them. `--packages` and `--jars` load classes in a child class loader. A second copy of Avro or
+Parquet there breaks casts across the class loader boundary. Put engine jars on the flat
+classpath with `spark.driver.extraClassPath` and `spark.executor.extraClassPath` instead.
+:::
+
+## Run a sync without a job
+
+The jar also ships a `spark-submit` entry point. Use it when you want the sync as a separate
+step. Pass the jar as the application jar, not with `--jars`:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  --master 'local[*]' \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --basepath /path/to/hudi_table \
+  --sourceformat HUDI \
+  --targets ICEBERG,DELTA
+```
+
+To sync many tables in one submit, pass `--datasetconfig` with a YAML file. `sourceFormat` and
+`targetFormats` apply to every table. Each entry under `datasets` needs only a `tableBasePath`.
+The config path can be local or on cloud storage.
+
+```yaml md title="dataset.yaml"
+sourceFormat: HUDI
+targetFormats:
+  - ICEBERG
+  - DELTA
+datasets:
+  - tableBasePath: s3://example-warehouse/db/store_sales
+  - tableBasePath: s3://example-warehouse/db/store_returns
+  - tableBasePath: s3://example-warehouse/db/item
+```
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --datasetconfig dataset.yaml
+```
+
+`--datasetconfig` and `--basepath` are mutually exclusive. Pass one of the two.
+
+| Option | Description |
+| --- | --- |
+| `--basepath` | Base path of the source table. |
+| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. |
+| `--targets` | Comma-separated target formats, such as `ICEBERG,DELTA`. |
+| `--datasetconfig` | Path to a YAML config that lists many tables. |
+| `--datapath` | Path to the data files when it differs from the base path. |
+| `--tablename` | Table name. Defaults to the last segment of the base path. |
+| `--namespace` | Dot-separated table namespace. |
+| `--partitionspec` | Hudi source partition field spec, such as `level:VALUE`. |
+| `--usedeltakernel` | Force the Spark-free Delta Kernel for a Delta source or target. |
+| `--help` | Print the usage text. |
+
+## Supported formats
+
+Paimon and Parquet are read-only sources. XTable writes no Paimon or Parquet target.
+
+| Source ↓ / Target → | Hudi | Iceberg | Delta |
+| --- | :---: | :---: | :---: |
+| **Hudi** | – | ✅ | ✅ |
+| **Iceberg** | ✅ | – | ✅ |
+| **Delta** | ✅ | ✅ | – |
+| **Paimon** | ✅ | ✅ | ✅ |
+| **Parquet** | ✅ | ✅ | ✅ |
+
+## Spark versions
+
+Hudi and Iceberg conversion use Spark-free core classes only, so they run on every Spark line
+below. Delta is the one engine that depends on the Spark version, and the jar picks the right
+implementation for you.
+
+| Spark version | Hudi and Iceberg | Delta implementation |
+| --- | :---: | --- |
+| 3.4.x | ✅ | Delta Standalone (`delta-core`) |
+| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected for you |
+
+On Spark 3.5 and newer, `delta-core` does not run. A Delta source or target goes through the
+Spark-free Delta Kernel instead, with no flag. To force Kernel on Spark 3.4, pass
+`--usedeltakernel`, or set `.useDeltaKernel(true)` on the `TableSyncSpec`.
+
+## Read a Hudi target
+
+XTable records the file listing of a Hudi target in the Hudi metadata table. Enable that table
+in the reader. Without the option, the sync reports success and the read returns zero rows.
+
+```scala md title="scala"
+spark.read
+  .option("hoodie.metadata.enable", "true")
+  .option("hoodie.datasource.read.extract.partition.values.from.path", "true")
+  .format("hudi")
+  .load(basePath)
+```
+
+Iceberg and Delta targets need no extra read option. One exception: the vectorized Arrow reader
+of Iceberg mis-casts timestamp columns on a table that another engine wrote. Pass
+`vectorization-enabled=false` when you see that.

Review Comment:
   Removed the whole section in 8248b3f6. Next steps now links to [Apache Spark](/docs/spark), which already documents the Hudi read options, so the version-specific detail lives in one place instead of two.
   
   Kept the note in `demo/spark-runtime/README.md`, since that demo returns zero rows without it — it now links to the same page.



##########
website/docs/spark-runtime.md:
##########
@@ -0,0 +1,237 @@
+---
+sidebar_position: 3
+title: "Run a sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run a sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an
+Apache Spark cluster. The sync reads the source table metadata and writes the target format
+metadata next to the data files that already exist. It rewrites no data.
+
+Add the jar to a Spark job you already run. You need no separate service and no separate cluster.
+
+## Get the jar
+
+The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`.
+Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+The [Downloads](/releases/downloads) page lists every release.
+
+## Add the sync to a Spark job
+
+This is the main use case. Your job already writes a table in one format. You want the same
+table to be readable in the other formats when the write completes.
+
+Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you
+already submit:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the
+table and pass the Hadoop configuration of the Spark session:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against the classes, add the same coordinates to your build with `provided` scope.
+
+`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls
+back to a full snapshot when an incremental sync is not safe, such as the first run. A call
+after every write is therefore safe to repeat.
+
+For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files
+there, and each target writes its metadata next to those files.
+
+:::note Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime)
+holds a complete job that runs both directions and checks the row counts.
+:::
+
+## Why `--jars` and not `--packages`
+
+The runtime jar carries only XTable code plus a few relocated helper libraries. It declares
+Spark, Hadoop, Hudi, Iceberg, Delta, Avro and Parquet as `provided`, so it uses the copies your
+cluster already supplies.
+
+Use `--jars`. It takes the file you downloaded and needs no network access at submit time.
+
+`--packages` also works. Every dependency in the published POM is `provided` or `test` scoped,
+so Ivy downloads this one jar and no transitive jars. It costs you a Maven Central lookup on
+each submit, which is the only reason to prefer `--jars`.
+
+:::danger Do not add the engines with `--packages`
+Add Hudi, Iceberg or Delta with `--packages` only when your cluster does not already supply
+them. `--packages` and `--jars` load classes in a child class loader. A second copy of Avro or
+Parquet there breaks casts across the class loader boundary. Put engine jars on the flat
+classpath with `spark.driver.extraClassPath` and `spark.executor.extraClassPath` instead.
+:::
+
+## Run a sync without a job
+
+The jar also ships a `spark-submit` entry point. Use it when you want the sync as a separate
+step. Pass the jar as the application jar, not with `--jars`:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  --master 'local[*]' \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --basepath /path/to/hudi_table \
+  --sourceformat HUDI \
+  --targets ICEBERG,DELTA
+```
+
+To sync many tables in one submit, pass `--datasetconfig` with a YAML file. `sourceFormat` and
+`targetFormats` apply to every table. Each entry under `datasets` needs only a `tableBasePath`.
+The config path can be local or on cloud storage.
+
+```yaml md title="dataset.yaml"
+sourceFormat: HUDI
+targetFormats:
+  - ICEBERG
+  - DELTA
+datasets:
+  - tableBasePath: s3://example-warehouse/db/store_sales
+  - tableBasePath: s3://example-warehouse/db/store_returns
+  - tableBasePath: s3://example-warehouse/db/item
+```
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --datasetconfig dataset.yaml
+```
+
+`--datasetconfig` and `--basepath` are mutually exclusive. Pass one of the two.
+
+| Option | Description |
+| --- | --- |
+| `--basepath` | Base path of the source table. |
+| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. |
+| `--targets` | Comma-separated target formats, such as `ICEBERG,DELTA`. |
+| `--datasetconfig` | Path to a YAML config that lists many tables. |
+| `--datapath` | Path to the data files when it differs from the base path. |
+| `--tablename` | Table name. Defaults to the last segment of the base path. |
+| `--namespace` | Dot-separated table namespace. |
+| `--partitionspec` | Hudi source partition field spec, such as `level:VALUE`. |
+| `--usedeltakernel` | Force the Spark-free Delta Kernel for a Delta source or target. |
+| `--help` | Print the usage text. |
+
+## Supported formats
+
+Paimon and Parquet are read-only sources. XTable writes no Paimon or Parquet target.
+
+| Source ↓ / Target → | Hudi | Iceberg | Delta |
+| --- | :---: | :---: | :---: |
+| **Hudi** | – | ✅ | ✅ |
+| **Iceberg** | ✅ | – | ✅ |
+| **Delta** | ✅ | ✅ | – |
+| **Paimon** | ✅ | ✅ | ✅ |
+| **Parquet** | ✅ | ✅ | ✅ |
+
+## Spark versions
+
+Hudi and Iceberg conversion use Spark-free core classes only, so they run on every Spark line
+below. Delta is the one engine that depends on the Spark version, and the jar picks the right
+implementation for you.
+
+| Spark version | Hudi and Iceberg | Delta implementation |
+| --- | :---: | --- |
+| 3.4.x | ✅ | Delta Standalone (`delta-core`) |
+| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected for you |
+
+On Spark 3.5 and newer, `delta-core` does not run. A Delta source or target goes through the

Review Comment:
   Fixed in 8248b3f6. The Spark version table now reads Delta Standalone / Delta Kernel, and `delta-core` no longer appears on either page.



##########
website/docs/spark-runtime.md:
##########
@@ -0,0 +1,237 @@
+---
+sidebar_position: 3
+title: "Run a sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run a sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an
+Apache Spark cluster. The sync reads the source table metadata and writes the target format
+metadata next to the data files that already exist. It rewrites no data.
+
+Add the jar to a Spark job you already run. You need no separate service and no separate cluster.
+
+## Get the jar
+
+The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`.
+Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+The [Downloads](/releases/downloads) page lists every release.
+
+## Add the sync to a Spark job
+
+This is the main use case. Your job already writes a table in one format. You want the same
+table to be readable in the other formats when the write completes.
+
+Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you
+already submit:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the
+table and pass the Hadoop configuration of the Spark session:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against the classes, add the same coordinates to your build with `provided` scope.
+
+`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls
+back to a full snapshot when an incremental sync is not safe, such as the first run. A call
+after every write is therefore safe to repeat.
+
+For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files
+there, and each target writes its metadata next to those files.
+
+:::note Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime)
+holds a complete job that runs both directions and checks the row counts.
+:::
+
+## Why `--jars` and not `--packages`
+
+The runtime jar carries only XTable code plus a few relocated helper libraries. It declares
+Spark, Hadoop, Hudi, Iceberg, Delta, Avro and Parquet as `provided`, so it uses the copies your
+cluster already supplies.
+
+Use `--jars`. It takes the file you downloaded and needs no network access at submit time.
+
+`--packages` also works. Every dependency in the published POM is `provided` or `test` scoped,
+so Ivy downloads this one jar and no transitive jars. It costs you a Maven Central lookup on
+each submit, which is the only reason to prefer `--jars`.
+
+:::danger Do not add the engines with `--packages`
+Add Hudi, Iceberg or Delta with `--packages` only when your cluster does not already supply
+them. `--packages` and `--jars` load classes in a child class loader. A second copy of Avro or
+Parquet there breaks casts across the class loader boundary. Put engine jars on the flat
+classpath with `spark.driver.extraClassPath` and `spark.executor.extraClassPath` instead.
+:::
+
+## Run a sync without a job
+
+The jar also ships a `spark-submit` entry point. Use it when you want the sync as a separate
+step. Pass the jar as the application jar, not with `--jars`:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  --master 'local[*]' \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --basepath /path/to/hudi_table \
+  --sourceformat HUDI \
+  --targets ICEBERG,DELTA
+```
+
+To sync many tables in one submit, pass `--datasetconfig` with a YAML file. `sourceFormat` and
+`targetFormats` apply to every table. Each entry under `datasets` needs only a `tableBasePath`.
+The config path can be local or on cloud storage.
+
+```yaml md title="dataset.yaml"
+sourceFormat: HUDI
+targetFormats:
+  - ICEBERG
+  - DELTA
+datasets:
+  - tableBasePath: s3://example-warehouse/db/store_sales
+  - tableBasePath: s3://example-warehouse/db/store_returns
+  - tableBasePath: s3://example-warehouse/db/item
+```
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --datasetconfig dataset.yaml
+```
+
+`--datasetconfig` and `--basepath` are mutually exclusive. Pass one of the two.
+
+| Option | Description |
+| --- | --- |
+| `--basepath` | Base path of the source table. |
+| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. |
+| `--targets` | Comma-separated target formats, such as `ICEBERG,DELTA`. |
+| `--datasetconfig` | Path to a YAML config that lists many tables. |
+| `--datapath` | Path to the data files when it differs from the base path. |
+| `--tablename` | Table name. Defaults to the last segment of the base path. |
+| `--namespace` | Dot-separated table namespace. |
+| `--partitionspec` | Hudi source partition field spec, such as `level:VALUE`. |
+| `--usedeltakernel` | Force the Spark-free Delta Kernel for a Delta source or target. |
+| `--help` | Print the usage text. |
+
+## Supported formats
+
+Paimon and Parquet are read-only sources. XTable writes no Paimon or Parquet target.
+
+| Source ↓ / Target → | Hudi | Iceberg | Delta |
+| --- | :---: | :---: | :---: |
+| **Hudi** | – | ✅ | ✅ |
+| **Iceberg** | ✅ | – | ✅ |
+| **Delta** | ✅ | ✅ | – |
+| **Paimon** | ✅ | ✅ | ✅ |
+| **Parquet** | ✅ | ✅ | ✅ |
+
+## Spark versions
+
+Hudi and Iceberg conversion use Spark-free core classes only, so they run on every Spark line
+below. Delta is the one engine that depends on the Spark version, and the jar picks the right
+implementation for you.

Review Comment:
   Reworded in 8248b3f6 to "Converting to and from Hudi and Iceberg doesn't require Spark at all, so those run on any of the Spark versions below."



##########
website/docs/spark-runtime.md:
##########
@@ -0,0 +1,237 @@
+---
+sidebar_position: 3
+title: "Run a sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run a sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an
+Apache Spark cluster. The sync reads the source table metadata and writes the target format
+metadata next to the data files that already exist. It rewrites no data.
+
+Add the jar to a Spark job you already run. You need no separate service and no separate cluster.
+
+## Get the jar
+
+The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`.
+Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+The [Downloads](/releases/downloads) page lists every release.
+
+## Add the sync to a Spark job
+
+This is the main use case. Your job already writes a table in one format. You want the same
+table to be readable in the other formats when the write completes.
+
+Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you
+already submit:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the
+table and pass the Hadoop configuration of the Spark session:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against the classes, add the same coordinates to your build with `provided` scope.
+
+`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls
+back to a full snapshot when an incremental sync is not safe, such as the first run. A call
+after every write is therefore safe to repeat.
+
+For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files
+there, and each target writes its metadata next to those files.
+
+:::note Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime)
+holds a complete job that runs both directions and checks the row counts.
+:::
+
+## Why `--jars` and not `--packages`

Review Comment:
   Removed in 8248b3f6.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [855] Add the 0.4.0 release notes and a Spark runtime quickstart with a runnable demo [incubator-xtable]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 merged PR #913:
URL: https://github.com/apache/incubator-xtable/pull/913


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [855] Add the 0.4.0 release notes and a Spark runtime quickstart with a runnable demo [incubator-xtable]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on code in PR #913:
URL: https://github.com/apache/incubator-xtable/pull/913#discussion_r3849571398


##########
website/docs/how-to-spark-runtime.md:
##########
@@ -0,0 +1,220 @@
+---
+sidebar_position: 3
+title: "Run an XTable sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run an XTable sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache
+Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source
+table's metadata and writes the target format's metadata alongside the data that is already there.
+
+The jar is meant to be dropped into a Spark job you already run, so you don't need a separate
+process or a separate cluster to keep a table interoperable across formats.
+
+## Getting the jar
+
+The jar is published to Maven Central as
+`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and
+Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
+full list of releases.
+
+## Adding the sync to a Spark job
+
+This is what the jar is for. Your job already writes a table in one format, and you want that same
+table to be interoperable with the others as soon as the write finishes.
+
+Add the jar to the `spark-submit` you already use, alongside your application jar:

Review Comment:
   Please update the structure so it first covers building the orders-job, and then details how to run it using spark-submit.



##########
website/docs/how-to-spark-runtime.md:
##########
@@ -0,0 +1,220 @@
+---
+sidebar_position: 3
+title: "Run an XTable sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run an XTable sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache
+Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source
+table's metadata and writes the target format's metadata alongside the data that is already there.
+
+The jar is meant to be dropped into a Spark job you already run, so you don't need a separate
+process or a separate cluster to keep a table interoperable across formats.
+
+## Getting the jar
+
+The jar is published to Maven Central as
+`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and
+Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
+full list of releases.
+
+## Adding the sync to a Spark job
+
+This is what the jar is for. Your job already writes a table in one format, and you want that same
+table to be interoperable with the others as soon as the write finishes.
+
+Add the jar to the `spark-submit` you already use, alongside your application jar:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and
+hand it the session's Hadoop configuration:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against these classes, add the same Maven coordinates to your build with `provided`
+scope.
+
+The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling
+back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That
+makes it safe to call after every write.
+
+:::note Tables are addressed by path
+The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent
+of the `RunSync` Iceberg catalog config (`-i`) yet. When a table's data files don't sit directly
+under `basePath`, set `dataPath` to wherever they do, because that's the location each target
+writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but nothing in Iceberg
+requires it — `write.data.path` and object-storage layouts can put data files anywhere — so check
+where your table actually writes.
+:::
+
+:::tip Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) is
+a complete job that syncs both directions and verifies the row counts.
+:::
+
+## Running a sync as its own job
+
+The jar also ships a `spark-submit` entry point, which is the equivalent of `RunSync` for this
+bundle. Pass the jar as the application jar rather than with `--jars`:

Review Comment:
   Could we mention that users need to create the Hudi table using the Quickstart guide before running spark-submit?



##########
website/docs/how-to-spark-runtime.md:
##########
@@ -0,0 +1,220 @@
+---
+sidebar_position: 3
+title: "Run an XTable sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run an XTable sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache
+Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source
+table's metadata and writes the target format's metadata alongside the data that is already there.
+
+The jar is meant to be dropped into a Spark job you already run, so you don't need a separate
+process or a separate cluster to keep a table interoperable across formats.
+
+## Getting the jar
+
+The jar is published to Maven Central as
+`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and
+Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
+full list of releases.
+
+## Adding the sync to a Spark job
+
+This is what the jar is for. Your job already writes a table in one format, and you want that same
+table to be interoperable with the others as soon as the write finishes.
+
+Add the jar to the `spark-submit` you already use, alongside your application jar:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and
+hand it the session's Hadoop configuration:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against these classes, add the same Maven coordinates to your build with `provided`
+scope.
+
+The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling
+back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That
+makes it safe to call after every write.
+
+:::note Tables are addressed by path
+The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent
+of the `RunSync` Iceberg catalog config (`-i`) yet. When a table's data files don't sit directly
+under `basePath`, set `dataPath` to wherever they do, because that's the location each target
+writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but nothing in Iceberg
+requires it — `write.data.path` and object-storage layouts can put data files anywhere — so check
+where your table actually writes.
+:::
+
+:::tip Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) is
+a complete job that syncs both directions and verifies the row counts.
+:::
+
+## Running a sync as its own job
+
+The jar also ships a `spark-submit` entry point, which is the equivalent of `RunSync` for this
+bundle. Pass the jar as the application jar rather than with `--jars`:
+
+```shell md title="shell"

Review Comment:
   Can we add a section on running XTableSparkSync with and without a configuration file or single-table vs. multi-table syncs?



##########
website/releases/downloads.mdx:
##########


Review Comment:
   Can we move 'Downloads' to the top, right after 'Latest Releases'?



##########
website/docs/how-to-spark-runtime.md:
##########
@@ -0,0 +1,220 @@
+---
+sidebar_position: 3
+title: "Run an XTable sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run an XTable sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache
+Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source
+table's metadata and writes the target format's metadata alongside the data that is already there.
+
+The jar is meant to be dropped into a Spark job you already run, so you don't need a separate
+process or a separate cluster to keep a table interoperable across formats.
+
+## Getting the jar
+
+The jar is published to Maven Central as
+`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and
+Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
+full list of releases.
+
+## Adding the sync to a Spark job

Review Comment:
   Move 'Adding the sync to a Spark job' after 'Running a sync as its own job'.
   
   Reason: The current order causes confusion for new users because references like orders-job.jar are introduced without context.



##########
website/releases/release-0.4.0-incubating.mdx:
##########
@@ -0,0 +1,79 @@
+---
+title: "Release 0.4.0-incubating"
+sidebar_position: -4
+---
+
+## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+
+This is the fourth official Apache release for Apache XTable (Incubating), an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files.
+Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake, and with this release it can also read Apache Paimon and Apache Parquet tables as sources.
+
+## ✨ What's Changed
+
+### 📦 Run an XTable sync on Apache Spark
+
+This release introduces `xtable-spark-runtime`, a runtime jar that runs a sync on a Spark cluster
+you already have. Add it to an existing job with `--jars` and call the sync right after your write,
+or submit it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar
+reuses the Hudi, Iceberg and Delta libraries already on your cluster.
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+It is published to Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`
+and supports Spark 3.4 and 3.5. See [Run an XTable sync on Apache Spark](/docs/how-to-spark-runtime).
+
+### 🔄 New source formats
+
+* **Apache Paimon source**, with incremental sync and column statistics.
+* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync.
+
+### 🆕 Delta Kernel

Review Comment:
   Could we include a callout/note here to explicitly state Spark 3.5 support?



##########
website/docs/how-to-spark-runtime.md:
##########
@@ -0,0 +1,220 @@
+---
+sidebar_position: 3
+title: "Run an XTable sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run an XTable sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache
+Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source
+table's metadata and writes the target format's metadata alongside the data that is already there.
+
+The jar is meant to be dropped into a Spark job you already run, so you don't need a separate
+process or a separate cluster to keep a table interoperable across formats.
+
+## Getting the jar
+
+The jar is published to Maven Central as
+`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and
+Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
+full list of releases.
+
+## Adding the sync to a Spark job
+
+This is what the jar is for. Your job already writes a table in one format, and you want that same
+table to be interoperable with the others as soon as the write finishes.
+
+Add the jar to the `spark-submit` you already use, alongside your application jar:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and
+hand it the session's Hadoop configuration:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against these classes, add the same Maven coordinates to your build with `provided`
+scope.
+
+The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling
+back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That
+makes it safe to call after every write.
+
+:::note Tables are addressed by path
+The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent
+of the `RunSync` Iceberg catalog config (`-i`) yet. When a table's data files don't sit directly
+under `basePath`, set `dataPath` to wherever they do, because that's the location each target
+writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but nothing in Iceberg
+requires it — `write.data.path` and object-storage layouts can put data files anywhere — so check
+where your table actually writes.
+:::
+
+:::tip Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) is
+a complete job that syncs both directions and verifies the row counts.
+:::
+
+## Running a sync as its own job
+
+The jar also ships a `spark-submit` entry point, which is the equivalent of `RunSync` for this
+bundle. Pass the jar as the application jar rather than with `--jars`:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  --master 'local[*]' \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --basepath /path/to/hudi_table \
+  --sourceformat HUDI \
+  --targets ICEBERG,DELTA
+```
+
+To sync several tables in one submit, use `--datasetconfig`. It takes the same YAML that `RunSync`
+uses, so an existing config works unchanged, and unlike `RunSync` the config itself may live on
+cloud storage:
+
+```yaml md title="my_config.yaml"
+sourceFormat: HUDI
+targetFormats:
+  - DELTA
+  - ICEBERG
+datasets:
+  -
+    tableBasePath: s3://tpc-ds-datasets/1GB/hudi/call_center
+    tableDataPath: s3://tpc-ds-datasets/1GB/hudi/call_center/data
+    tableName: call_center
+    namespace: my.db
+  -
+    tableBasePath: s3://tpc-ds-datasets/1GB/hudi/catalog_sales
+    tableName: catalog_sales
+    partitionSpec: cs_sold_date_sk:VALUE
+  -
+    tableBasePath: s3://hudi/multi-partition-dataset
+    tableName: multi_partition_dataset
+    partitionSpec: time_millis:DAY:yyyy-MM-dd,type:VALUE
+```
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --datasetconfig my_config.yaml
+```
+
+`--datasetconfig` and `--basepath` are mutually exclusive — pass one or the other.
+
+| Option | Description |
+| --- | --- |
+| `--basepath` | Base path of the source table. |
+| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. |
+| `--targets` | Comma-separated target formats, for example `ICEBERG,DELTA`. |
+| `--datasetconfig` | Path to a YAML config listing several tables. May be local or on cloud storage. |
+| `--datapath` | Path to the data files, when it differs from the base path. |
+| `--tablename` | Table name. Defaults to the last segment of the base path. |
+| `--namespace` | Dot-separated table namespace. |
+| `--partitionspec` | Hudi source partition field spec, for example `level:VALUE`. |
+| `--usedeltakernel` | Force Delta Kernel for a Delta source or target. |
+| `--help` | Print the usage text. |
+
+## Supported formats
+
+Paimon and Parquet are read-only sources; XTable does not write either format as a target.
+
+| Source ↓ / Target → | Hudi | Iceberg | Delta |
+| --- | :---: | :---: | :---: |
+| **Hudi** | – | ✅ | ✅ |
+| **Iceberg** | ✅ | – | ✅ |
+| **Delta** | ✅ | ✅ | – |
+| **Paimon** | ✅ | ✅ | ✅ |
+| **Parquet** | ✅ | ✅ | ✅ |
+
+## Spark version support
+
+Converting to and from Hudi and Iceberg doesn't require Spark at all, so those run on any of the
+Spark versions below. Delta is the only engine whose implementation depends on the Spark version,
+and the jar chooses the right one automatically:
+
+| Spark version | Hudi and Iceberg | Delta implementation |
+| --- | :---: | --- |
+| 3.4.x | ✅ | Delta Standalone |
+| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected automatically |

Review Comment:
   When we say 'newer', it implies compatibility with Spark 4.x. It's better to explicitly specify the supported version or remove the 'newer' word.



##########
website/releases/downloads.mdx:
##########
@@ -7,6 +7,11 @@ Apache XTable™ (incubating) releases are available under the Apache License, V
 
 To ensure that you have downloaded the true release, you should [verify](https://www.apache.org/info/verification.html) the integrity of the files using the signatures and checksums available in this page. All releases are signed using the [Apache XTable KEYS file](https://downloads.apache.org/incubator/xtable/KEYS).
 
+### Release 0.4.0-incubating
+* Source Release: [Apache XTable 0.4.0-incubating Source Release](https://dlcdn.apache.org/incubator/xtable/0.4.0-incubating/apache-xtable-0.4.0-incubating.src.tgz) ([asc](https://downloads.apache.org/incubator/xtable/0.4.0-incubating/apache-xtable-0.4.0-incubating.src.tgz.asc), [sha512](https://downloads.apache.org/incubator/xtable/0.4.0-incubating/apache-xtable-0.4.0-incubating.src.tgz.sha512))
+* Release Notes: [Release Notes 0.4.0-incubating](release-0.4.0-incubating.mdx)
+* Maven Search: [Apache XTable 0.4.0-incubating](https://central.sonatype.com/search?q=g:org.apache.xtable%20v:0.4.0-incubating&smo=true)
+
 ### Release 0.3.0-incubating

Review Comment:
   We need to add the links for each release hyper link.



##########
website/docs/how-to-spark-runtime.md:
##########
@@ -0,0 +1,220 @@
+---
+sidebar_position: 3
+title: "Run an XTable sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run an XTable sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache
+Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source
+table's metadata and writes the target format's metadata alongside the data that is already there.
+
+The jar is meant to be dropped into a Spark job you already run, so you don't need a separate
+process or a separate cluster to keep a table interoperable across formats.
+
+## Getting the jar
+
+The jar is published to Maven Central as
+`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and
+Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
+full list of releases.
+
+## Adding the sync to a Spark job
+
+This is what the jar is for. Your job already writes a table in one format, and you want that same
+table to be interoperable with the others as soon as the write finishes.
+
+Add the jar to the `spark-submit` you already use, alongside your application jar:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and
+hand it the session's Hadoop configuration:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against these classes, add the same Maven coordinates to your build with `provided`
+scope.
+
+The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling
+back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That
+makes it safe to call after every write.
+
+:::note Tables are addressed by path
+The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent
+of the `RunSync` Iceberg catalog config (`-i`) yet. When a table's data files don't sit directly
+under `basePath`, set `dataPath` to wherever they do, because that's the location each target
+writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but nothing in Iceberg
+requires it — `write.data.path` and object-storage layouts can put data files anywhere — so check
+where your table actually writes.
+:::
+
+:::tip Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) is
+a complete job that syncs both directions and verifies the row counts.
+:::
+
+## Running a sync as its own job
+
+The jar also ships a `spark-submit` entry point, which is the equivalent of `RunSync` for this
+bundle. Pass the jar as the application jar rather than with `--jars`:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  --master 'local[*]' \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --basepath /path/to/hudi_table \
+  --sourceformat HUDI \
+  --targets ICEBERG,DELTA
+```
+
+To sync several tables in one submit, use `--datasetconfig`. It takes the same YAML that `RunSync`
+uses, so an existing config works unchanged, and unlike `RunSync` the config itself may live on
+cloud storage:
+
+```yaml md title="my_config.yaml"
+sourceFormat: HUDI
+targetFormats:
+  - DELTA
+  - ICEBERG
+datasets:
+  -
+    tableBasePath: s3://tpc-ds-datasets/1GB/hudi/call_center
+    tableDataPath: s3://tpc-ds-datasets/1GB/hudi/call_center/data
+    tableName: call_center
+    namespace: my.db
+  -
+    tableBasePath: s3://tpc-ds-datasets/1GB/hudi/catalog_sales
+    tableName: catalog_sales
+    partitionSpec: cs_sold_date_sk:VALUE
+  -
+    tableBasePath: s3://hudi/multi-partition-dataset
+    tableName: multi_partition_dataset
+    partitionSpec: time_millis:DAY:yyyy-MM-dd,type:VALUE
+```
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --datasetconfig my_config.yaml
+```
+
+`--datasetconfig` and `--basepath` are mutually exclusive — pass one or the other.
+
+| Option | Description |
+| --- | --- |
+| `--basepath` | Base path of the source table. |
+| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. |
+| `--targets` | Comma-separated target formats, for example `ICEBERG,DELTA`. |
+| `--datasetconfig` | Path to a YAML config listing several tables. May be local or on cloud storage. |
+| `--datapath` | Path to the data files, when it differs from the base path. |
+| `--tablename` | Table name. Defaults to the last segment of the base path. |
+| `--namespace` | Dot-separated table namespace. |
+| `--partitionspec` | Hudi source partition field spec, for example `level:VALUE`. |
+| `--usedeltakernel` | Force Delta Kernel for a Delta source or target. |
+| `--help` | Print the usage text. |
+
+## Supported formats
+
+Paimon and Parquet are read-only sources; XTable does not write either format as a target.
+
+| Source ↓ / Target → | Hudi | Iceberg | Delta |
+| --- | :---: | :---: | :---: |
+| **Hudi** | – | ✅ | ✅ |
+| **Iceberg** | ✅ | – | ✅ |
+| **Delta** | ✅ | ✅ | – |
+| **Paimon** | ✅ | ✅ | ✅ |
+| **Parquet** | ✅ | ✅ | ✅ |
+
+## Spark version support
+
+Converting to and from Hudi and Iceberg doesn't require Spark at all, so those run on any of the
+Spark versions below. Delta is the only engine whose implementation depends on the Spark version,
+and the jar chooses the right one automatically:
+
+| Spark version | Hudi and Iceberg | Delta implementation |
+| --- | :---: | --- |
+| 3.4.x | ✅ | Delta Standalone |
+| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected automatically |
+
+Delta Standalone doesn't run on Spark 3.5, so on 3.5 and newer a Delta source or target is routed

Review Comment:
   Can we move/add this section under the Note section for better visibility?



##########
website/docs/how-to-spark-runtime.md:
##########
@@ -0,0 +1,220 @@
+---
+sidebar_position: 3
+title: "Run an XTable sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run an XTable sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache
+Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source
+table's metadata and writes the target format's metadata alongside the data that is already there.
+
+The jar is meant to be dropped into a Spark job you already run, so you don't need a separate
+process or a separate cluster to keep a table interoperable across formats.
+
+## Getting the jar
+
+The jar is published to Maven Central as
+`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and
+Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
+full list of releases.
+
+## Adding the sync to a Spark job
+
+This is what the jar is for. Your job already writes a table in one format, and you want that same
+table to be interoperable with the others as soon as the write finishes.
+
+Add the jar to the `spark-submit` you already use, alongside your application jar:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and
+hand it the session's Hadoop configuration:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against these classes, add the same Maven coordinates to your build with `provided`
+scope.
+
+The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling
+back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That
+makes it safe to call after every write.
+
+:::note Tables are addressed by path
+The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent
+of the `RunSync` Iceberg catalog config (`-i`) yet. When a table's data files don't sit directly
+under `basePath`, set `dataPath` to wherever they do, because that's the location each target
+writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but nothing in Iceberg
+requires it — `write.data.path` and object-storage layouts can put data files anywhere — so check
+where your table actually writes.
+:::
+
+:::tip Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) is

Review Comment:
   when i click on link, '404 - page not found' error is dispalyed.



##########
website/releases/downloads.mdx:
##########
@@ -7,6 +7,11 @@ Apache XTable™ (incubating) releases are available under the Apache License, V
 
 To ensure that you have downloaded the true release, you should [verify](https://www.apache.org/info/verification.html) the integrity of the files using the signatures and checksums available in this page. All releases are signed using the [Apache XTable KEYS file](https://downloads.apache.org/incubator/xtable/KEYS).
 
+### Release 0.4.0-incubating

Review Comment:
   We need to add the links for each release hyper link.



##########
website/docs/how-to-spark-runtime.md:
##########
@@ -0,0 +1,220 @@
+---
+sidebar_position: 3
+title: "Run an XTable sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run an XTable sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache
+Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source
+table's metadata and writes the target format's metadata alongside the data that is already there.
+
+The jar is meant to be dropped into a Spark job you already run, so you don't need a separate
+process or a separate cluster to keep a table interoperable across formats.
+
+## Getting the jar
+
+The jar is published to Maven Central as
+`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and
+Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
+full list of releases.
+
+## Adding the sync to a Spark job
+
+This is what the jar is for. Your job already writes a table in one format, and you want that same
+table to be interoperable with the others as soon as the write finishes.
+
+Add the jar to the `spark-submit` you already use, alongside your application jar:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and
+hand it the session's Hadoop configuration:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";

Review Comment:
   In the other quick-start examples, we used local paths, but here an S3 path is used instead. Could we update this to keep it consistent?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [855] Add the 0.4.0 release notes and a Spark runtime quickstart with a runnable demo [incubator-xtable]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #913:
URL: https://github.com/apache/incubator-xtable/pull/913#discussion_r3848815264


##########
website/docs/how-to-spark-runtime.md:
##########
@@ -0,0 +1,218 @@
+---
+sidebar_position: 3
+title: "Run an XTable sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run an XTable sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache
+Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source
+table's metadata and writes the target format's metadata alongside the data that is already there.
+
+The jar is meant to be dropped into a Spark job you already run, so you don't need a separate
+process or a separate cluster to keep a table interoperable across formats.
+
+## Getting the jar
+
+The jar is published to Maven Central as
+`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and
+Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
+full list of releases.
+
+## Adding the sync to a Spark job
+
+This is what the jar is for. Your job already writes a table in one format, and you want that same
+table to be interoperable with the others as soon as the write finishes.
+
+Add the jar to the `spark-submit` you already use, alongside your application jar:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and
+hand it the session's Hadoop configuration:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against these classes, add the same Maven coordinates to your build with `provided`
+scope.
+
+The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling
+back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That
+makes it safe to call after every write.
+
+:::note Tables are addressed by path
+The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent
+of the `RunSync` Iceberg catalog config (`-i`) yet. For an Iceberg source, point `basePath` at the

Review Comment:
   Good catch — that was wrong. `<basePath>/data` is just what the Hadoop catalog happens to produce; `write.data.path` and object-storage layouts can put data files anywhere, and Iceberg guarantees nothing about the layout.
   
   Fixed in db557cdf. The note now frames `dataPath` by what it's actually for rather than by an Iceberg convention:
   
   > When a table's data files don't sit directly under `basePath`, set `dataPath` to wherever they do, because that's the location each target writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but nothing in Iceberg requires it — `write.data.path` and object-storage layouts can put data files anywhere — so check where your table actually writes.
   
   The demo had the same overstatement in a code comment, so I scoped that one to the catalog the demo configures. No behaviour change — `dataPath` was always a free-form path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [855] Add the 0.4.0 release notes and a Spark runtime quickstart with a runnable demo [incubator-xtable]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #913:
URL: https://github.com/apache/incubator-xtable/pull/913#discussion_r3848795619


##########
website/docs/how-to-spark-runtime.md:
##########
@@ -0,0 +1,218 @@
+---
+sidebar_position: 3
+title: "Run an XTable sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run an XTable sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache
+Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source
+table's metadata and writes the target format's metadata alongside the data that is already there.
+
+The jar is meant to be dropped into a Spark job you already run, so you don't need a separate
+process or a separate cluster to keep a table interoperable across formats.
+
+## Getting the jar
+
+The jar is published to Maven Central as
+`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and
+Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
+full list of releases.
+
+## Adding the sync to a Spark job
+
+This is what the jar is for. Your job already writes a table in one format, and you want that same
+table to be interoperable with the others as soon as the write finishes.
+
+Add the jar to the `spark-submit` you already use, alongside your application jar:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and
+hand it the session's Hadoop configuration:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against these classes, add the same Maven coordinates to your build with `provided`
+scope.
+
+The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling
+back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That
+makes it safe to call after every write.
+
+:::note Tables are addressed by path
+The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent
+of the `RunSync` Iceberg catalog config (`-i`) yet. For an Iceberg source, point `basePath` at the

Review Comment:
   ```
   For an Iceberg source, point `basePath` at the
   table root and set `dataPath` to `<basePath>/data`, since that is where Iceberg keeps its data
   files and where each target writes its metadata
   ```
   This is not really true..the data files can be present anywhere for iceberg.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [855] Add the 0.4.0 release notes and a Spark runtime quickstart with a runnable demo [incubator-xtable]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #913:
URL: https://github.com/apache/incubator-xtable/pull/913#discussion_r3848657318


##########
website/releases/release-0.4.0-incubating.mdx:
##########
@@ -0,0 +1,81 @@
+---
+title: "Release 0.4.0-incubating"
+sidebar_position: -4
+---
+
+## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+
+This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files.
+Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats.
+
+## ✨ What's Changed
+
+### 📦 Run a sync on Apache Spark
+
+This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster
+you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit
+it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi,
+Iceberg and Delta libraries your cluster already supplies.
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It
+supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime).
+
+### 🔄 New source formats
+
+* **Apache Paimon source**, with incremental sync and column statistics.
+* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync.
+
+### 🆕 Delta Kernel
+
+* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs

Review Comment:
   DeltaKernelConversionSource is also present? 



##########
website/docs/spark-runtime.md:
##########
@@ -0,0 +1,237 @@
+---
+sidebar_position: 3
+title: "Run a sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run a sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an
+Apache Spark cluster. The sync reads the source table metadata and writes the target format
+metadata next to the data files that already exist. It rewrites no data.
+
+Add the jar to a Spark job you already run. You need no separate service and no separate cluster.
+
+## Get the jar
+
+The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`.
+Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+The [Downloads](/releases/downloads) page lists every release.
+
+## Add the sync to a Spark job
+
+This is the main use case. Your job already writes a table in one format. You want the same
+table to be readable in the other formats when the write completes.
+
+Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you
+already submit:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the
+table and pass the Hadoop configuration of the Spark session:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against the classes, add the same coordinates to your build with `provided` scope.
+
+`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls
+back to a full snapshot when an incremental sync is not safe, such as the first run. A call
+after every write is therefore safe to repeat.
+
+For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files
+there, and each target writes its metadata next to those files.
+
+:::note Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime)
+holds a complete job that runs both directions and checks the row counts.
+:::
+
+## Why `--jars` and not `--packages`
+
+The runtime jar carries only XTable code plus a few relocated helper libraries. It declares
+Spark, Hadoop, Hudi, Iceberg, Delta, Avro and Parquet as `provided`, so it uses the copies your
+cluster already supplies.
+
+Use `--jars`. It takes the file you downloaded and needs no network access at submit time.
+
+`--packages` also works. Every dependency in the published POM is `provided` or `test` scoped,
+so Ivy downloads this one jar and no transitive jars. It costs you a Maven Central lookup on
+each submit, which is the only reason to prefer `--jars`.
+
+:::danger Do not add the engines with `--packages`
+Add Hudi, Iceberg or Delta with `--packages` only when your cluster does not already supply
+them. `--packages` and `--jars` load classes in a child class loader. A second copy of Avro or
+Parquet there breaks casts across the class loader boundary. Put engine jars on the flat
+classpath with `spark.driver.extraClassPath` and `spark.executor.extraClassPath` instead.
+:::
+
+## Run a sync without a job
+
+The jar also ships a `spark-submit` entry point. Use it when you want the sync as a separate
+step. Pass the jar as the application jar, not with `--jars`:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  --master 'local[*]' \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --basepath /path/to/hudi_table \
+  --sourceformat HUDI \
+  --targets ICEBERG,DELTA
+```
+
+To sync many tables in one submit, pass `--datasetconfig` with a YAML file. `sourceFormat` and
+`targetFormats` apply to every table. Each entry under `datasets` needs only a `tableBasePath`.
+The config path can be local or on cloud storage.
+
+```yaml md title="dataset.yaml"
+sourceFormat: HUDI
+targetFormats:
+  - ICEBERG
+  - DELTA
+datasets:
+  - tableBasePath: s3://example-warehouse/db/store_sales
+  - tableBasePath: s3://example-warehouse/db/store_returns
+  - tableBasePath: s3://example-warehouse/db/item
+```
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --datasetconfig dataset.yaml
+```
+
+`--datasetconfig` and `--basepath` are mutually exclusive. Pass one of the two.
+
+| Option | Description |
+| --- | --- |
+| `--basepath` | Base path of the source table. |
+| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. |
+| `--targets` | Comma-separated target formats, such as `ICEBERG,DELTA`. |
+| `--datasetconfig` | Path to a YAML config that lists many tables. |
+| `--datapath` | Path to the data files when it differs from the base path. |
+| `--tablename` | Table name. Defaults to the last segment of the base path. |
+| `--namespace` | Dot-separated table namespace. |
+| `--partitionspec` | Hudi source partition field spec, such as `level:VALUE`. |
+| `--usedeltakernel` | Force the Spark-free Delta Kernel for a Delta source or target. |
+| `--help` | Print the usage text. |
+
+## Supported formats
+
+Paimon and Parquet are read-only sources. XTable writes no Paimon or Parquet target.
+
+| Source ↓ / Target → | Hudi | Iceberg | Delta |
+| --- | :---: | :---: | :---: |
+| **Hudi** | – | ✅ | ✅ |
+| **Iceberg** | ✅ | – | ✅ |
+| **Delta** | ✅ | ✅ | – |
+| **Paimon** | ✅ | ✅ | ✅ |
+| **Parquet** | ✅ | ✅ | ✅ |
+
+## Spark versions
+
+Hudi and Iceberg conversion use Spark-free core classes only, so they run on every Spark line
+below. Delta is the one engine that depends on the Spark version, and the jar picks the right
+implementation for you.
+
+| Spark version | Hudi and Iceberg | Delta implementation |
+| --- | :---: | --- |
+| 3.4.x | ✅ | Delta Standalone (`delta-core`) |
+| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected for you |
+
+On Spark 3.5 and newer, `delta-core` does not run. A Delta source or target goes through the

Review Comment:
   Avoid using delta-core use delta standalone or delta-kernel. 



##########
website/docs/spark-runtime.md:
##########
@@ -0,0 +1,237 @@
+---
+sidebar_position: 3
+title: "Run a sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run a sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an
+Apache Spark cluster. The sync reads the source table metadata and writes the target format
+metadata next to the data files that already exist. It rewrites no data.
+
+Add the jar to a Spark job you already run. You need no separate service and no separate cluster.
+
+## Get the jar
+
+The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`.
+Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+The [Downloads](/releases/downloads) page lists every release.
+
+## Add the sync to a Spark job
+
+This is the main use case. Your job already writes a table in one format. You want the same
+table to be readable in the other formats when the write completes.
+
+Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you
+already submit:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the
+table and pass the Hadoop configuration of the Spark session:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against the classes, add the same coordinates to your build with `provided` scope.
+
+`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls
+back to a full snapshot when an incremental sync is not safe, such as the first run. A call
+after every write is therefore safe to repeat.
+
+For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files
+there, and each target writes its metadata next to those files.
+
+:::note Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime)
+holds a complete job that runs both directions and checks the row counts.
+:::
+
+## Why `--jars` and not `--packages`
+
+The runtime jar carries only XTable code plus a few relocated helper libraries. It declares
+Spark, Hadoop, Hudi, Iceberg, Delta, Avro and Parquet as `provided`, so it uses the copies your
+cluster already supplies.
+
+Use `--jars`. It takes the file you downloaded and needs no network access at submit time.
+
+`--packages` also works. Every dependency in the published POM is `provided` or `test` scoped,
+so Ivy downloads this one jar and no transitive jars. It costs you a Maven Central lookup on
+each submit, which is the only reason to prefer `--jars`.
+
+:::danger Do not add the engines with `--packages`
+Add Hudi, Iceberg or Delta with `--packages` only when your cluster does not already supply
+them. `--packages` and `--jars` load classes in a child class loader. A second copy of Avro or
+Parquet there breaks casts across the class loader boundary. Put engine jars on the flat
+classpath with `spark.driver.extraClassPath` and `spark.executor.extraClassPath` instead.
+:::
+
+## Run a sync without a job
+
+The jar also ships a `spark-submit` entry point. Use it when you want the sync as a separate
+step. Pass the jar as the application jar, not with `--jars`:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  --master 'local[*]' \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --basepath /path/to/hudi_table \
+  --sourceformat HUDI \
+  --targets ICEBERG,DELTA
+```
+
+To sync many tables in one submit, pass `--datasetconfig` with a YAML file. `sourceFormat` and
+`targetFormats` apply to every table. Each entry under `datasets` needs only a `tableBasePath`.
+The config path can be local or on cloud storage.
+
+```yaml md title="dataset.yaml"
+sourceFormat: HUDI
+targetFormats:
+  - ICEBERG
+  - DELTA
+datasets:
+  - tableBasePath: s3://example-warehouse/db/store_sales
+  - tableBasePath: s3://example-warehouse/db/store_returns
+  - tableBasePath: s3://example-warehouse/db/item
+```
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --datasetconfig dataset.yaml
+```
+
+`--datasetconfig` and `--basepath` are mutually exclusive. Pass one of the two.
+
+| Option | Description |
+| --- | --- |
+| `--basepath` | Base path of the source table. |
+| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. |
+| `--targets` | Comma-separated target formats, such as `ICEBERG,DELTA`. |
+| `--datasetconfig` | Path to a YAML config that lists many tables. |
+| `--datapath` | Path to the data files when it differs from the base path. |
+| `--tablename` | Table name. Defaults to the last segment of the base path. |
+| `--namespace` | Dot-separated table namespace. |
+| `--partitionspec` | Hudi source partition field spec, such as `level:VALUE`. |
+| `--usedeltakernel` | Force the Spark-free Delta Kernel for a Delta source or target. |
+| `--help` | Print the usage text. |
+
+## Supported formats
+
+Paimon and Parquet are read-only sources. XTable writes no Paimon or Parquet target.
+
+| Source ↓ / Target → | Hudi | Iceberg | Delta |
+| --- | :---: | :---: | :---: |
+| **Hudi** | – | ✅ | ✅ |
+| **Iceberg** | ✅ | – | ✅ |
+| **Delta** | ✅ | ✅ | – |
+| **Paimon** | ✅ | ✅ | ✅ |
+| **Parquet** | ✅ | ✅ | ✅ |
+
+## Spark versions
+
+Hudi and Iceberg conversion use Spark-free core classes only, so they run on every Spark line
+below. Delta is the one engine that depends on the Spark version, and the jar picks the right
+implementation for you.

Review Comment:
   Spark free core classes is very vauge and meaningless say don't need spark for the conversion?



##########
website/docs/spark-runtime.md:
##########
@@ -0,0 +1,237 @@
+---
+sidebar_position: 3
+title: "Run a sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run a sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an
+Apache Spark cluster. The sync reads the source table metadata and writes the target format
+metadata next to the data files that already exist. It rewrites no data.
+
+Add the jar to a Spark job you already run. You need no separate service and no separate cluster.
+
+## Get the jar
+
+The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`.
+Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+The [Downloads](/releases/downloads) page lists every release.
+
+## Add the sync to a Spark job
+
+This is the main use case. Your job already writes a table in one format. You want the same
+table to be readable in the other formats when the write completes.
+
+Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you
+already submit:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the
+table and pass the Hadoop configuration of the Spark session:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against the classes, add the same coordinates to your build with `provided` scope.
+
+`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls
+back to a full snapshot when an incremental sync is not safe, such as the first run. A call
+after every write is therefore safe to repeat.
+
+For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files
+there, and each target writes its metadata next to those files.
+
+:::note Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime)
+holds a complete job that runs both directions and checks the row counts.
+:::
+
+## Why `--jars` and not `--packages`
+
+The runtime jar carries only XTable code plus a few relocated helper libraries. It declares
+Spark, Hadoop, Hudi, Iceberg, Delta, Avro and Parquet as `provided`, so it uses the copies your
+cluster already supplies.
+
+Use `--jars`. It takes the file you downloaded and needs no network access at submit time.
+
+`--packages` also works. Every dependency in the published POM is `provided` or `test` scoped,
+so Ivy downloads this one jar and no transitive jars. It costs you a Maven Central lookup on
+each submit, which is the only reason to prefer `--jars`.
+
+:::danger Do not add the engines with `--packages`
+Add Hudi, Iceberg or Delta with `--packages` only when your cluster does not already supply
+them. `--packages` and `--jars` load classes in a child class loader. A second copy of Avro or
+Parquet there breaks casts across the class loader boundary. Put engine jars on the flat
+classpath with `spark.driver.extraClassPath` and `spark.executor.extraClassPath` instead.
+:::
+
+## Run a sync without a job
+
+The jar also ships a `spark-submit` entry point. Use it when you want the sync as a separate
+step. Pass the jar as the application jar, not with `--jars`:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  --master 'local[*]' \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --basepath /path/to/hudi_table \
+  --sourceformat HUDI \
+  --targets ICEBERG,DELTA
+```
+
+To sync many tables in one submit, pass `--datasetconfig` with a YAML file. `sourceFormat` and
+`targetFormats` apply to every table. Each entry under `datasets` needs only a `tableBasePath`.
+The config path can be local or on cloud storage.
+
+```yaml md title="dataset.yaml"
+sourceFormat: HUDI
+targetFormats:
+  - ICEBERG
+  - DELTA
+datasets:
+  - tableBasePath: s3://example-warehouse/db/store_sales
+  - tableBasePath: s3://example-warehouse/db/store_returns
+  - tableBasePath: s3://example-warehouse/db/item
+```
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --datasetconfig dataset.yaml
+```
+
+`--datasetconfig` and `--basepath` are mutually exclusive. Pass one of the two.
+
+| Option | Description |
+| --- | --- |
+| `--basepath` | Base path of the source table. |
+| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. |
+| `--targets` | Comma-separated target formats, such as `ICEBERG,DELTA`. |
+| `--datasetconfig` | Path to a YAML config that lists many tables. |
+| `--datapath` | Path to the data files when it differs from the base path. |
+| `--tablename` | Table name. Defaults to the last segment of the base path. |
+| `--namespace` | Dot-separated table namespace. |
+| `--partitionspec` | Hudi source partition field spec, such as `level:VALUE`. |
+| `--usedeltakernel` | Force the Spark-free Delta Kernel for a Delta source or target. |
+| `--help` | Print the usage text. |
+
+## Supported formats
+
+Paimon and Parquet are read-only sources. XTable writes no Paimon or Parquet target.
+
+| Source ↓ / Target → | Hudi | Iceberg | Delta |
+| --- | :---: | :---: | :---: |
+| **Hudi** | – | ✅ | ✅ |
+| **Iceberg** | ✅ | – | ✅ |
+| **Delta** | ✅ | ✅ | – |
+| **Paimon** | ✅ | ✅ | ✅ |
+| **Parquet** | ✅ | ✅ | ✅ |
+
+## Spark versions
+
+Hudi and Iceberg conversion use Spark-free core classes only, so they run on every Spark line
+below. Delta is the one engine that depends on the Spark version, and the jar picks the right
+implementation for you.
+
+| Spark version | Hudi and Iceberg | Delta implementation |
+| --- | :---: | --- |
+| 3.4.x | ✅ | Delta Standalone (`delta-core`) |
+| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected for you |
+
+On Spark 3.5 and newer, `delta-core` does not run. A Delta source or target goes through the
+Spark-free Delta Kernel instead, with no flag. To force Kernel on Spark 3.4, pass
+`--usedeltakernel`, or set `.useDeltaKernel(true)` on the `TableSyncSpec`.
+
+## Read a Hudi target
+
+XTable records the file listing of a Hudi target in the Hudi metadata table. Enable that table
+in the reader. Without the option, the sync reports success and the read returns zero rows.
+
+```scala md title="scala"
+spark.read
+  .option("hoodie.metadata.enable", "true")
+  .option("hoodie.datasource.read.extract.partition.values.from.path", "true")
+  .format("hudi")
+  .load(basePath)
+```
+
+Iceberg and Delta targets need no extra read option. One exception: the vectorized Arrow reader
+of Iceberg mis-casts timestamp columns on a table that another engine wrote. Pass
+`vectorization-enabled=false` when you see that.

Review Comment:
   Remove this `Read a Hudi target` section and internal details on read completely..these are open gaps depending on the hudi/iceberg versions used not something that needs to be documented in xtable docs.



##########
website/docs/setup.md:
##########
@@ -23,4 +23,7 @@ For more information on the steps, follow the project's GitHub [README.md](https
 
 ## Next Steps
 See the [Quickstart](/docs/how-to) guide to learn to use Apache XTable™ (Incubating) to add interoperability between
-different table formats.
\ No newline at end of file
+different table formats.
+
+To run a sync on an Apache Spark cluster you already have, see
+[Run a sync on Apache Spark](/docs/spark-runtime).

Review Comment:
   Should this have the same how-to prefix? 



##########
website/releases/release-0.4.0-incubating.mdx:
##########
@@ -0,0 +1,81 @@
+---
+title: "Release 0.4.0-incubating"
+sidebar_position: -4
+---
+
+## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+
+This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files.
+Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats.
+
+## ✨ What's Changed
+
+### 📦 Run a sync on Apache Spark
+
+This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster
+you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit
+it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi,
+Iceberg and Delta libraries your cluster already supplies.
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It
+supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime).
+
+### 🔄 New source formats
+
+* **Apache Paimon source**, with incremental sync and column statistics.
+* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync.
+
+### 🆕 Delta Kernel
+
+* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs
+  on Spark 3.5 and newer, where `delta-core` does not.
+
+### 🔧 Conversion improvements
+
+* Delta source: extract statistics for nested fields.
+* Hudi source: support nested partition columns when hive-style partitioning is on.
+* Fall back to Parquet footers when the metadata table has no column statistics.
+* Throw on a bad Delta checkpoint CRC by default.

Review Comment:
   `Throw on a bad Delta checkpoint CRC by default` 
   Re-word this say fail fast instead? 



##########
website/docs/spark-runtime.md:
##########
@@ -0,0 +1,237 @@
+---
+sidebar_position: 3
+title: "Run a sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run a sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an
+Apache Spark cluster. The sync reads the source table metadata and writes the target format
+metadata next to the data files that already exist. It rewrites no data.
+
+Add the jar to a Spark job you already run. You need no separate service and no separate cluster.
+
+## Get the jar
+
+The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`.
+Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+The [Downloads](/releases/downloads) page lists every release.
+
+## Add the sync to a Spark job
+
+This is the main use case. Your job already writes a table in one format. You want the same
+table to be readable in the other formats when the write completes.
+
+Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you
+already submit:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the
+table and pass the Hadoop configuration of the Spark session:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against the classes, add the same coordinates to your build with `provided` scope.
+
+`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls
+back to a full snapshot when an incremental sync is not safe, such as the first run. A call
+after every write is therefore safe to repeat.
+
+For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files
+there, and each target writes its metadata next to those files.
+
+:::note Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime)
+holds a complete job that runs both directions and checks the row counts.
+:::
+
+## Why `--jars` and not `--packages`

Review Comment:
   Remove this section..the user knows how spark works and not something that needs to be added in xtable docs. 



##########
website/releases/release-0.4.0-incubating.mdx:
##########
@@ -0,0 +1,81 @@
+---
+title: "Release 0.4.0-incubating"
+sidebar_position: -4
+---
+
+## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+
+This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files.
+Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats.
+
+## ✨ What's Changed
+
+### 📦 Run a sync on Apache Spark
+
+This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster
+you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit
+it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi,
+Iceberg and Delta libraries your cluster already supplies.
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It
+supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime).
+
+### 🔄 New source formats
+
+* **Apache Paimon source**, with incremental sync and column statistics.
+* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync.
+
+### 🆕 Delta Kernel
+
+* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs
+  on Spark 3.5 and newer, where `delta-core` does not.
+
+### 🔧 Conversion improvements
+
+* Delta source: extract statistics for nested fields.
+* Hudi source: support nested partition columns when hive-style partitioning is on.
+* Fall back to Parquet footers when the metadata table has no column statistics.
+* Throw on a bad Delta checkpoint CRC by default.
+* Pass additional properties through to a Hudi target.

Review Comment:
   `Pass additional properties through to a Hudi target` 
   This is not worth something that needs to be documented on the site.



##########
website/docs/spark-runtime.md:
##########
@@ -0,0 +1,237 @@
+---
+sidebar_position: 3
+title: "Run a sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run a sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an
+Apache Spark cluster. The sync reads the source table metadata and writes the target format
+metadata next to the data files that already exist. It rewrites no data.
+
+Add the jar to a Spark job you already run. You need no separate service and no separate cluster.
+
+## Get the jar
+
+The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`.
+Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+The [Downloads](/releases/downloads) page lists every release.
+
+## Add the sync to a Spark job
+
+This is the main use case. Your job already writes a table in one format. You want the same
+table to be readable in the other formats when the write completes.
+
+Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you
+already submit:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the
+table and pass the Hadoop configuration of the Spark session:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against the classes, add the same coordinates to your build with `provided` scope.
+
+`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls
+back to a full snapshot when an incremental sync is not safe, such as the first run. A call
+after every write is therefore safe to repeat.
+
+For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files

Review Comment:
   For iceberg document the catalog path instead? No one uses the absolute paths for iceberg writes. 



##########
website/releases/release-0.4.0-incubating.mdx:
##########
@@ -0,0 +1,81 @@
+---
+title: "Release 0.4.0-incubating"
+sidebar_position: -4
+---
+
+## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to))
+
+This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files.
+Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats.
+
+## ✨ What's Changed
+
+### 📦 Run a sync on Apache Spark
+
+This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster
+you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit
+it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi,
+Iceberg and Delta libraries your cluster already supplies.
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It
+supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime).
+
+### 🔄 New source formats
+
+* **Apache Paimon source**, with incremental sync and column statistics.
+* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync.
+
+### 🆕 Delta Kernel
+
+* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs

Review Comment:
   `Delta Kernel needs no Spark, so a Delta sync now runs on Spark 3.5 and newer, where `delta-core` does not` 
   Simplify this it's repetitive, and delta-core the user isn't aware of the internal details like delta-core is the standalone library..?  



##########
website/docs/spark-runtime.md:
##########
@@ -0,0 +1,237 @@
+---
+sidebar_position: 3
+title: "Run a sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run a sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an
+Apache Spark cluster. The sync reads the source table metadata and writes the target format
+metadata next to the data files that already exist. It rewrites no data.
+
+Add the jar to a Spark job you already run. You need no separate service and no separate cluster.
+
+## Get the jar
+
+The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`.
+Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+The [Downloads](/releases/downloads) page lists every release.
+
+## Add the sync to a Spark job
+
+This is the main use case. Your job already writes a table in one format. You want the same
+table to be readable in the other formats when the write completes.
+
+Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you
+already submit:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the
+table and pass the Hadoop configuration of the Spark session:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against the classes, add the same coordinates to your build with `provided` scope.
+
+`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls
+back to a full snapshot when an incremental sync is not safe, such as the first run. A call
+after every write is therefore safe to repeat.
+
+For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files

Review Comment:
   Using --catalogConfig? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [855] Add the 0.4.0 release notes and a Spark runtime quickstart with a runnable demo [incubator-xtable]

Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #913:
URL: https://github.com/apache/incubator-xtable/pull/913#discussion_r3848767930


##########
website/docs/spark-runtime.md:
##########
@@ -0,0 +1,237 @@
+---
+sidebar_position: 3
+title: "Run a sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run a sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an
+Apache Spark cluster. The sync reads the source table metadata and writes the target format
+metadata next to the data files that already exist. It rewrites no data.
+
+Add the jar to a Spark job you already run. You need no separate service and no separate cluster.
+
+## Get the jar
+
+The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`.
+Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+The [Downloads](/releases/downloads) page lists every release.
+
+## Add the sync to a Spark job
+
+This is the main use case. Your job already writes a table in one format. You want the same
+table to be readable in the other formats when the write completes.
+
+Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you
+already submit:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the
+table and pass the Hadoop configuration of the Spark session:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against the classes, add the same coordinates to your build with `provided` scope.
+
+`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls
+back to a full snapshot when an incremental sync is not safe, such as the first run. A call
+after every write is therefore safe to repeat.
+
+For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files

Review Comment:
   Checked `XTableSparkSync` on the 0.4.0 tag and there's no catalog support at all — the options are `--basepath`, `--datapath`, `--sourceformat`, `--targets`, `--tablename`, `--namespace`, `--partitionspec`, `--datasetconfig`, `--usedeltakernel`. So the runtime jar is path-based only today and I can't document catalog usage that doesn't exist yet.
   
   Added a short note in 8248b3f6 instead of burying it:
   
   > **Tables are addressed by path** — The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent of the `RunSync` Iceberg catalog config (`-i`) yet. For an Iceberg source, point `basePath` at the table root and set `dataPath` to `<basePath>/data`.
   
   Happy to open a follow-up issue for catalog support in the runtime jar if you want it tracked.



##########
website/docs/spark-runtime.md:
##########
@@ -0,0 +1,237 @@
+---
+sidebar_position: 3
+title: "Run a sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run a sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an
+Apache Spark cluster. The sync reads the source table metadata and writes the target format
+metadata next to the data files that already exist. It rewrites no data.
+
+Add the jar to a Spark job you already run. You need no separate service and no separate cluster.
+
+## Get the jar
+
+The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`.
+Download it once:
+
+```shell md title="shell"
+curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+The [Downloads](/releases/downloads) page lists every release.
+
+## Add the sync to a Spark job
+
+This is the main use case. Your job already writes a table in one format. You want the same
+table to be readable in the other formats when the write completes.
+
+Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you
+already submit:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+  --class com.example.OrdersJob \
+  orders-job.jar
+```
+
+Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the
+table and pass the Hadoop configuration of the Spark session:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+  TableSyncSpec.builder()
+    .key("orders")
+    .basePath(basePath)
+    .sourceFormat("HUDI")
+    .targets(JArrays.asList("ICEBERG", "DELTA"))
+    .build(),
+  spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+    .sync(
+        TableSyncSpec.builder()
+            .key("orders")
+            .basePath(basePath)
+            .sourceFormat("HUDI")
+            .targets(Arrays.asList("ICEBERG", "DELTA"))
+            .build(),
+        spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against the classes, add the same coordinates to your build with `provided` scope.
+
+`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls
+back to a full snapshot when an incremental sync is not safe, such as the first run. A call
+after every write is therefore safe to repeat.
+
+For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files

Review Comment:
   There's no `--catalogConfig` on `XTableSparkSync` — only `RunSync` has the Iceberg catalog config, as `-i`, and it's a separate file from the dataset YAML.
   
   Worth noting the dataset config itself *is* shared. I diffed `RunSync.DatasetConfig` against `XTableSparkSync.DatasetConfig` and the field sets are identical (`sourceFormat`, `targetFormats`, and `datasets[]` with `tableBasePath`, `tableDataPath`, `tableName`, `partitionSpec`, `namespace`), which matches the javadoc: *"mirrors the RunSync dataset YAML so configs are shareable."*
   
   So in 8248b3f6 I dropped my invented example and used the config from the project README verbatim, and the page now says it's the same file `RunSync` takes. One bonus: `XTableSparkSync` reads that path through the Spark Hadoop config, so the config can live on S3/GCS/ABFS, whereas `RunSync` uses `Files.newInputStream` and needs it local.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org