You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@xtable.apache.org by "vinishjail97 (via GitHub)" <gi...@apache.org> on 2026/09/08 21:21:26 UTC
Re: [PR] [920] Address the review comments on the 0.4.0 Spark runtime docs [incubator-xtable]
vinishjail97 commented on code in PR #922:
URL: https://github.com/apache/incubator-xtable/pull/922#discussion_r3962285712
##########
website/docs/how-to-spark-runtime.md:
##########
@@ -28,22 +32,101 @@ Every engine dependency is `provided`, so the jar is about 4 MB and reuses the H
Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
full list of releases.
-## Adding the sync to a Spark job
+## Prerequisites
+
+XTable converts the metadata of a table that already exists, so you need a source table before you
+can sync anything. If you don't have one, follow
+[Creating your first interoperable table](/docs/how-to) to create the Hudi table `people` under
+`file:///tmp/hudi-dataset`. Every example on this page syncs that table.
+
+You also need a Spark 3.4.x or 3.5.x installation with `$SPARK_HOME` set. See
Review Comment:
The spark runtime POM marks the Hudi, Iceberg, and Delta dependencies as `provided`, and `ITXTableSparkRuntimeBundle` adds both `spark.driver.extraClassPath` and `spark.executor.extraClassPath` before invoking this entry point.
The linked Quickstart adds Hudi only to its earlier PySpark process, not this separate submit, while this example also needs Iceberg and Delta. Would it be reasonable to document the required engine coordinates or classpath here and include them in the runnable command? I think this blocks the quickstart from working as written.
##########
website/docs/how-to-spark-runtime.md:
##########
@@ -28,22 +32,101 @@ Every engine dependency is `provided`, so the jar is about 4 MB and reuses the H
Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
full list of releases.
-## Adding the sync to a Spark job
+## Prerequisites
+
+XTable converts the metadata of a table that already exists, so you need a source table before you
+can sync anything. If you don't have one, follow
+[Creating your first interoperable table](/docs/how-to) to create the Hudi table `people` under
+`file:///tmp/hudi-dataset`. Every example on this page syncs that table.
+
+You also need a Spark 3.4.x or 3.5.x installation with `$SPARK_HOME` set. See
+[Spark version support](#spark-version-support) for what each line provides.
-This is what the jar is for. Your job already writes a table in one format, and you want that same
-table to be interoperable with the others as soon as the write finishes.
+## Running a sync as its own job
+
+The jar ships a `spark-submit` entry point, `org.apache.xtable.spark.XTableSparkSync`. Pass the jar
+as the application jar rather than with `--jars`. It runs in one of two modes.
-Add the jar to the `spark-submit` you already use, alongside your application jar:
+### Syncing a single table
+
+Describe the table with command line options. This is the shortest path from a source table to an
+interoperable one:
```shell md title="shell"
$SPARK_HOME/bin/spark-submit \
- --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
- --class com.example.OrdersJob \
- orders-job.jar
+ --class org.apache.xtable.spark.XTableSparkSync \
+ --master 'local[*]' \
+ xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+ --basepath file:///tmp/hudi-dataset/people \
+ --sourceformat HUDI \
+ --targets ICEBERG,DELTA
```
-Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and
-hand it the session's Hadoop configuration:
+After this finishes, `/tmp/hudi-dataset/people` carries Iceberg and Delta metadata next to the Hudi
+data files, and all three formats read the same rows.
+
+### Syncing several tables from a config file
+
+To sync more than one table in a single submit, list them in a YAML file and pass
+`--datasetconfig`. It takes the same config that `RunSync` uses, so an existing file works
+unchanged:
+
+```yaml md title="my_config.yaml"
+sourceFormat: HUDI
+targetFormats:
+ - DELTA
+ - ICEBERG
+datasets:
+ -
+ tableBasePath: file:///tmp/hudi-dataset/people
+ tableName: people
+ -
+ tableBasePath: file:///tmp/hudi-dataset/orders
Review Comment:
I am not sure this example can succeed after following the prerequisite above. The Quickstart creates only `people`, while this configuration also syncs `orders` and `events`. `XTableSparkSync` records each missing source as a failure and exits nonzero after processing the list. Would it work to use source paths that the page creates, or label these two entries as placeholders and explain that readers must create them first? Because the page says every example uses the created `people` table, I think the current example is blocking for a runnable quickstart.
##########
website/docs/how-to-spark-runtime.md:
##########
@@ -28,22 +32,101 @@ Every engine dependency is `provided`, so the jar is about 4 MB and reuses the H
Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
full list of releases.
-## Adding the sync to a Spark job
+## Prerequisites
+
+XTable converts the metadata of a table that already exists, so you need a source table before you
+can sync anything. If you don't have one, follow
+[Creating your first interoperable table](/docs/how-to) to create the Hudi table `people` under
+`file:///tmp/hudi-dataset`. Every example on this page syncs that table.
+
+You also need a Spark 3.4.x or 3.5.x installation with `$SPARK_HOME` set. See
+[Spark version support](#spark-version-support) for what each line provides.
-This is what the jar is for. Your job already writes a table in one format, and you want that same
-table to be interoperable with the others as soon as the write finishes.
+## Running a sync as its own job
+
+The jar ships a `spark-submit` entry point, `org.apache.xtable.spark.XTableSparkSync`. Pass the jar
+as the application jar rather than with `--jars`. It runs in one of two modes.
-Add the jar to the `spark-submit` you already use, alongside your application jar:
+### Syncing a single table
+
+Describe the table with command line options. This is the shortest path from a source table to an
+interoperable one:
```shell md title="shell"
$SPARK_HOME/bin/spark-submit \
- --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
- --class com.example.OrdersJob \
- orders-job.jar
+ --class org.apache.xtable.spark.XTableSparkSync \
+ --master 'local[*]' \
+ xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+ --basepath file:///tmp/hudi-dataset/people \
+ --sourceformat HUDI \
+ --targets ICEBERG,DELTA
Review Comment:
The linked Quickstart writes `people` partitioned by `city`, and its own RunSync configuration passes `partitionSpec: city:VALUE`. This command and both `TableSyncSpec` examples omit that setting. In 0.4.0, `HudiSourceConfig` converts a missing setting to an empty partition-field list, so the target metadata is created as unpartitioned. Would it be reasonable to add `--partitionspec city:VALUE` here and `.partitionSpec("city:VALUE")` in both embedded examples? I think preserving the source partition specification is part of the documented interoperability result, so this looks blocking to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org