You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@xtable.apache.org by "rangareddy (via GitHub)" <gi...@apache.org> on 2026/08/26 06:36:10 UTC
[I] Docs: Delta Kernel is not auto-selected for in-job syncs, only for the XTableSparkSync CLI [incubator-xtable]
rangareddy opened a new issue, #921:
URL: https://github.com/apache/incubator-xtable/issues/921
`website/docs/how-to-spark-runtime.md` tells readers that the Delta implementation is chosen for
them on Spark 3.5:
> Delta is the only engine whose implementation depends on the Spark version, and the jar chooses
> the right one automatically
> | 3.5.x and newer | ✅ | Delta Kernel, selected automatically |
> Delta Standalone doesn't run on Spark 3.5, so on 3.5 and newer a Delta source or target is
> routed through Delta Kernel with no flag needed.
That holds for the CLI entry point only. In `XTableSparkSync.main`, on the `0.4.0-incubating` tag:
```java
boolean deltaInvolved =
TableFormat.DELTA.equals(sourceFormat) || targetFormats.contains(TableFormat.DELTA);
boolean useDeltaKernel = useDeltaKernelFlag;
if (!useDeltaKernel && deltaInvolved && isSparkAtLeast35(spark.version())) {
useDeltaKernel = true;
...
}
```
The in-job API has no such logic. `XTableSyncService.sync` reads the flag straight off the spec:
```java
ConversionSourceProvider<?> sourceProvider =
sourceProviderFor(spec.getSourceFormat(), spec.isUseDeltaKernel());
```
and `TableSyncSpec.useDeltaKernel` is a plain `boolean` that defaults to `false`. Nothing in
`xtable-spark-runtime` inspects the Spark version outside `XTableSparkSync`; grepping the module
on the tag returns `isSparkAtLeast35` as the only call site.
### Impact
The page leads with the embedded API ("This is what the jar is for") and its example syncs to
`DELTA`. A reader who copies that example onto Spark 3.5 gets no Kernel auto-selection, falls
through to `DeltaConversionSourceProvider`, and hits the `delta-core` path that the same page says
does not run on 3.5. The one sentence that does mention `.useDeltaKernel(true)` frames it as a
Spark 3.4 opt-in, which points the reader away from the case where it is actually required.
### Suggested fix
Separate the two entry points in the "Spark version support" section: `XTableSparkSync`
auto-enables Kernel when Delta is involved and Spark is 3.5 or later, while `XTableSyncService`
requires `.useDeltaKernel(true)` explicitly on Spark 3.5.
Split out of #920 so that issue stays scoped to the review comments on #913.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Docs: Delta Kernel is not auto-selected for in-job syncs, only for the XTableSparkSync CLI [incubator-xtable]
Posted by "salimTutuncu (via GitHub)" <gi...@apache.org>.
salimTutuncu commented on issue #921:
URL: https://github.com/apache/incubator-xtable/issues/921#issuecomment-5541368644
Related gap on the same docs page, found while validating the published 0.4.0 artifact: the "reuses the Hudi, Iceberg and Delta libraries your cluster already has" statement needs version qualifiers, because the bundle is only binary-compatible with specific engine versions. Exact measured results (local, upstream OSS binaries):
| Engine dependency | Works | Fails (NoSuchMethodError) |
|---|---|---|
| Hudi source reader | 0.14.1 | 0.15.0, 1.0.2 (`HoodieTableMetaClient$Builder.setConf(Configuration)`) |
| Delta Kernel | 4.0.0 | 3.3.0 (`Snapshot.getVersion()`) |
| Iceberg target | 1.9.2 | not tested beyond 1.9.2 |
We have not tested vendor-patched builds or any versions beyond that table, so it is deliberately narrow. For managed-runtime relevance (per vendor documentation, not our tests): AWS Glue 5.1 lists Hudi 1.0.2 and Delta 3.3.2 (https://docs.aws.amazon.com/glue/latest/dg/release-notes.html), while Amazon EMR 7.1 lists Hudi 0.14.1 (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-710-release.html), so depending on the platform release, users either hit the mismatch as a runtime `NoSuchMethodError` or are unaffected.
Suggestion: alongside the Kernel auto-select fix, add a short engine-compatibility note for the 0.4.0 artifact to this page (tested-works: Hudi 0.14.1, Delta Kernel 4.0.0, Iceberg 1.9.2; tested-incompatible: Hudi 0.15.0 and 1.0.2, Kernel 3.3.0), so the "reuses your cluster's libraries" sentence carries its real preconditions. Note this compatibility warning concerns the 0.4.0 artifact; current `main` compiles against Hudi 1.2.0 (pom, checked 2026-09-04).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Docs: Delta Kernel is not auto-selected for in-job syncs, only for the XTableSparkSync CLI [incubator-xtable]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #921:
URL: https://github.com/apache/incubator-xtable/issues/921#issuecomment-5421600207
A second inaccuracy in the same table, found while verifying the first.
`website/docs/how-to-spark-runtime.md` labels the Spark 3.4 row "Delta Standalone":
> | 3.4.x | ✅ | Delta Standalone |
The bundle does not depend on Delta Standalone. On the `0.4.0-incubating` tag,
`xtable-spark-runtime/pom.xml` declares exactly three `io.delta` dependencies, all `provided`:
* `delta-core_${scala.binary.version}`
* `delta-kernel-api`
* `delta-kernel-defaults`
`grep -c delta-standalone xtable-spark-runtime/pom.xml` returns `0`. The non-Kernel path is
`DeltaConversionSourceProvider`, which is the `delta-core` reader; `XTableSyncService` notes this
inline as "The default delta-core reader is Spark-3.4-only".
Delta Standalone (`delta.standalone.version` `3.3.0`) is declared in the root `pom.xml`
`dependencyManagement`, so it is used elsewhere in the project, but it is not what this bundle
runs on Spark 3.4.
So the row should say `delta-core`, not Delta Standalone. Worth fixing together with the Kernel
auto-selection wording above, since both are in the same table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org