You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@xtable.apache.org by "salimTutuncu (via GitHub)" <gi...@apache.org> on 2026/09/15 12:16:33 UTC
[I] 0.4.x spark-runtime: Hudi source fails with NoSuchMethodError on Hudi 0.15+/1.x; #772's upgrade does not cover the released runtime line [incubator-xtable]
salimTutuncu opened a new issue, #941:
URL: https://github.com/apache/incubator-xtable/issues/941
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.
### Please describe the bug 🐞
The published `xtable-spark-runtime_2.12:0.4.0-incubating` artifact fails against every Hudi version in the 0.15+/1.x generation with a binary linkage error when reading a Hudi source:
```text
java.lang.NoSuchMethodError: 'org.apache.hudi.common.table.HoodieTableMetaClient$Builder
org.apache.hudi.common.table.HoodieTableMetaClient$Builder.setConf(org.apache.hadoop.conf.Configuration)'
```
The Spark Runtime documentation states the thin bundle "reuses the Hudi, Iceberg and Delta libraries your cluster already has" (https://xtable.apache.org/docs/how-to-spark-runtime). For Hudi this only holds for 0.14.x: per vendor documentation, current managed Spark runtimes bundle newer generations (e.g. AWS Glue 5.1 lists Hudi 1.0.2, Amazon EMR 7.9 lists Hudi 0.15.0-amzn-6), so on those platforms the Hudi source path of the published artifact is unusable as-is. Measured results with upstream OSS bundles: Hudi 0.14.1 works; 0.15.0 and 1.0.2 fail with the error above.
### Why this is not a duplicate of #762
#762 / PR #772 upgraded `main` to Hudi 1.x (current `main` pom pins 1.2.0). However, the spark-runtime module does not exist on `main`; it lives on `branch-0.4`, which was deliberately kept on the Hudi 0.x line for the 0.4.0 release (#843). So the merged upgrade fixes future releases cut from `main`, but no released or currently releasable spark-runtime artifact benefits from it. This issue is about the released 0.4.x line and its documented compatibility contract. (Context also posted as a comment on #921.)
### Root cause
The 0.4.0 bundle is compiled against Hudi 0.14.0 (https://github.com/apache/incubator-xtable/blob/e8f22867a6cb297f58b248150220fce71188d16a/pom.xml#L73). Its bytecode calls `HoodieTableMetaClient$Builder.setConf(org.apache.hadoop.conf.Configuration)` (https://github.com/apache/incubator-xtable/blob/e8f22867a6cb297f58b248150220fce71188d16a/xtable-core/src/main/java/org/apache/xtable/hudi/HudiConversionSourceProvider.java#L34-L50). Hudi 0.15.0 removed that overload in favor of `setConf(org.apache.hudi.storage.StorageConfiguration<?>)`; confirmed via `javap` against the 0.14.1, 0.15.0, and 1.0.2 bundles. Because the dependency is `provided`, the incompatibility surfaces only at runtime, on the user's cluster.
A secondary consequence for inline use on Spark 3.5: writing Hudi requires the 0.15+ bundle (the 0.14.x spark3.4 bundle cannot run on Spark 3.5), while syncing requires 0.14.x, so no single classpath supports the documented write-then-sync pattern for Hudi at all with the published artifact.
### To Reproduce
Environment: JDK 17, `pyspark==3.5.5`; jars from Maven Central: xtable-spark-runtime 0.4.0-incubating, iceberg-spark-runtime-3.5 1.9.2, hudi-spark3.5-bundle 0.15.0 (and optionally hudi-spark3.4-bundle 0.14.1 for the contrast run). Generator script included below.
```bash
# 1. write a partitioned Hudi COW table with Hudi 0.15.0 (the generation current runtimes ship)
spark-submit --jars hudi-spark3.5-bundle_2.12-0.15.0.jar \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
gen_hudi.py /tmp/orders_hudi
# 2. sync with the same Hudi on the classpath
spark-submit --master 'local[2]' --class org.apache.xtable.spark.XTableSparkSync \
--jars hudi-spark3.5-bundle_2.12-0.15.0.jar,iceberg-spark-runtime-3.5_2.12-1.9.2.jar \
xtable-spark-runtime_2.12-0.4.0-incubating.jar \
--basepath /tmp/orders_hudi --sourceformat HUDI --targets ICEBERG \
--partitionspec order_date:VALUE
```
Observed: the `NoSuchMethodError` above, nonzero exit. Same failure with hudi-spark3.5-bundle 1.0.2. Contrast: replacing the Hudi jar with `hudi-spark3.4-bundle_2.12-0.14.1.jar` makes the same sync succeed with a fully correct target (rows, partitions, stats verified).
### Expected behavior
One of, in order of preference: (a) a 0.4.x patch release of the spark-runtime compatible with the Hudi 0.15+/1.x generation, or built with the `main` Hudi upgrade; (b) failing that, the documented compatibility contract for 0.4.0 (Hudi 0.14.x only) stated on the Spark Runtime page, plus a fail-fast version check that reports "unsupported Hudi version X, 0.4.0 requires 0.14.x" instead of a raw `NoSuchMethodError`.
### Affected versions
`0.4.0-incubating` published artifact; `branch-0.4` HEAD carries the same 0.14 pin. Not applicable to `main` (module absent there; `main` pom is on Hudi 1.2.0 as of 2026-09-04).
#### Reproduction script
gen_hudi.py
```python
import sys
from pyspark.sql import SparkSession, functions as F
path = sys.argv[1]
spark = SparkSession.builder.getOrCreate()
df = (spark.range(0, 1000).withColumn("order_id", F.col("id"))
.withColumn("ts", F.lit(0).cast("long"))
.withColumn("order_date", F.lit("2026-08-01")).drop("id").repartition(2))
df.write.format("hudi").options(**{
"hoodie.table.name": "orders_hudi",
"hoodie.datasource.write.table.type": "COPY_ON_WRITE",
"hoodie.datasource.write.recordkey.field": "order_id",
"hoodie.datasource.write.precombine.field": "ts",
"hoodie.datasource.write.partitionpath.field": "order_date",
"hoodie.datasource.write.hive_style_partitioning": "true",
}).mode("overwrite").save(path)
print("wrote 1000 rows ->", path)
```
### Are you willing to submit PR?
- [ ] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 0.4.x spark-runtime: Hudi source fails with NoSuchMethodError on Hudi 0.15+/1.x; #772's upgrade does not cover the released runtime line [incubator-xtable]
Posted by "vinishjail97 (via GitHub) via commits" <co...@xtable.apache.org>.
vinishjail97 commented on issue #941:
URL: https://github.com/apache/incubator-xtable/issues/941#issuecomment-6534513354
Thanks again for the detailed report, @salimTutuncu. Here is where this stands for 0.5.0.
`xtable-spark-runtime` is now on `main` (#964) and is built against Hudi 1.2.1. On `main` the Hudi sync works with Hudi 1.2.x on the classpath, and it reads tables written by Hudi 0.15.0 and 1.0.2 without upgrading them.
On Glue 5.1, which includes Hudi 1.0.2, the sync job can use Hudi 1.2.1 as AWS documents under [Using a different Hudi version](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-hudi.html):
- pass `xtable-spark-runtime` and `hudi-spark3.5-bundle_2.12-1.2.1.jar` with `--extra-jars`
- set `--user-jars-first true`
- leave `hudi` out of `--datalake-formats`
The jobs that write the table can keep using Glue's built-in Hudi. We have validated these Hudi versions on Spark, but not on Glue itself, so feedback from a Glue run would be very welcome.
#995 adds this setup to the Spark runtime page. As you suggested earlier, it would be good to close this issue against 0.5.0 once that docs PR lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 0.4.x spark-runtime: Hudi source fails with NoSuchMethodError on Hudi 0.15+/1.x; #772's upgrade does not cover the released runtime line [incubator-xtable]
Posted by "salimTutuncu (via GitHub)" <gi...@apache.org>.
salimTutuncu commented on issue #941:
URL: https://github.com/apache/incubator-xtable/issues/941#issuecomment-5693259888
Thanks @vinishjail97, that's great news. #838 addresses exactly this, so from my side it's fine to track this issue as resolved-by-0.5.0; feel free to link/close it against the release when the runtime lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] 0.4.x spark-runtime: Hudi source fails with NoSuchMethodError on Hudi 0.15+/1.x; #772's upgrade does not cover the released runtime line [incubator-xtable]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on issue #941:
URL: https://github.com/apache/incubator-xtable/issues/941#issuecomment-5689365235
Hi @salimTutuncu, thanks for reporting the issue.
There's a draft PR to get spark-runtime published that works with hudi 1.x https://github.com/apache/incubator-xtable/pull/838 and will be part of the 0.5.0 release that's planned for 30th September.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org