You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/10/09 03:14:38 UTC
[PR] fix(spark): key catalog lookups on the resolved catalog entry, not the table config [hudi]
yihua opened a new pull request, #20247:
URL: https://github.com/apache/hudi/pull/20247
### Describe the issue this Pull Request addresses
part of #16933, follows #20240
`hoodie.database.name` in `hoodie.properties` is a copy of an identity the catalog owns, and the Spark read path used it, or the session's current database when absent, as a key into the catalog for two things: the simple bucket index options and the catalog-backed partition listing. Both read the wrong entry, or none, once that copy is stale or the table is registered under another name. The catalog-backed listing also returned stale results for partitions meta sync had not yet recorded.
### Summary and Changelog
The catalog integration now passes the identity Spark resolved down to the read path as two internal options, `hoodie.datasource.read.catalog.table.database` and `hoodie.datasource.read.catalog.table.name`, set by `HoodieInternalV2Table.v1TableWrapper` (storage properties, which Spark forwards as options) and by each Spark version's V2-to-V1 fallback; `HoodieCatalogTable.catalogProperties` strips them so they never reach write configs or TBLPROPERTIES. The bucket index lookup in `HoodieFileIndex` keys on that identity and skips the merge when it is absent; `getDatabaseName` and the session fallback are gone. `CatalogBackedTableMetadata` keys on the same identity, fails closed with a clear error when the listing option is on and no identity is present, and uses the catalog's partitions only while the entry's `last_commit_time_sync` (and completion time, when recorded) match the latest completed commit, otherwise it warns and lists the file system. `upgrade_table` and `downgrade_tab
le` pass a database only for a qualified table name instead of stamping the session's current one. Unread copies of the database name in `HoodieCatalogTable` and `FileSystemBackedTableMetadata` are removed.
Tests: `TestHoodieFileIndex` (identity used, no-identity skip, failure path), `TestCatalogBackedTableMetadata` (fail closed, fresh entry used), `TestPartitionPushDownWhenListingPaths` (stale catalog falls back, which failed before with a dropped partition), `TestUpgradeOrDowngradeProcedure` (no session database in the write config). All fallbacks compile on Spark 3.3 through 4.2.
This is step one of #16933: the config stays, but nothing keys a catalog operation on it any more. Step two removes it from the table config.
### Impact
A read with the catalog partition listing option enabled but no catalog identity (a path read, or a session catalog without Hudi's catalog plugin) now fails with an explicit error instead of guessing a database. Reads through `HoodieCatalog` are unaffected. No other user-facing change.
### Risk Level
low. The identity options are internal and stripped from table properties; the fail-closed case only affects an opt-in option used without a catalog.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): key catalog lookups on the resolved catalog entry, not the table config [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20247:
URL: https://github.com/apache/hudi/pull/20247#issuecomment-6073701752
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20247?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 80.63%. Comparing base ([`4cfba4c`](https://app.codecov.io/gh/apache/hudi/commit/4cfba4ca4079c4abd09348f0826374db78c8ab35?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cd651f2`](https://app.codecov.io/gh/apache/hudi/commit/cd651f2bbdbe2736a145b58e3db0e5c9abcc0dce?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20247 +/- ##
=========================================
Coverage 80.62% 80.63%
+ Complexity 35057 35041 -16
=========================================
Files 2552 2552
Lines 143390 143359 -31
Branches 17450 17447 -3
=========================================
- Hits 115607 115591 -16
+ Misses 19862 19854 -8
+ Partials 7921 7914 -7
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.16% <ø> (-0.01%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.82% <ø> (+0.02%)` | :arrow_up: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.96% <ø> (-0.01%)` | :arrow_down: |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.08% <100.00%> (+0.02%)` | :arrow_up: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.14% <ø> (-0.04%)` | :arrow_down: |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.80% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `71.11% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.25% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.52% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.06% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20247/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20247/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20247/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.97% <ø> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20247?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...e/hudi/metadata/FileSystemBackedTableMetadata.java](https://app.codecov.io/gh/apache/hudi/pull/20247?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FFileSystemBackedTableMetadata.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvRmlsZVN5c3RlbUJhY2tlZFRhYmxlTWV0YWRhdGEuamF2YQ==) | `94.48% <ø> (-0.09%)` | :arrow_down: |
| [...main/scala/org/apache/hudi/DataSourceOptions.scala](https://app.codecov.io/gh/apache/hudi/pull/20247?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FDataSourceOptions.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9EYXRhU291cmNlT3B0aW9ucy5zY2FsYQ==) | `95.25% <ø> (ø)` | |
| [...c/main/scala/org/apache/hudi/HoodieFileIndex.scala](https://app.codecov.io/gh/apache/hudi/pull/20247?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FHoodieFileIndex.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9Ib29kaWVGaWxlSW5kZXguc2NhbGE=) | `82.39% <100.00%> (-0.37%)` | :arrow_down: |
| [...la/org/apache/hudi/SparkHoodieTableFileIndex.scala](https://app.codecov.io/gh/apache/hudi/pull/20247?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FSparkHoodieTableFileIndex.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9TcGFya0hvb2RpZVRhYmxlRmlsZUluZGV4LnNjYWxh) | `74.22% <ø> (-0.27%)` | :arrow_down: |
| [...che/hudi/metadata/CatalogBackedTableMetadata.scala](https://app.codecov.io/gh/apache/hudi/pull/20247?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2Fmetadata%2FCatalogBackedTableMetadata.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9tZXRhZGF0YS9DYXRhbG9nQmFja2VkVGFibGVNZXRhZGF0YS5zY2FsYQ==) | `54.83% <ø> (+7.11%)` | :arrow_up: |
| [...park/sql/catalyst/catalog/HoodieCatalogTable.scala](https://app.codecov.io/gh/apache/hudi/pull/20247?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fcatalyst%2Fcatalog%2FHoodieCatalogTable.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2NhdGFseXN0L2NhdGFsb2cvSG9vZGllQ2F0YWxvZ1RhYmxlLnNjYWxh) | `79.14% <ø> (+0.30%)` | :arrow_up: |
| [...spark/sql/hudi/catalog/HoodieInternalV2Table.scala](https://app.codecov.io/gh/apache/hudi/pull/20247?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fhudi%2Fcatalog%2FHoodieInternalV2Table.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2h1ZGkvY2F0YWxvZy9Ib29kaWVJbnRlcm5hbFYyVGFibGUuc2NhbGE=) | `31.81% <ø> (-1.52%)` | :arrow_down: |
| [...mmand/procedures/UpgradeOrDowngradeProcedure.scala](https://app.codecov.io/gh/apache/hudi/pull/20247?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fhudi%2Fcommand%2Fprocedures%2FUpgradeOrDowngradeProcedure.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9zcGFyay9zcWwvaHVkaS9jb21tYW5kL3Byb2NlZHVyZXMvVXBncmFkZU9yRG93bmdyYWRlUHJvY2VkdXJlLnNjYWxh) | `86.95% <ø> (-0.55%)` | :arrow_down: |
| [...park/sql/hudi/analysis/HoodieSpark35Analysis.scala](https://app.codecov.io/gh/apache/hudi/pull/20247?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark3.5.x%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fhudi%2Fanalysis%2FHoodieSpark35Analysis.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmszLjUueC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL3NwYXJrL3NxbC9odWRpL2FuYWx5c2lzL0hvb2RpZVNwYXJrMzVBbmFseXNpcy5zY2FsYQ==) | `3.94% <ø> (+0.05%)` | :arrow_up: |
| [...park/sql/hudi/analysis/HoodieSpark42Analysis.scala](https://app.codecov.io/gh/apache/hudi/pull/20247?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark4.2.x%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fhudi%2Fanalysis%2FHoodieSpark42Analysis.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcms0LjIueC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL3NwYXJrL3NxbC9odWRpL2FuYWx5c2lzL0hvb2RpZVNwYXJrNDJBbmFseXNpcy5zY2FsYQ==) | `45.71% <ø> (-1.51%)` | :arrow_down: |
... and [10 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20247/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org