You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prashantwason (via GitHub)" <gi...@apache.org> on 2026/04/30 05:15:48 UTC
[I] CatalogBackedTableMetadata throws NoSuchTableException for spark.read.format(hudi).load(path) on tables not in Spark catalog [hudi]
prashantwason opened a new issue, #18656:
URL: https://github.com/apache/hudi/issues/18656
## Bug Description
**What happened:**
`CatalogBackedTableMetadata` (added in #18265) calls
`sparkSession.sessionState.catalog.getTableMetadata(tableIdentifier)` unconditionally and lets `NoSuchTableException` propagate. This breaks `spark.read.format("hudi").load(path)` for any Hudi table that isn't registered in Spark's session catalog when `hoodie.datasource.read.file.index.list.partitions.from.catalog=true` (the new opt-in config from the same PR).
`spark.read.format("hudi").load(path)` is a documented Hudi API and must work without catalog registration. The new code path silently makes that API depend on catalog registration as soon as the config flag is on.
**What you expected:**
Two correct behaviors:
1. `spark.read.format("hudi").load(path)` works on any valid Hudi table directory, registered in a catalog or not.
2. When `list.partitions.from.catalog=true` AND the table isn't in the catalog, fall back to `FileSystemBackedTableMetadata` (the parent class) and emit a warning so operators see the gap. Don't throw.
**Steps to reproduce:**
1. Initialize a Hudi table on disk without registering it in any catalog:
```scala
HoodieTableMetaClient.newTableBuilder()
.setTableType(HoodieTableType.COPY_ON_WRITE)
.setDatabaseName("rawdatatmp")
.setTableName("hudi_partition_metadata_repro")
.setRecordKeyFields("uuid")
.setPartitionFields("datestr")
.setHiveStylePartitioningEnable(true)
.initTable(storageConf, basepath)
```
2. Write 3 rows in 2 partitions via `INSERT_OVERWRITE`.
3. Read via `spark.read.format("hudi").load(basepath)` with `--conf spark.hoodie.datasource.read.file.index.list.partitions.from.catalog=true`.
4. Throws:
```
NoSuchTableException: Table or view 'hudi_partition_metadata_repro' not found in database 'rawdatatmp'
at CatalogBackedTableMetadata.catalogTable$lzycompute(CatalogBackedTableMetadata.scala:55)
at CatalogBackedTableMetadata.isPartitionedTable(CatalogBackedTableMetadata.scala:58)
at CatalogBackedTableMetadata.getPartitionPathWithPathPrefixes(CatalogBackedTableMetadata.scala:82)
at BaseHoodieTableFileIndex.listPartitionPaths(BaseHoodieTableFileIndex.java:392)
...
```
Setting `hoodie.datasource.read.file.index.list.partitions.from.catalog=false` works as a workaround (forces the parent `FileSystemBackedTableMetadata` to be used directly).
**Root cause:**
`CatalogBackedTableMetadata.scala:55`:
```scala
private lazy val catalogTable = sparkSession.sessionState.catalog.getTableMetadata(tableIdentifier)
```
The `lazy val` doesn't handle the lookup failing. Each of the three override methods (`getAllPartitionPaths`, `getPartitionPathWithPathPrefixes`, `getPartitionPathWithPathPrefixUsingFilterExpression`) reads `catalogTable.partitionColumnNames` first, which forces the lazy evaluation and throws.
A fix is in PR #<PR_NUMBER>.
## Environment
**Hudi version:** master (regression introduced in #18265, present from there onward)
**Query engine:** Spark (any 3.x with Hive support)
**Relevant configs:** `hoodie.datasource.read.file.index.list.partitions.from.catalog=true`
## Logs and Stack Trace
```
Caused by: org.apache.spark.sql.catalyst.analysis.NoSuchTableException:
Table or view 'hudi_partition_metadata_repro' not found in database 'rawdatatmp'
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.requireTableExists(SessionCatalog.scala:226)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTableRawMetadata(SessionCatalog.scala:535)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTableMetadata(SessionCatalog.scala:521)
at org.apache.hudi.metadata.CatalogBackedTableMetadata.catalogTable$lzycompute(CatalogBackedTableMetadata.scala:55)
at org.apache.hudi.metadata.CatalogBackedTableMetadata.catalogTable(CatalogBackedTableMetadata.scala:55)
at org.apache.hudi.metadata.CatalogBackedTableMetadata.isPartitionedTable(CatalogBackedTableMetadata.scala:58)
at org.apache.hudi.metadata.CatalogBackedTableMetadata.getPartitionPathWithPathPrefixes(CatalogBackedTableMetadata.scala:82)
at org.apache.hudi.BaseHoodieTableFileIndex.listPartitionPaths(BaseHoodieTableFileIndex.java:392)
at org.apache.hudi.BaseHoodieTableFileIndex.getAllQueryPartitionPaths(BaseHoodieTableFileIndex.java:230)
at org.apache.hudi.SparkHoodieTableFileIndex.listMatchingPartitionPaths(SparkHoodieTableFileIndex.scala:249)
at org.apache.hudi.HoodieFileIndex.prunePartitionsAndGetFileSlices(HoodieFileIndex.scala:343)
at org.apache.hudi.HoodieFileIndex.filterFileSlices(HoodieFileIndex.scala:227)
... Spark catalyst optimizer frames ...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CatalogBackedTableMetadata throws NoSuchTableException for spark.read.format(hudi).load(path) on tables not in Spark catalog [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on issue #18656:
URL: https://github.com/apache/hudi/issues/18656#issuecomment-4349936870
Closing. After investigation the original failure mode was specific to HIVE_SYNC_MODE=HIVEQL (a separate HiveCLI session adds partitions, leaving Spark's cached Hive client stale on reads). HIVE_SYNC_MODE=HMS resolves the symptom because writes and reads share the catalog client. The narrower 'load(path) on a table not in any catalog throws NoSuchTableException' issue is real but lower priority; we'll re-evaluate before re-filing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CatalogBackedTableMetadata throws NoSuchTableException for spark.read.format(hudi).load(path) on tables not in Spark catalog [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason closed issue #18656: CatalogBackedTableMetadata throws NoSuchTableException for spark.read.format(hudi).load(path) on tables not in Spark catalog
URL: https://github.com/apache/hudi/issues/18656
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org