You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2026/10/16 06:19:45 UTC
[I] `HoodieCatalog.alterTable` throws `MatchError` for `DeleteColumn` / `UpdateColumnNullability` / `UpdateColumnPosition` and rejects idempotent `SetProperty` [hudi]
nsivabalan opened a new issue, #20371:
URL: https://github.com/apache/hudi/issues/20371
### Bug Description
`HoodieCatalog.alterTable(ident, changes*)` groups changes by class and handles only `AddColumn`, `UpdateColumnType` and `UpdateColumnComment`:
1. Any `ColumnChange` subtype other than those two hits an inner `match` with no default case and fails with `scala.MatchError` (`HoodieCatalog.scala:238`): `DeleteColumn`, `RenameColumn`, `UpdateColumnNullability`, `UpdateColumnPosition`, `UpdateColumnDefaultValue`. A `MatchError` gives the caller no indication that the operation is unsupported.
2. `SetProperty` / `RemoveProperty` throw `UnsupportedOperationException("not supported table change: …")` (`:255`) even when the property value is unchanged. Spark's own `V2SessionCatalog` accepts these; any DSv2 caller that syncs table properties through `alterTable` (Spark 4.1 Declarative Pipelines does this on **every** run for every table) fails on the first re-run.
Additionally, `HoodieInternalV2Table.schema()`/`columns()` includes the five `_hoodie_*` meta columns, so DSv2 callers that diff the user schema against `Table.columns()` will issue `DeleteColumn` for them and hit (1).
Expected:
- A clear `UnsupportedOperationException` (or proper support) for unhandled `ColumnChange`s instead of `MatchError`.
- `SetProperty`/`RemoveProperty` applied to the catalog table properties, idempotently; reject only changes to Hudi table-identity properties (`primaryKey`, `preCombineField`, `type`, key generator…) with an actionable message.
- A decision on meta columns in `Table.columns()` (hide them for DSv2 callers, or treat `DeleteColumn` of `_hoodie_*` as a no-op).
Spark 4.1 also calls `createTable(Identifier, TableInfo)` → default `createTable(Identifier, Column[], …)`, which `HoodieCatalog` does not override (only the `StructType` overload is), so `DelegatingCatalogExtension` forwards it to the session catalog and the table is registered with `provider=hudi` but **no `hoodie.properties`**. It can be split into its own issue if preferred.
### Environment
Hudi 1.2.1 (`hudi-spark4.1-bundle_2.13`, Maven Central), Spark 4.1.1 (Spark Connect server), `spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog`, `HoodieSparkSessionExtension`, local filesystem, macOS. Behaviour cross-checked against current `master` source.
### Steps to reproduce
```scala
import org.apache.spark.sql.connector.catalog.{Identifier, TableChange}
spark.sql("CREATE TABLE default.t (id STRING, v INT) USING hudi TBLPROPERTIES (primaryKey='id')")
val cat = spark.sessionState.catalogManager.catalog("spark_catalog").asInstanceOf[org.apache.spark.sql.connector.catalog.TableCatalog]
val id = Identifier.of(Array("default"), "t")
cat.alterTable(id, TableChange.setProperty("comment", "x")) // UnsupportedOperationException
cat.alterTable(id, TableChange.deleteColumn(Array("v"), false)) // scala.MatchError
cat.alterTable(id, TableChange.updateColumnNullability(Array("v"), true)) // scala.MatchError
```
### Logs and Stack Trace
```
java.lang.UnsupportedOperationException: not supported table change: org.apache.spark.sql.connector.catalog.TableChange$SetProperty
at org.apache.spark.sql.hudi.catalog.HoodieCatalog.$anonfun$alterTable$2(HoodieCatalog.scala:256)
scala.MatchError: org.apache.spark.sql.connector.catalog.TableChange$DeleteColumn@1cdc90c (of class org.apache.spark.sql.connector.catalog.TableChange$DeleteColumn)
at org.apache.spark.sql.hudi.catalog.HoodieCatalog.$anonfun$alterTable$4(HoodieCatalog.scala:238)
at org.apache.spark.sql.hudi.catalog.HoodieCatalog.alterTable(HoodieCatalog.scala:226)
```
Code pointer: `hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala` (`alterTable`, `createTable`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `HoodieCatalog.alterTable` throws `MatchError` for `DeleteColumn` / `UpdateColumnNullability` / `UpdateColumnPosition` and rejects idempotent `SetProperty` [hudi]
Posted by "anshulsingh-py (via GitHub)" <gi...@apache.org>.
anshulsingh-py commented on issue #20371:
URL: https://github.com/apache/hudi/issues/20371#issuecomment-6336778977
Following up now that I've seen #20401, which lists this issue as "catalog creation and idempotent reconciliation". I wanted to check the scope rather than assume.
My plan is to do the `alterTable` fixes first (the `MatchError` for unhandled `ColumnChange`s, and idempotent `SetProperty` / `RemoveProperty`), then the `createTable(Identifier, TableInfo)` overload, either in the same PR or as a follow-up, whichever you'd prefer.
@danny0405 @nsivabalan could either of you confirm that split works, and assign this to me if so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `HoodieCatalog.alterTable` throws `MatchError` for `DeleteColumn` / `UpdateColumnNullability` / `UpdateColumnPosition` and rejects idempotent `SetProperty` [hudi]
Posted by "anshulsingh-py (via GitHub)" <gi...@apache.org>.
anshulsingh-py commented on issue #20371:
URL: https://github.com/apache/hudi/issues/20371#issuecomment-6336581970
Hi @nsivabalan, I'd like to take this one. As an external contributor I can't self-assign, so could you assign it to me?
The scope I'm planning is the first two items in the description:
- Unsupported `ColumnChange` subtypes get a clear `UnsupportedOperationException` instead of `MatchError`, unless you'd rather have real support for any of them.
- `SetProperty` / `RemoveProperty` applied idempotently, rejecting only changes to table-identity properties (`primaryKey`, `preCombineField`, `type`, key generator, ...) with an actionable message.
Two things I'd like your steer on before touching them:
1. The `_hoodie_*` meta columns in `Table.columns()`: hide them, or make `DeleteColumn` on them a no-op?
2. The `createTable(Identifier, TableInfo)` gap - I'm happy to leave it for a separate issue, as the description suggests.
Since this is listed as a prerequisite in RFC-112, is anyone already covering it as part of #20378 or the RFC work? I don't want to duplicate that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org