[I] SparkCatalogMetaStoreClient.alter_table fails with 'spark.sql.* table property keys not allowed' on Spark-managed Hudi tables [hudi]

["prashantwason (via GitHub)" <gi...@apache.org>]

prashantwason opened a new issue, #18653:
URL: https://github.com/apache/hudi/issues/18653

   ## Bug Description
   
   **What happened:**
   
   `SparkCatalogMetaStoreClient.alter_table` (and `createTable`) fail when the incoming `Table` carries properties starting with `spark.sql.`. Spark's `HiveExternalCatalog.alterTable` / `createTable` rejects such keys with:
   
   ```
   AnalysisException: Cannot persist <db>.<table> into Hive metastore as table property keys
   may not start with 'spark.sql.': [spark.sql.create.version, spark.sql.sources.provider,
   spark.sql.sources.schema.partCol.0, spark.sql.sources.schema.numParts,
   spark.sql.sources.schema.numPartCols, spark.sql.sources.schema.part.0]
   ```
   
   This blocks `HoodieHiveSyncClient` whenever the synced table was originally created/touched by Spark (which is the common case for tables with provider=hudi).
   
   **What you expected:**
   
   Hive sync via `SparkCatalogMetaStoreClient` should succeed for any table the user already has in their Spark catalog. The `spark.sql.*` keys are written by Spark itself; round-tripping them through alter_table is a no-op semantically because Spark re-derives them from the `CatalogTable`.
   
   **Steps to reproduce:**
   
   1. Have a Hudi table registered in Spark's Hive metastore (created via `CREATE TABLE ... USING hudi` or any flow that goes through `HiveExternalCatalog`).
   2. Configure `hoodie.datasource.hive_sync.use_spark_catalog=true` and `HIVE_SYNC_ENABLED=true`.
   3. Write to the table; Hudi triggers `HoodieHiveSyncClient.syncSchema` → `client.alter_table`.
   4. `SparkCatalogMetaStoreClient.alter_table` calls `toCatalogTable` (which preserves all parameters including `spark.sql.*`), then `externalCatalog.alterTable` → `AnalysisException`.
   
   **Root cause:**
   
   `SparkCatalogMetaStoreClient.scala:300-323` (`toCatalogTable`) passes `table.getParameters` straight through:
   
   ```scala
   properties = Option(table.getParameters).map(_.asScala.toMap).getOrElse(Map.empty)
   ```
   
   Spark's `HiveExternalCatalog.verifyTableProperties` then rejects any key starting with `spark.sql.`. Since `getTable` reads them from HMS (where Spark writes them on createTable), they are present whenever the table was Spark-managed — i.e. essentially always for a Hudi+Spark workflow.
   
   A fix is in PR #<PR_NUMBER>.
   
   ## Environment
   
   **Hudi version:** master (verified on `apache/master` as of `8d348cc6b927`)
   **Query engine:** Spark (any 3.x)
   **Relevant configs:** `hoodie.datasource.hive_sync.use_spark_catalog=true` with `HIVE_SYNC_ENABLED=true`
   
   ## Logs and Stack Trace
   
   ```
   Caused by: org.apache.spark.sql.AnalysisException: Cannot persist
     rawdatatmp.hudi_trips_partitioned_empty_table into Hive metastore as table property keys
     may not start with 'spark.sql.': [spark.sql.create.version,
     spark.sql.sources.provider, spark.sql.sources.schema.partCol.0,
     spark.sql.sources.schema.numParts, spark.sql.sources.schema.numPartCols,
     spark.sql.sources.schema.part.0]
       at org.apache.spark.sql.hive.HiveExternalCatalog.verifyTableProperties(HiveExternalCatalog.scala:...)
       at org.apache.spark.sql.hive.HiveExternalCatalog.doAlterTable(HiveExternalCatalog.scala:...)
       at org.apache.spark.sql.catalyst.catalog.ExternalCatalog.alterTable(ExternalCatalog.scala:...)
       at org.apache.spark.sql.hive.SparkCatalogMetaStoreClient.alter_table(SparkCatalogMetaStoreClient.scala:69)
       at org.apache.hudi.hive.HoodieHiveSyncClient.updateTableProperties(HoodieHiveSyncClient.java:...)
       ... HoodieHiveSyncClient sync flow ...
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] SparkCatalogMetaStoreClient.alter_table fails with 'spark.sql.* table property keys not allowed' on Spark-managed Hudi tables [hudi]

["danny0405 (via GitHub)" <gi...@apache.org>]

danny0405 closed issue #18653: SparkCatalogMetaStoreClient.alter_table fails with 'spark.sql.* table property keys not allowed' on Spark-managed Hudi tables
URL: https://github.com/apache/hudi/issues/18653


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org