You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/10/23 00:46:57 UTC
[PR] [SPARK-60493][SQL] Disallow Hive incompatible partition column types in Hive serde tables [spark]
dongjoon-hyun opened a new pull request, #59734:
URL: https://github.com/apache/spark/pull/59734
### What changes were proposed in this pull request?
This PR aims to disallow Hive incompatible partition column types in Hive serde tables with a clear error, `UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE`. The affected types are `TIMESTAMP_NTZ`, ANSI intervals, the nanosecond timestamp types (`TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)`), and `TIME`.
Since SPARK-51418 (#50182), `HiveClientImpl.toHiveTable` stores Hive incompatible partition column types with a placeholder type, `<derived from deserializer>`, in Hive metastore. This works for data source tables because Spark restores their schema from the table properties. However, it applies to Hive serde tables too. `HiveClientImpl` drops the placeholder columns when it reads a table, so `HiveExternalCatalog.restoreHiveSerdeTable` sees a schema mismatch with the table properties and falls back to the Hive metastore schema, which has no partition column.
This PR adds a check to the Hive serde table branch of `HiveExternalCatalog.createTable`. It uses `isHiveCompatibleDataType`, the same condition as the placeholder. Since all table creation paths go through it, the check covers the following:
- `CREATE TABLE ... STORED AS` and `CREATE TABLE ... USING HIVE`
- `CREATE TABLE ... AS SELECT`
- `CREATE TABLE ... LIKE` with `STORED AS` or `USING HIVE`
- `DataFrameWriter.saveAsTable` and `DataFrameWriterV2.create` with the `hive` provider
Data source tables are not affected.
### Why are the changes needed?
Since Apache Spark 4.0.0, the following statement succeeds, but it creates a broken table without the partition column `p`.
```sql
CREATE TABLE t (i INT) PARTITIONED BY (p TIMESTAMP_NTZ) STORED AS PARQUET;
```
```
spark-sql> INSERT INTO t PARTITION (p = TIMESTAMP_NTZ'2026-01-01 00:00:00') VALUES (1);
[PARTITIONS_NOT_FOUND] The partition(s) `p` cannot be found in table `spark_catalog`.`default`.`t`.
...
spark-sql> SHOW PARTITIONS t;
[INVALID_PARTITION_OPERATION.PARTITION_SCHEMA_IS_EMPTY] The partition command is invalid. Table `spark_catalog`.`default`.`t` is not partitioned. SQLSTATE: 42601
```
`CREATE TABLE AS SELECT` creates the table without the partition column, so the values of `p` are silently lost.
```sql
CREATE TABLE t PARTITIONED BY (p) STORED AS PARQUET
AS SELECT 1 AS i, TIMESTAMP_NTZ'2026-01-01 00:00:00' AS p;
SELECT * FROM t; -- returns only `i`
```
With `spark.sql.timestampType=TIMESTAMP_NTZ`, `PARTITIONED BY (p TIMESTAMP)` has the same issue. `TIME` partition columns have it too since SPARK-60482 (#59720), which is not released yet. Moreover, Hive cannot read these tables (`Unknown type <derived from deserializer>`), and `ALTER TABLE ADD COLUMNS` corrupts the table metadata so that even `DROP TABLE` fails.
```
The metadata is corrupted. Unable to find the partition column names from the schema. schema: struct<i:int,j:int>. Partition columns: [p]
```
Before SPARK-51418, Hive metastore rejected these tables at the table creation, e.g., `InvalidObjectException(message:Invalid partition column type: timestamp_ntz)`.
### Does this PR introduce _any_ user-facing change?
Yes. Creating a Hive serde table with these partition column types fails with the following error instead of creating a broken table. Data source tables are not affected.
```
[UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The Hive datasource doesn't support the column `p` of the type "TIMESTAMP_NTZ". SQLSTATE: 0A000
```
Previously, some CTAS statements with ANSI interval partition columns, e.g., `STORED AS PARQUET`, failed with `UNSUPPORTED_FEATURE.HIVE_WITH_ANSI_INTERVALS` after creating and dropping the table. Now, they fail with the above error before creating the table.
This PR doesn't change the existing tables.
### How was this patch tested?
Pass the CIs with the newly added test case in `HiveDDLSuite`. The test case fails without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60493][SQL] Disallow Hive incompatible partition column types in Hive serde tables [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #59734:
URL: https://github.com/apache/spark/pull/59734#discussion_r4360115263
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala:
##########
@@ -270,6 +271,14 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
hiveTable.withNewStorage(locationUri = tableLocation),
ignoreIfExists)
} else {
+ // Hive incompatible partition column types are stored with a placeholder type in Hive
+ // metastore (see `HiveClientImpl.toHiveTable`), which only data source tables can restore.
+ schemaWithNoCollation
+ .filter(f => tableDefinition.partitionColumnNames.contains(f.name))
+ .find(f => !isHiveCompatibleDataType(f.dataType))
+ .foreach { f =>
+ throw QueryCompilationErrors.dataTypeUnsupportedByDataSourceError("Hive", f)
Review Comment:
Thank you. You're right. In 3e200d3774c, I moved the check into `HiveExternalCatalog.verifyHiveSerdePartitionColumnTypes`, which `HiveExternalCatalog.createTable` still uses, and `CreateHiveTableAsSelectCommand` now runs it on the new table definition before it drops the existing table. The new table definition is computed once and passed to `createTable` too. I also added a test case which overwrites an existing table through `saveAsTable` and checks that the table and its rows are kept.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60493][SQL] Disallow Hive incompatible partition column types in Hive serde tables [spark]
Posted by "spark-test (via GitHub)" <gi...@apache.org>.
spark-test commented on code in PR #59734:
URL: https://github.com/apache/spark/pull/59734#discussion_r4355413135
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala:
##########
@@ -270,6 +271,14 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
hiveTable.withNewStorage(locationUri = tableLocation),
ignoreIfExists)
} else {
+ // Hive incompatible partition column types are stored with a placeholder type in Hive
+ // metastore (see `HiveClientImpl.toHiveTable`), which only data source tables can restore.
+ schemaWithNoCollation
+ .filter(f => tableDefinition.partitionColumnNames.contains(f.name))
+ .find(f => !isHiveCompatibleDataType(f.dataType))
+ .foreach { f =>
+ throw QueryCompilationErrors.dataTypeUnsupportedByDataSourceError("Hive", f)
Review Comment:
**Non-blocking (P2):** This check runs too late for `saveAsTable` overwrites. Take an existing table `t` and `df.write.format("hive").mode("overwrite").partitionBy("pt").saveAsTable("t")`, where `pt` is `TIMESTAMP_NTZ`, `TIME`, a nanosecond timestamp, or an ANSI interval. `SaveAsV1TableCommand` plans a `CreateHiveTableAsSelectCommand`, and the partition column passes analysis: `PreprocessTableCreation` only applies `canPartitionOn` to partition columns, and `DDLUtils.checkTableColumns` sees only the data columns. `CreateHiveTableAsSelectCommand.run` then drops `t` (line 72) and only afterwards calls `catalog.createTable` (line 81, outside the write's try/catch), which reaches this `throw`. The user gets `UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE`, but `t`, and for a managed table its data, is already gone.
That contradicts the SPARK-60387 overwrite contract in `SaveAsV1TableCommand` ("drops the table as late as it can, so that a write that fails its checks keeps the table"), which `MetastoreDataSourcesSuite` pins for the Hive format. This rejection depends only on the query schema and the partition column names, so it can run before the drop. The merge target also lost `t` on this input (it replaced it with a broken table), so this isn't a new silent data loss. Still, the new failure now lands after the destructive step instead of before it. Only the overwrite of an existing table is affected; the creation forms the new test covers all target tables that don't exist yet, and they are rejected before anything is created.
**Recommended change:** Move the Hive serde partition-type check into a shared validator on the HiveExternalCatalog companion object and keep calling it from HiveExternalCatalog.createTable. In CreateHiveTableAsSelectCommand.run, compute the new table's schema before the existing-table drop and run the same validator on that schema and the table's partition columns before dropping anything. Add Hive DDL test coverage that overwrites an existing table through saveAsTable with an incompatible partition column.
**Why this works:** The validator applies the same collation stripping, exact partition-name selection, and Hive compatibility predicate that decide whether toHiveTable writes a placeholder partition type. Because the CTAS command now runs it before catalog.dropTable, a rejected overwrite throws before any catalog or data mutation. This restores the saveAsTable overwrite guarantee that a write which fails its checks keeps the existing table. createTable continues to reject the same definitions for producers that perform no prior drop.
**Scope:** Share the Hive serde partition-type validation and run it before the Hive CTAS overwrite drops the existing table.
**Compatibility:** Hive serde tables with Hive compatible partition types, and data source tables with any of these partition types, keep being created and overwritten exactly as before.
**Risks:** Computing the CTAS table schema before the drop must yield exactly the schema later passed to catalog.createTable; reuse one computed value for both rather than deriving it twice.
**Constraints:** Keep the rejected type domain identical to the HiveClientImpl.toHiveTable placeholder condition, using the same collation stripping, exact partition-name match, and compatibility predicate, so no partition type is rejected on one path and accepted on another.
**Exclusions:**
- Existing tables already stored with placeholder partition column types stay unchanged, as the PR declares; this repair only moves the rejection earlier for new definitions.
**Success:** Overwriting an existing table through DataFrameWriter.format("hive").mode("overwrite").partitionBy(...).saveAsTable with a TIMESTAMP_NTZ, ANSI interval, nanosecond timestamp, or TIME partition column fails with UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE, and the existing table still exists with its original schema and rows. Every Hive serde creation form that the PR already rejects still fails with the same error and leaves no table behind. Data source tables with these partition types, and Hive serde tables with Hive compatible partition types, are still created and overwritten as before.
**Verification:**
- **Regression:** Overwriting an existing Hive table through saveAsTable with each Hive incompatible partition type fails with UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE and leaves the original table and its rows readable.
- **Behavior:** The existing creation coverage (STORED AS, USING HIVE, CTAS, saveAsTable into a new table, CREATE TABLE LIKE with STORED AS, and TIMESTAMP under the TIMESTAMP_NTZ session default) still reports the same error parameters with no table left behind.
- **Compatibility:** A Hive CTAS or saveAsTable overwrite with Hive compatible partition types still replaces the table, and data source tables with these partition types are still accepted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org