You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2026/08/12 15:15:30 UTC
(kyuubi) branch branch-1.12 updated: [KYUUBI #6403][KSHC] Fix write into partitioned table with non-last partition column
This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch branch-1.12
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.12 by this push:
new 211a37584a [KYUUBI #6403][KSHC] Fix write into partitioned table with non-last partition column
211a37584a is described below
commit 211a37584ab578680ce7d0c084cb4eaefaf7dc9f
Author: Kaifei Yi <yi...@apache.org>
AuthorDate: Wed Aug 12 23:15:07 2026 +0800
[KYUUBI #6403][KSHC] Fix write into partitioned table with non-last partition column
### Why are the changes needed?
Fixes [#6403](https://github.com/apache/kyuubi/issues/6403).
Writing a partitioned Hive table via the Kyuubi Spark Hive connector fails when the partition column is **not the last column** of the schema, e.g. `df.write.partitionBy("mid_col").saveAsTable(...)`.
There are two root causes:
1. `HiveWrite` split data/partition columns **positionally** with `take`/`takeRight`, assuming partition columns are always trailing. When a partition column is in the middle, columns are mis-classified and the metastore rejects the written partition spec:
```
Table$ValidationFailureSemanticException: Partition spec {favorite_color=, favorite_numbers=2} contains non-partition columns
```
2. The Hive metastore reorders partition columns to the end of the schema, but the CTAS query output keeps the original column order. Spark's `ResolveOutputRelation` resolves output columns **by position**, so a non-last partition column is mis-cast onto a trailing data column (`INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_SAFELY_CAST`).
This fixes both:
- `HiveWrite` now splits data/partition columns **by name** against `table.partitionColumnNames` (also for the write `dataSchema` derivation).
- `HiveTableCatalog.createTable` preserves the requested schema order on the returned table, so the subsequent CTAS write resolves output columns against the original order.
Affects 1.8.1 and current `master`.
### How was this patch tested?
Added a regression test in `HiveQuerySuite` that writes a table whose partition column is in the middle of the schema, using **mixed column types** (`String`/`String`/`Int`) so the schema-order bug is genuinely exercised (an all-`String` case would not surface the cast failure).
Verified locally with `-Pspark-3.5`:
- `HiveQuerySuite`: 21 tests passed (incl. the new case)
- `HiveCatalogSuite`: 24 tests passed
- `dev/reformat` (Spotless + Scalastyle) passed
### Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Claude:claude-opus-4-8
Closes #7622 from yikf/kyuubi-6403-non-last-partition-column.
Closes #6403
08a5c53bc [Kaifei Yi] Fix write into partitioned table with non-last partition column
Authored-by: Kaifei Yi <yi...@apache.org>
Signed-off-by: Cheng Pan <ch...@apache.org>
(cherry picked from commit 4b8a225c00caf49f6d21ad8a8988fb362da6495b)
Signed-off-by: Cheng Pan <ch...@apache.org>
---
.../kyuubi/spark/connector/hive/HiveTable.scala | 9 +++-
.../spark/connector/hive/HiveTableCatalog.scala | 6 ++-
.../connector/hive/write/HiveBatchWrite.scala | 16 +++++--
.../spark/connector/hive/write/HiveWrite.scala | 29 +++++++++--
.../spark/connector/hive/HiveQuerySuite.scala | 56 ++++++++++++++++++++++
5 files changed, 105 insertions(+), 11 deletions(-)
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTable.scala b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTable.scala
index 06b0bc067e..4c2d016858 100644
--- a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTable.scala
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTable.scala
@@ -46,7 +46,8 @@ import org.apache.kyuubi.spark.connector.hive.write.HiveWriteBuilder
case class HiveTable(
sparkSession: SparkSession,
catalogTable: CatalogTable,
- hiveTableCatalog: HiveTableCatalog)
+ hiveTableCatalog: HiveTableCatalog,
+ requestedSchema: Option[StructType] = None)
extends Table with SupportsRead with SupportsWrite with Logging {
lazy val dataSchema: StructType = catalogTable.dataSchema
@@ -86,7 +87,11 @@ case class HiveTable(
}
}
- override def schema(): StructType = catalogTable.schema
+ // The metastore reorders partition columns to the end, but Spark's ResolveOutputRelation
+ // resolves output columns by position, so a CTAS with a non-last partition column mis-casts
+ // (CANNOT_SAFELY_CAST). Expose the requested order only via the V2 Table.schema() that feeds
+ // output resolution, keeping catalogTable (partition columns trailing) untouched. KYUUBI #6403.
+ override def schema(): StructType = requestedSchema.getOrElse(catalogTable.schema)
override def properties(): util.Map[String, String] = catalogTable.properties.asJava
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala
index 2cb6c79f3e..c191ee7c6f 100644
--- a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala
@@ -398,7 +398,11 @@ class HiveTableCatalog(sparkSession: SparkSession)
throw new TableAlreadyExistsException(ident)
}
- loadTable(ident)
+ // Preserve the requested schema order so the subsequent CTAS write resolves output columns
+ // against the original order. Override only the V2 Table.schema() rather than rebuilding the
+ // CatalogTable, which would break the trailing-partition invariant and drop fields
+ // newCatalogTable does not forward. See KYUUBI #6403.
+ loadTable(ident).asInstanceOf[HiveTable].copy(requestedSchema = Some(schema))
}
override def alterTable(ident: Identifier, changes: TableChange*): Table = {
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/write/HiveBatchWrite.scala b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/write/HiveBatchWrite.scala
index 2a30ac434c..73dcddac4f 100644
--- a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/write/HiveBatchWrite.scala
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/write/HiveBatchWrite.scala
@@ -17,6 +17,8 @@
package org.apache.kyuubi.spark.connector.hive.write
+import java.util.Locale
+
import scala.util.control.NonFatal
import org.apache.hadoop.conf.Configuration
@@ -152,17 +154,23 @@ class HiveBatchWrite(
val caseInsensitiveDpMap = CaseInsensitiveMap(dpMap)
+ // Lower-case the partition column names when reconstructing the path: the written
+ // directory is lower-cased while the restored table.partitionColumnNames may keep the
+ // original case, which would miss the directory on a case-sensitive FS. This alignment is
+ // not observable with bundled Hive 2.3.10 (moveFile(replace=true) already deletes the
+ // destination), so it is not covered by a test. See KYUUBI #6403.
val updatedPartitionSpec = dynamicPartition.map {
- case (key, Some(null)) => key -> ExternalCatalogUtils.DEFAULT_PARTITION_NAME
- case (key, Some(value)) => key -> value
+ case (key, Some(null)) =>
+ key.toLowerCase(Locale.ROOT) -> ExternalCatalogUtils.DEFAULT_PARTITION_NAME
+ case (key, Some(value)) => key.toLowerCase(Locale.ROOT) -> value
case (key, None) if caseInsensitiveDpMap.contains(key) =>
- key -> caseInsensitiveDpMap(key)
+ key.toLowerCase(Locale.ROOT) -> caseInsensitiveDpMap(key)
case (key, _) =>
throw KyuubiHiveConnectorException(
s"Dynamic partition key ${toSQLValue(key, StringType)} " +
"is not among written partition paths.")
}
- val partitionColumnNames = table.partitionColumnNames
+ val partitionColumnNames = table.partitionColumnNames.map(_.toLowerCase(Locale.ROOT))
val tablePath = new Path(table.location)
val partitionPath = ExternalCatalogUtils.generatePartitionPath(
updatedPartitionSpec,
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/write/HiveWrite.scala b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/write/HiveWrite.scala
index cb66221095..842f6bac8b 100644
--- a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/write/HiveWrite.scala
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/write/HiveWrite.scala
@@ -32,6 +32,7 @@ import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.catalog.CatalogTable
import org.apache.spark.sql.catalyst.catalog.CatalogTypes.TablePartitionSpec
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
import org.apache.spark.sql.connector.distributions.{Distribution, Distributions}
import org.apache.spark.sql.connector.expressions.{Expressions, SortDirection, SortOrder}
import org.apache.spark.sql.connector.write.{BatchWrite, LogicalWriteInfo, RequiresDistributionAndOrdering}
@@ -66,8 +67,27 @@ case class HiveWrite(
private val tableLocation = hiveTable.getDataLocation
private val allColumns = info.schema().toAttributes
- private val dataColumns = allColumns.take(allColumns.length - hiveTable.getPartCols.size())
- private val partColumns = allColumns.takeRight(hiveTable.getPartCols.size())
+ // Split data and partition columns by name rather than by position. Users can create a
+ // partitioned table whose partition columns are not the trailing columns of the schema
+ // (e.g. df.write.partitionBy("mid_col")), so a positional take/takeRight would mis-classify
+ // columns and lead the metastore to reject the written partition spec. Match names with the
+ // session resolver because the Hive metastore lower-cases partition column names while the
+ // write schema preserves the user's original case. See KYUUBI #6403.
+ private val resolver = sparkSession.sessionState.conf.resolver
+ private def isPartitionColumn(name: String): Boolean =
+ table.partitionColumnNames.exists(resolver(_, name))
+ // partColumns keeps the query's original column case so requiredOrdering() resolves against the
+ // query.
+ private val (partColumns, dataColumns) =
+ allColumns.partition(col => isPartitionColumn(col.name))
+
+ // SPARK-28054: the Hive metastore keeps partition columns with lower-cased names, so the written
+ // partition directories must use lower-cased names; otherwise loadDynamicPartitions rejects the
+ // spec with "contains non-partition columns" when the schema preserves the user's original case.
+ // Rename via withName to keep the same exprId, so the value projection against allColumns is
+ // unaffected. See KYUUBI #6403.
+ private val outputPartColumns: Seq[AttributeReference] =
+ partColumns.map(col => col.withName(col.name.toLowerCase(Locale.ROOT)))
lazy val tableDesc: TableDesc = new TableDesc(
hiveTable.getInputFormatClass,
@@ -131,7 +151,8 @@ case class HiveWrite(
customPartitionLocations: Map[TablePartitionSpec, String],
options: Map[String, String]): WriteJobDescription = {
val hiveFileFormat = getHiveFileFormat(fileSinkConf)
- val dataSchema = StructType(info.schema().fields.take(dataColumns.length))
+ val dataSchema = StructType(
+ info.schema().filterNot(field => isPartitionColumn(field.name)))
val outputWriterFactory = hiveFileFormat.prepareWrite(sparkSession, job, options, dataSchema)
val metrics: Map[String, SQLMetric] = BasicWriteJobStatsTracker.metrics
val serializableHadoopConf = new SerializableConfiguration(hadoopConf)
@@ -143,7 +164,7 @@ case class HiveWrite(
outputWriterFactory = outputWriterFactory,
allColumns = allColumns,
dataColumns = dataColumns,
- partitionColumns = partColumns,
+ partitionColumns = outputPartColumns,
bucketSpec = None,
path = pathName,
customPartitionLocations = customPartitionLocations,
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/HiveQuerySuite.scala b/extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/HiveQuerySuite.scala
index 33c69e7f91..44483c8900 100644
--- a/extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/HiveQuerySuite.scala
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/HiveQuerySuite.scala
@@ -18,6 +18,7 @@
package org.apache.kyuubi.spark.connector.hive
import org.apache.spark.sql.{AnalysisException, Row, SparkSession}
+import org.apache.spark.sql.types.{DataTypes, StructField, StructType}
class HiveQuerySuite extends KyuubiHiveTest {
@@ -309,6 +310,61 @@ class HiveQuerySuite extends KyuubiHiveTest {
}
}
+ test("[KYUUBI #6403] Write into partitioned table with non-last partition column") {
+ val table = "hive.default.kyuubi_6403"
+ try {
+ spark.sql(s"DROP TABLE IF EXISTS $table")
+ val schema = StructType(Array(
+ StructField("name", DataTypes.StringType, nullable = false),
+ StructField("favorite_color", DataTypes.StringType, nullable = false),
+ StructField("favorite_number", DataTypes.IntegerType, nullable = false)))
+
+ val data = Seq(
+ Row("Alyssa", "blue", 1),
+ Row("Ben", "red", 2))
+
+ val usersDF = spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
+ // "favorite_color" is not the last column of the schema
+ usersDF.write.partitionBy("favorite_color").saveAsTable(table)
+
+ // Partition columns are moved to the end of the schema when reading back.
+ checkQueryResult(
+ s"select * from $table order by name",
+ spark,
+ Array(Row.apply("Alyssa", 1, "blue"), Row.apply("Ben", 2, "red")))
+ } finally {
+ spark.sql(s"DROP TABLE IF EXISTS $table")
+ }
+ }
+
+ test("[KYUUBI #6403] Write into partitioned table with upper-case partition column name") {
+ val table = "hive.default.kyuubi_6403_upper"
+ try {
+ spark.sql(s"DROP TABLE IF EXISTS $table")
+ val schema = StructType(Array(
+ StructField("name", DataTypes.StringType, nullable = false),
+ StructField("FavoriteColor", DataTypes.StringType, nullable = false),
+ StructField("favorite_number", DataTypes.IntegerType, nullable = false)))
+
+ val data = Seq(
+ Row("Alyssa", "blue", 1),
+ Row("Ben", "red", 2))
+
+ val usersDF = spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
+ // The Hive metastore lower-cases partition column names while the write schema keeps the
+ // original case, so the data/partition split must match names case-insensitively.
+ usersDF.write.partitionBy("FavoriteColor").saveAsTable(table)
+
+ // Partition columns are moved to the end of the schema when reading back.
+ checkQueryResult(
+ s"select * from $table order by name",
+ spark,
+ Array(Row.apply("Alyssa", 1, "blue"), Row.apply("Ben", 2, "red")))
+ } finally {
+ spark.sql(s"DROP TABLE IF EXISTS $table")
+ }
+ }
+
test("ORC filter pushdown") {
val table = "hive.default.orc_filter_pushdown"
dropTableAfter(table) {