You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2026/08/24 14:08:19 UTC
(kyuubi) branch master updated: [KYUUBI #6943][2/2] OrcScan and ParquetScan support DPP
This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 0e26b7030a [KYUUBI #6943][2/2] OrcScan and ParquetScan support DPP
0e26b7030a is described below
commit 0e26b7030ad75824f80e95e05da1129fd50f361d
Author: maomaodev <li...@tencent.com>
AuthorDate: Mon Aug 24 22:08:08 2026 +0800
[KYUUBI #6943][2/2] OrcScan and ParquetScan support DPP
### Why are the changes needed?
Part 2 of 2 to add KSHC support for dynamic partition pruning (DPP). See https://github.com/apache/kyuubi/issues/6943.
- [x] Add DPP support in `HiveScan` for non-Parquet/ORC tables.
- [x] Add DPP support in `ParquetScan` / `ORCScan` for Parquet/ORC tables.
### How was this patch tested?
#### 1. UT & TPC-DS benchmark
- Unit tests
- Manual test: TPC-DS benchmark (10 GB dataset, ORC and Parquet separately). Spark configuration used for the benchmark(Spark 3.5.7, Kyuubi 1.12.0-SNAPSHOT):
```
spark.driver.cores 1
spark.driver.memory 4g
spark.executor.cores 1
spark.executor.instances 10
spark.executor.memory 4g
spark.master yarn
spark.shuffle.service.enabled true
spark.yarn.appMasterEnv.JAVA_HOME /usr/local/jdk-17
spark.executorEnv.JAVA_HOME /usr/local/jdk-17
```
#### 2. ORC benchmark
- **Overall performance (sum of 99)**
| Dimension | Vanilla Spark | KSHC Before | KSHC Now |
| ----------------- | ------------: | ----------: | ----------: |
| Total time | 2481.85 s | 3353.43 s | 2197.26 s |
| vs. Vanilla Spark | — | +35.12% | −11.47% |
| vs. KSHC Before | — | — | **−34.48%** |
- **DPP hit subset (73/99)**
DPP trigger was detected by matching `runtime partition filter` in the driver logs.
```
3,4,5,6,7,8,10,11,12,13,14,15,17,18,19,20,23,25,26,27,29,30,31,32,33,
35,36,38,40,42,43,45,46,47,48,49,50,51,52,53,54,55,56,57,58,60,61,63,
64,65,66,67,68,69,70,71,72,74,75,77,78,79,80,81,83,85,86,87,89,91,92,
97,98
```
| Dimension | Vanilla Spark | KSHC Before | KSHC Now |
| ----------------- | ------------: | ----------: | ----------: |
| Subset total time | 1823.48 s | 2642.11 s | 1484.39 s |
| vs. Vanilla Spark | — | +44.89% | −18.60% |
| vs. KSHC Before | — | — | **−43.82%** |
On the DPP-hit subset, KSHC Now provides a 43.82% speedup over KSHC Before, noticeably larger than the overall 34.48%, indicating the performance benefit mainly comes from queries where DPP is triggered.
#### 3. Parquet benchmark
- **Overall performance (sum of 99)**
| Dimension | Vanilla Spark | KSHC Before | KSHC Now |
| ----------------- | ------------: | ----------: | ----------: |
| Total time | 2325.13 s | 3363.57 s | 2152.18 s |
| vs. Vanilla Spark | — | +44.66% | −7.44% |
| vs. KSHC Before | — | — | **−36.02%** |
- **DPP hit subset (73/99)**
DPP trigger was detected by matching `runtime partition filter` in the driver logs.
```
3,4,5,6,7,8,10,11,12,13,14,15,17,18,19,20,23,25,26,27,29,30,31,32,33,
35,36,38,40,42,43,45,46,47,48,49,50,51,52,53,54,55,56,57,58,60,61,63,
64,65,66,67,68,69,70,71,72,74,75,77,78,79,80,81,83,85,86,87,89,91,92,
97,98
```
| Dimension | Vanilla Spark | KSHC Before | KSHC Now |
| ----------------- | ------------: | ----------: | ----------: |
| Subset total time | 1619.13 s | 2487.20 s | 1369.55 s |
| vs. Vanilla Spark | — | +53.61% | −15.41% |
| vs. KSHC Before | — | — | **−44.94%** |
On the DPP-hit subset, KSHC Now provides a 44.94% speedup over KSHC Before, noticeably larger than the overall 36.02%, indicating the performance benefit mainly comes from queries where DPP is triggered.
#### 4. Result correctness
Compared each of the 99 result files between KSHC Now and Vanilla Spark for both ORC and Parquet. ORC: 94/99 byte-identical and 98/99 row-multiset-identical; Parquet: identical figures. The 4 row-order-only diffs (q31/q65/q71/q79) come from queries whose `ORDER BY` clause does not totally order the output. The single multiset diff (q39) is sub-ULP floating-point rounding in `stddev`-style aggregates and is also present between KSHC Before and Vanilla Spark, so it is unrelated to this [...]
#### 5. Spark 4.0.1 benchmark
The same TPC-DS benchmark was also run against Spark 4.0.1 with KSHC. Results align with the Spark 3.5.7 numbers: KSHC matches or outperforms the native Hive path on DPP-eligible queries, and produces identical result sets. Full Spark 4.0.1 benchmark result are omitted here to keep the report compact, they can be shared on request.
#### **6. Known tradeoff: native engine offload**
`KyuubiParquetScan` / `KyuubiOrcScan` wrap Spark's built-in `ParquetScan` / `OrcScan` rather than subclass them, so DPP can be layered on without touching Spark internals. Native engines like Gluten and Comet identify the file format by scan class name, so KSHC-converted Parquet/ORC tables silently fall back to the JVM read path. Before this PR, KSHC-converted tables went through vanilla `ParquetScan` / `OrcScan` and were offloadable; after this PR they are not.
### Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Claude Opus 4.7
Closes #7476 from maomaodev/kyuubi_6943.
Closes #6943
b0005139f [lifumao] Fix UT
21242e24d [lifumao] add comments
7cc6b0508 [lifumao] Cache the Ctor in a lazy val
007194d04 [lifumao] [KYUUBI #6943][2/2] OrcScan and ParquetScan support DPP
Authored-by: maomaodev <li...@tencent.com>
Signed-off-by: Cheng Pan <ch...@apache.org>
---
.../kyuubi/spark/connector/hive/HiveTable.scala | 22 +-
.../spark/connector/hive/HiveTableCatalog.scala | 13 +-
.../spark/connector/hive/read/HiveScan.scala | 23 +-
.../spark/connector/hive/read/KyuubiOrcScan.scala | 171 ++++++++++++++
.../connector/hive/read/KyuubiOrcScanBuilder.scala | 126 +++++++++++
.../connector/hive/read/KyuubiParquetScan.scala | 249 +++++++++++++++++++++
.../hive/read/KyuubiParquetScanBuilder.scala | 127 +++++++++++
.../hive/kyuubi/connector/HiveBridgeHelper.scala | 39 ++++
.../hive/DynamicPartitionPruningSuite.scala | 189 +++++++++++++---
.../spark/connector/hive/HiveCatalogSuite.scala | 53 ++++-
10 files changed, 969 insertions(+), 43 deletions(-)
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTable.scala b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTable.scala
index 4c2d016858..8d43f8d6d2 100644
--- a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTable.scala
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTable.scala
@@ -33,14 +33,12 @@ import org.apache.spark.sql.connector.catalog.TableCapability.{BATCH_READ, BATCH
import org.apache.spark.sql.connector.expressions.Transform
import org.apache.spark.sql.connector.read.ScanBuilder
import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder}
-import org.apache.spark.sql.execution.datasources.v2.orc.OrcScanBuilder
-import org.apache.spark.sql.execution.datasources.v2.parquet.ParquetScanBuilder
import org.apache.spark.sql.hive.kyuubi.connector.HiveBridgeHelper.{BucketSpecHelper, LogicalExpressions}
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.util.CaseInsensitiveStringMap
import org.apache.kyuubi.spark.connector.hive.KyuubiHiveConnectorConf.{READ_CONVERT_METASTORE_ORC, READ_CONVERT_METASTORE_PARQUET}
-import org.apache.kyuubi.spark.connector.hive.read.{HiveCatalogFileIndex, HiveScanBuilder}
+import org.apache.kyuubi.spark.connector.hive.read.{HiveCatalogFileIndex, HiveScanBuilder, KyuubiOrcScanBuilder, KyuubiParquetScanBuilder}
import org.apache.kyuubi.spark.connector.hive.write.HiveWriteBuilder
case class HiveTable(
@@ -109,10 +107,24 @@ case class HiveTable(
override def newScanBuilder(options: CaseInsensitiveStringMap): ScanBuilder = {
convertedProvider match {
case Some("ORC") if sparkSession.sessionState.conf.getConf(READ_CONVERT_METASTORE_ORC) =>
- OrcScanBuilder(sparkSession, fileIndex, schema, dataSchema, options)
+ new KyuubiOrcScanBuilder(
+ sparkSession,
+ fileIndex,
+ schema,
+ dataSchema,
+ options,
+ catalogTable,
+ hiveTableCatalog)
case Some("PARQUET")
if sparkSession.sessionState.conf.getConf(READ_CONVERT_METASTORE_PARQUET) =>
- ParquetScanBuilder(sparkSession, fileIndex, schema, dataSchema, options)
+ new KyuubiParquetScanBuilder(
+ sparkSession,
+ fileIndex,
+ schema,
+ dataSchema,
+ options,
+ catalogTable,
+ hiveTableCatalog)
case _ => HiveScanBuilder(sparkSession, fileIndex, dataSchema, catalogTable)
}
}
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala
index 3d5c27ebb8..be91dec816 100644
--- a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala
@@ -77,7 +77,18 @@ class HiveTableCatalog(sparkSession: SparkSession)
SupportsNamespaces.PROP_LOCATION,
SupportsNamespaces.PROP_OWNER)
- private lazy val hadoopConf: Configuration = {
+ /**
+ * Cached Hadoop [[Configuration]] snapshot taken at first catalog use.
+ */
+ private lazy val hadoopConf: Configuration = buildHadoopConf()
+
+ /**
+ * Non-cached Hadoop [[Configuration]] for scan builders. Re-evaluated on
+ * every call so mid-session confs reach readers.
+ */
+ def newScanHadoopConf(): Configuration = buildHadoopConf()
+
+ private def buildHadoopConf(): Configuration = {
val conf = sparkSession.sessionState.newHadoopConf()
catalogOptions.asScala.foreach { case (k, v) => conf.set(k, v) }
if (catalogOptions.containsKey("hive.metastore.uris")) {
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/HiveScan.scala b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/HiveScan.scala
index 088fa6167e..7792ee7b44 100644
--- a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/HiveScan.scala
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/HiveScan.scala
@@ -28,7 +28,7 @@ import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable
import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Expression}
import org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection
import org.apache.spark.sql.connector.expressions.NamedReference
-import org.apache.spark.sql.connector.read.{PartitionReaderFactory, SupportsRuntimeFiltering}
+import org.apache.spark.sql.connector.read.{PartitionReaderFactory, Scan, SupportsRuntimeFiltering}
import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
import org.apache.spark.sql.execution.datasources.v2.FileScan
import org.apache.spark.sql.hive.kyuubi.connector.HiveBridgeHelper.HiveClientImpl
@@ -181,6 +181,27 @@ case class HiveScan(
// SupportsRuntimeFiltering implementation
// -------------------------------------------------------------------------------
+ /**
+ * The default [[Scan.ColumnarSupportMode.PARTITION_DEFINED]] (SPARK-44505)
+ * would drive `DataSourceV2ScanExecBase.supportsColumnar` to materialise
+ * `inputPartitions` during planning (via `HiveScan.partitions` ->
+ * `HiveCatalogFileIndex.listHiveFiles`), triggering a full-table HDFS
+ * listing before runtime filters arrive via
+ * [[SupportsRuntimeFiltering.filter]] and cancelling DPP's end-to-end win.
+ *
+ * [[HivePartitionReaderFactory]] only implements the row-based
+ * `createReader` path (no `supportColumnarReads` / `createColumnarReader`),
+ * so `HiveScan` is always row-based. Returning `UNSUPPORTED`
+ * is semantically equivalent to the default behaviour but short-circuits
+ * `supportsColumnar` without touching `inputPartitions`.
+ *
+ * NOTE: If [[HivePartitionReaderFactory]] ever gains columnar support,
+ * remove this override so `supportsColumnar` reflects reality, otherwise
+ * columnar-capable partitions would be silently reported as row-based.
+ */
+ override def columnarSupportMode(): Scan.ColumnarSupportMode =
+ Scan.ColumnarSupportMode.UNSUPPORTED
+
override def filterAttributes(): Array[NamedReference] = {
HiveRuntimeFilterSupport.filterAttributes(readPartitionSchema.fieldNames.toSeq)
}
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/KyuubiOrcScan.scala b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/KyuubiOrcScan.scala
new file mode 100644
index 0000000000..8c141df952
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/KyuubiOrcScan.scala
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.spark.connector.hive.read
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.connector.expressions.NamedReference
+import org.apache.spark.sql.connector.expressions.aggregate.Aggregation
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReaderFactory, Scan, SupportsRuntimeFiltering}
+import org.apache.spark.sql.execution.WholeStageCodegenExec
+import org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex
+import org.apache.spark.sql.execution.datasources.orc.OrcUtils
+import org.apache.spark.sql.execution.datasources.v2.FileScan
+import org.apache.spark.sql.execution.datasources.v2.orc.OrcScan
+import org.apache.spark.sql.sources.Filter
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+
+/**
+ * A DPP-aware wrapper around Spark's built-in [[OrcScan]] that adds
+ * [[SupportsRuntimeFiltering]] so Dynamic Partition Pruning can push runtime
+ * IN predicates down to the Hive partitioned scan.
+ *
+ * Implementation notes:
+ * 1. Only DPP-specific methods ([[filter]] / [[filterAttributes]] /
+ * [[planInputPartitions]]) contain custom logic, all other methods
+ * delegate to the wrapped [[OrcScan]].
+ * 2. [[equals]] uses a `KyuubiOrcScan` type pattern before delegating to
+ * `inner.equals`, so a plain [[OrcScan]] never compares equal and is
+ * never reused in its place during exchange/subquery reuse. [[hashCode]]
+ * is a constant, matching [[FileScan]]'s default.
+ * 3. Native engines (Gluten / Comet) identify scans by class name, so this
+ * wrapper is not recognized and falls back to JVM reads.
+ */
+class KyuubiOrcScan(
+ val sparkSession: SparkSession,
+ val hadoopConf: Configuration,
+ val fileIndex: PartitioningAwareFileIndex,
+ val dataSchema: StructType,
+ val readDataSchema: StructType,
+ val readPartitionSchema: StructType,
+ val options: CaseInsensitiveStringMap,
+ val pushedAggregate: Option[Aggregation],
+ val pushedFilters: Array[Filter],
+ val partitionFilters: Seq[Expression],
+ val dataFilters: Seq[Expression],
+ val catalogTable: CatalogTable)
+ extends FileScan
+ with SupportsRuntimeFiltering {
+
+ private[hive] val inner: OrcScan = OrcScan(
+ sparkSession,
+ hadoopConf,
+ fileIndex,
+ dataSchema,
+ readDataSchema,
+ readPartitionSchema,
+ options,
+ pushedAggregate,
+ pushedFilters,
+ partitionFilters,
+ dataFilters)
+
+ private var runtimeFilters: Seq[Expression] = Seq.empty
+
+ private val isCaseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+
+ /**
+ * The default [[Scan.ColumnarSupportMode.PARTITION_DEFINED]] (SPARK-44505)
+ * would drive `DataSourceV2ScanExecBase.supportsColumnar` to materialise
+ * `inputPartitions` during planning (via `FileScan.partitions` ->
+ * `HiveCatalogFileIndex.listFiles`), triggering a full-table HDFS listing
+ * before runtime filters arrive via [[SupportsRuntimeFiltering.filter]] and
+ * cancelling DPP's end-to-end win.
+ *
+ * We instead decide from sqlConf + schema, matching
+ * `OrcPartitionReaderFactory.supportColumnarReads` in all non-empty cases.
+ * When DPP prunes every partition, Spark's default would return
+ * `UNSUPPORTED` on the empty list, we still return `SUPPORTED`, adding a
+ * harmless `ColumnarToRow` on an empty RDD.
+ */
+ override def columnarSupportMode(): Scan.ColumnarSupportMode = {
+ val sqlConf = sparkSession.sessionState.conf
+ val schema = StructType(readDataSchema.fields ++ readPartitionSchema.fields)
+ val supportsColumnar = sqlConf.orcVectorizedReaderEnabled &&
+ sqlConf.wholeStageEnabled &&
+ !WholeStageCodegenExec.isTooManyFields(sqlConf, schema) &&
+ schema.forall(s =>
+ OrcUtils.supportColumnarReads(
+ s.dataType,
+ sqlConf.orcVectorizedReaderNestedColumnEnabled))
+ if (supportsColumnar) Scan.ColumnarSupportMode.SUPPORTED
+ else Scan.ColumnarSupportMode.UNSUPPORTED
+ }
+
+ override def filterAttributes(): Array[NamedReference] = {
+ // Under aggregate pushdown, the scan outputs aggregate columns only, so
+ // partition columns may be absent from its output, runtime filtering on
+ // them is also meaningless once results are aggregated.
+ if (pushedAggregate.nonEmpty) Array.empty[NamedReference]
+ else HiveRuntimeFilterSupport.filterAttributes(readPartitionSchema.fieldNames.toSeq)
+ }
+
+ override def filter(filters: Array[Filter]): Unit = {
+ runtimeFilters = HiveRuntimeFilterSupport.toCatalystPartitionFilters(
+ filters,
+ fileIndex.partitionSchema,
+ isCaseSensitive)
+ if (runtimeFilters.nonEmpty) {
+ logInfo(s"Received ${runtimeFilters.length} runtime partition filter(s) for " +
+ s"${catalogTable.identifier}")
+ logDebug(s"Runtime partition filter(s) for ${catalogTable.identifier}: " +
+ s"${runtimeFilters.mkString(", ")}")
+ }
+ }
+
+ override def planInputPartitions(): Array[InputPartition] = {
+ if (runtimeFilters.isEmpty) {
+ inner.planInputPartitions()
+ } else {
+ // Delegate planning to a sibling OrcScan carrying the merged
+ // partitionFilters ++ runtimeFilters so DPP predicates take effect.
+ val sibling = OrcScan(
+ sparkSession,
+ hadoopConf,
+ fileIndex,
+ dataSchema,
+ readDataSchema,
+ readPartitionSchema,
+ options,
+ pushedAggregate,
+ pushedFilters,
+ partitionFilters ++ runtimeFilters,
+ dataFilters)
+ sibling.planInputPartitions()
+ }
+ }
+
+ override def isSplitable(path: Path): Boolean = inner.isSplitable(path)
+
+ override def readSchema(): StructType = inner.readSchema()
+
+ override def getMetaData(): Map[String, String] = inner.getMetaData()
+
+ override def createReaderFactory(): PartitionReaderFactory = inner.createReaderFactory()
+
+ override def equals(obj: Any): Boolean = obj match {
+ case that: KyuubiOrcScan => this.inner.equals(that.inner)
+ case _ => false
+ }
+
+ override def hashCode(): Int = getClass.hashCode()
+}
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/KyuubiOrcScanBuilder.scala b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/KyuubiOrcScanBuilder.scala
new file mode 100644
index 0000000000..e4a2e45756
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/KyuubiOrcScanBuilder.scala
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.spark.connector.hive.read
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.connector.expressions.aggregate.Aggregation
+import org.apache.spark.sql.connector.read.SupportsPushDownAggregates
+import org.apache.spark.sql.execution.datasources.{AggregatePushDownUtils, PartitioningAwareFileIndex}
+import org.apache.spark.sql.execution.datasources.v2.FileScanBuilder
+import org.apache.spark.sql.hive.kyuubi.connector.HiveBridgeHelper
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.sources.Filter
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+
+import org.apache.kyuubi.spark.connector.hive.HiveTableCatalog
+
+/**
+ * A ScanBuilder that mirrors Spark's built-in [[OrcScanBuilder]] but builds
+ * [[KyuubiOrcScan]] instances, which additionally implement
+ * `SupportsRuntimeFiltering` so that Dynamic Partition Pruning works when
+ * a Hive ORC table goes through Spark's vectorized ORC reader path.
+ *
+ * Filter, aggregate and column pushdown behaviour matches [[OrcScanBuilder]].
+ */
+class KyuubiOrcScanBuilder(
+ sparkSession: SparkSession,
+ fileIndex: PartitioningAwareFileIndex,
+ schema: StructType,
+ dataSchema: StructType,
+ options: CaseInsensitiveStringMap,
+ catalogTable: CatalogTable,
+ hiveTableCatalog: HiveTableCatalog)
+ extends FileScanBuilder(sparkSession, fileIndex, dataSchema)
+ with SupportsPushDownAggregates {
+
+ /**
+ * Cloned from a freshly-built per-catalog Hadoop [[Configuration]] so
+ * per-catalog settings and mid-session confs are both honored, matching
+ * Spark's built-in `OrcScanBuilder.hadoopConf`. Cloned so per-scan
+ * `options` do not pollute the source instance.
+ */
+ lazy val hadoopConf: Configuration = {
+ val conf = new Configuration(hiveTableCatalog.newScanHadoopConf())
+ // Hadoop Configurations are case sensitive.
+ options.asCaseSensitiveMap.asScala.foreach { case (k, v) => conf.set(k, v) }
+ conf
+ }
+
+ private var finalSchema = new StructType()
+
+ private var pushedAggregations = Option.empty[Aggregation]
+
+ override protected val supportsNestedSchemaPruning: Boolean = true
+
+ override def build(): KyuubiOrcScan = {
+ // the `finalSchema` is either pruned in pushAggregation (if aggregates are
+ // pushed down), or pruned in readDataSchema() (in regular column pruning). These
+ // two are mutual exclusive.
+ if (pushedAggregations.isEmpty) {
+ finalSchema = readDataSchema()
+ }
+ new KyuubiOrcScan(
+ sparkSession,
+ hadoopConf,
+ fileIndex,
+ dataSchema,
+ finalSchema,
+ readPartitionSchema(),
+ options,
+ pushedAggregations,
+ pushedDataFilters,
+ partitionFilters,
+ dataFilters,
+ catalogTable)
+ }
+
+ override def pushDataFilters(dataFilters: Array[Filter]): Array[Filter] = {
+ if (sparkSession.sessionState.conf.orcFilterPushDown) {
+ HiveBridgeHelper.orcConvertibleFilters(
+ readDataSchema(),
+ SQLConf.get.caseSensitiveAnalysis,
+ dataFilters.toSeq).toArray
+ } else {
+ Array.empty[Filter]
+ }
+ }
+
+ override def pushAggregation(aggregation: Aggregation): Boolean = {
+ if (!sparkSession.sessionState.conf.orcAggregatePushDown) {
+ return false
+ }
+
+ AggregatePushDownUtils.getSchemaForPushedAggregation(
+ aggregation,
+ schema,
+ partitionNameSet,
+ dataFilters) match {
+
+ case Some(schema) =>
+ finalSchema = schema
+ this.pushedAggregations = Some(aggregation)
+ true
+ case _ => false
+ }
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/KyuubiParquetScan.scala b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/KyuubiParquetScan.scala
new file mode 100644
index 0000000000..c517f54bc1
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/KyuubiParquetScan.scala
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.spark.connector.hive.read
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.connector.expressions.NamedReference
+import org.apache.spark.sql.connector.expressions.aggregate.Aggregation
+import org.apache.spark.sql.connector.read.{InputPartition, PartitionReaderFactory, Scan, SupportsRuntimeFiltering}
+import org.apache.spark.sql.execution.WholeStageCodegenExec
+import org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex
+import org.apache.spark.sql.execution.datasources.parquet.ParquetUtils
+import org.apache.spark.sql.execution.datasources.v2.FileScan
+import org.apache.spark.sql.execution.datasources.v2.parquet.ParquetScan
+import org.apache.spark.sql.sources.Filter
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+
+import org.apache.kyuubi.util.reflect.{DynClasses, DynConstructors}
+
+/**
+ * A DPP-aware wrapper around Spark's built-in [[ParquetScan]] that adds
+ * [[SupportsRuntimeFiltering]] so Dynamic Partition Pruning can push runtime
+ * IN predicates down to the Hive partitioned scan.
+ *
+ * Implementation notes:
+ * 1. Only DPP-specific methods ([[filter]] / [[filterAttributes]] /
+ * [[planInputPartitions]]) contain custom logic, all other methods
+ * delegate to the wrapped [[ParquetScan]].
+ * 2. [[equals]] uses a `KyuubiParquetScan` type pattern before delegating to
+ * `inner.equals`, so a plain [[ParquetScan]] never compares equal and is
+ * never reused in its place during exchange/subquery reuse. [[hashCode]]
+ * is a constant, matching [[FileScan]]'s default.
+ * 3. Native engines (Gluten / Comet) identify scans by class name, so this
+ * wrapper is not recognized and falls back to JVM reads.
+ */
+class KyuubiParquetScan(
+ val sparkSession: SparkSession,
+ val hadoopConf: Configuration,
+ val fileIndex: PartitioningAwareFileIndex,
+ val dataSchema: StructType,
+ val readDataSchema: StructType,
+ val readPartitionSchema: StructType,
+ val pushedFilters: Array[Filter],
+ val options: CaseInsensitiveStringMap,
+ val pushedAggregate: Option[Aggregation],
+ val partitionFilters: Seq[Expression],
+ val dataFilters: Seq[Expression],
+ val catalogTable: CatalogTable)
+ extends FileScan
+ with SupportsRuntimeFiltering {
+
+ private[hive] val inner: ParquetScan = KyuubiParquetScan.newParquetScan(
+ sparkSession,
+ hadoopConf,
+ fileIndex,
+ dataSchema,
+ readDataSchema,
+ readPartitionSchema,
+ pushedFilters,
+ options,
+ pushedAggregate,
+ partitionFilters,
+ dataFilters)
+
+ private var runtimeFilters: Seq[Expression] = Seq.empty
+
+ private val isCaseSensitive = sparkSession.sessionState.conf.caseSensitiveAnalysis
+
+ /**
+ * The default [[Scan.ColumnarSupportMode.PARTITION_DEFINED]] (SPARK-44505)
+ * would drive `DataSourceV2ScanExecBase.supportsColumnar` to materialise
+ * `inputPartitions` during planning (via `FileScan.partitions` ->
+ * `HiveCatalogFileIndex.listFiles`), triggering a full-table HDFS listing
+ * before runtime filters arrive via [[SupportsRuntimeFiltering.filter]] and
+ * cancelling DPP's end-to-end win.
+ *
+ * We instead decide from sqlConf + schema, matching
+ * `ParquetPartitionReaderFactory.supportColumnarReads` in all non-empty
+ * cases. When DPP prunes every partition, Spark's default would return
+ * `UNSUPPORTED` on the empty list, we still return `SUPPORTED`, adding a
+ * harmless `ColumnarToRow` on an empty RDD.
+ */
+ override def columnarSupportMode(): Scan.ColumnarSupportMode = {
+ val sqlConf = sparkSession.sessionState.conf
+ val schema = StructType(readDataSchema.fields ++ readPartitionSchema.fields)
+ val supportsColumnar = ParquetUtils.isBatchReadSupportedForSchema(sqlConf, schema) &&
+ sqlConf.wholeStageEnabled &&
+ !WholeStageCodegenExec.isTooManyFields(sqlConf, schema)
+ if (supportsColumnar) Scan.ColumnarSupportMode.SUPPORTED
+ else Scan.ColumnarSupportMode.UNSUPPORTED
+ }
+
+ override def filterAttributes(): Array[NamedReference] = {
+ // Under aggregate pushdown, the scan outputs aggregate columns only, so
+ // partition columns may be absent from its output, runtime filtering on
+ // them is also meaningless once results are aggregated.
+ if (pushedAggregate.nonEmpty) Array.empty[NamedReference]
+ else HiveRuntimeFilterSupport.filterAttributes(readPartitionSchema.fieldNames.toSeq)
+ }
+
+ override def filter(filters: Array[Filter]): Unit = {
+ runtimeFilters = HiveRuntimeFilterSupport.toCatalystPartitionFilters(
+ filters,
+ fileIndex.partitionSchema,
+ isCaseSensitive)
+ if (runtimeFilters.nonEmpty) {
+ logInfo(s"Received ${runtimeFilters.length} runtime partition filter(s) for " +
+ s"${catalogTable.identifier}")
+ logDebug(s"Runtime partition filter(s) for ${catalogTable.identifier}: " +
+ s"${runtimeFilters.mkString(", ")}")
+ }
+ }
+
+ override def planInputPartitions(): Array[InputPartition] = {
+ if (runtimeFilters.isEmpty) {
+ inner.planInputPartitions()
+ } else {
+ // Delegate planning to a sibling ParquetScan carrying the merged
+ // partitionFilters ++ runtimeFilters so DPP predicates take effect.
+ val sibling = KyuubiParquetScan.newParquetScan(
+ sparkSession,
+ hadoopConf,
+ fileIndex,
+ dataSchema,
+ readDataSchema,
+ readPartitionSchema,
+ pushedFilters,
+ options,
+ pushedAggregate,
+ partitionFilters ++ runtimeFilters,
+ dataFilters)
+ sibling.planInputPartitions()
+ }
+ }
+
+ override def isSplitable(path: Path): Boolean = inner.isSplitable(path)
+
+ override def readSchema(): StructType = inner.readSchema()
+
+ override def getMetaData(): Map[String, String] = inner.getMetaData()
+
+ override def createReaderFactory(): PartitionReaderFactory = inner.createReaderFactory()
+
+ override def equals(obj: Any): Boolean = obj match {
+ case that: KyuubiParquetScan => this.inner.equals(that.inner)
+ case _ => false
+ }
+
+ override def hashCode(): Int = getClass.hashCode()
+}
+
+object KyuubiParquetScan {
+
+ // Element type of `Array[VariantExtraction]` in Spark 4.1+, null when absent.
+ private lazy val variantExtractionCls: Class[_] = DynClasses.builder()
+ .impl("org.apache.spark.sql.connector.read.VariantExtraction")
+ .orNull()
+ .build()
+
+ private lazy val emptyVariantExtractions: AnyRef =
+ if (variantExtractionCls == null) null
+ else java.lang.reflect.Array.newInstance(variantExtractionCls, 0).asInstanceOf[AnyRef]
+
+ private lazy val parquetScanCtor: DynConstructors.Ctor[ParquetScan] =
+ if (variantExtractionCls != null) {
+ DynConstructors.builder()
+ .impl( // SPARK-53880 / SPARK-54656 (4.1.0): adds trailing Array[VariantExtraction]
+ classOf[ParquetScan],
+ classOf[SparkSession],
+ classOf[Configuration],
+ classOf[PartitioningAwareFileIndex],
+ classOf[StructType],
+ classOf[StructType],
+ classOf[StructType],
+ classOf[Array[Filter]],
+ classOf[CaseInsensitiveStringMap],
+ classOf[Option[Aggregation]],
+ classOf[Seq[Expression]],
+ classOf[Seq[Expression]],
+ emptyVariantExtractions.getClass)
+ .buildChecked[ParquetScan]()
+ } else {
+ DynConstructors.builder()
+ .impl( // Spark 4.0 and previous
+ classOf[ParquetScan],
+ classOf[SparkSession],
+ classOf[Configuration],
+ classOf[PartitioningAwareFileIndex],
+ classOf[StructType],
+ classOf[StructType],
+ classOf[StructType],
+ classOf[Array[Filter]],
+ classOf[CaseInsensitiveStringMap],
+ classOf[Option[Aggregation]],
+ classOf[Seq[Expression]],
+ classOf[Seq[Expression]])
+ .buildChecked[ParquetScan]()
+ }
+
+ // scalastyle:off parameter.number
+ private[hive] def newParquetScan(
+ sparkSession: SparkSession,
+ hadoopConf: Configuration,
+ fileIndex: PartitioningAwareFileIndex,
+ dataSchema: StructType,
+ readDataSchema: StructType,
+ readPartitionSchema: StructType,
+ pushedFilters: Array[Filter],
+ options: CaseInsensitiveStringMap,
+ pushedAggregate: Option[Aggregation],
+ partitionFilters: Seq[Expression],
+ dataFilters: Seq[Expression]): ParquetScan = {
+ // `DynConstructors` truncates trailing args to match the resolved ctor arity,
+ // so always passing `emptyVariantExtractions` is safe on pre-4.1 Spark too.
+ parquetScanCtor.newInstance(
+ sparkSession,
+ hadoopConf,
+ fileIndex,
+ dataSchema,
+ readDataSchema,
+ readPartitionSchema,
+ pushedFilters,
+ options,
+ pushedAggregate,
+ partitionFilters,
+ dataFilters,
+ emptyVariantExtractions)
+ }
+ // scalastyle:on parameter.number
+}
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/KyuubiParquetScanBuilder.scala b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/KyuubiParquetScanBuilder.scala
new file mode 100644
index 0000000000..5deac91f41
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/KyuubiParquetScanBuilder.scala
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.spark.connector.hive.read
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.connector.expressions.aggregate.Aggregation
+import org.apache.spark.sql.connector.read.SupportsPushDownAggregates
+import org.apache.spark.sql.execution.datasources.{AggregatePushDownUtils, PartitioningAwareFileIndex}
+import org.apache.spark.sql.execution.datasources.v2.FileScanBuilder
+import org.apache.spark.sql.hive.kyuubi.connector.HiveBridgeHelper
+import org.apache.spark.sql.sources.Filter
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+
+import org.apache.kyuubi.spark.connector.hive.HiveTableCatalog
+
+/**
+ * A ScanBuilder that mirrors Spark's built-in [[ParquetScanBuilder]] but builds
+ * [[KyuubiParquetScan]] instances, which additionally implement
+ * `SupportsRuntimeFiltering` so that Dynamic Partition Pruning works when
+ * a Hive Parquet table goes through Spark's vectorized Parquet reader path.
+ *
+ * Filter, aggregate and column pushdown behaviour matches [[ParquetScanBuilder]].
+ * Gap: on Spark 4.1+ [[ParquetScanBuilder]] also mixes in
+ * `SupportsPushDownVariantExtractions`, which this builder does not support,
+ * but this is unreachable in practice. Hive-serde Parquet tables have no way
+ * to declare `variant` data columns (Hive 4.2 adds `variant` only for Iceberg
+ * tables, see HIVE-29184).
+ */
+class KyuubiParquetScanBuilder(
+ sparkSession: SparkSession,
+ fileIndex: PartitioningAwareFileIndex,
+ schema: StructType,
+ dataSchema: StructType,
+ options: CaseInsensitiveStringMap,
+ catalogTable: CatalogTable,
+ hiveTableCatalog: HiveTableCatalog)
+ extends FileScanBuilder(sparkSession, fileIndex, dataSchema)
+ with SupportsPushDownAggregates {
+
+ /**
+ * Cloned from a freshly-built per-catalog Hadoop [[Configuration]] so
+ * per-catalog settings and mid-session confs are both honored, matching
+ * Spark's built-in `ParquetScanBuilder.hadoopConf`. Cloned so per-scan
+ * `options` do not pollute the source instance.
+ */
+ lazy val hadoopConf: Configuration = {
+ val conf = new Configuration(hiveTableCatalog.newScanHadoopConf())
+ // Hadoop Configurations are case sensitive.
+ options.asCaseSensitiveMap.asScala.foreach { case (k, v) => conf.set(k, v) }
+ conf
+ }
+
+ private var finalSchema = new StructType()
+
+ private var pushedAggregations = Option.empty[Aggregation]
+
+ override protected val supportsNestedSchemaPruning: Boolean = true
+
+ override def build(): KyuubiParquetScan = {
+ // the `finalSchema` is either pruned in pushAggregation (if aggregates are
+ // pushed down), or pruned in readDataSchema() (in regular column pruning). These
+ // two are mutual exclusive.
+ if (pushedAggregations.isEmpty) {
+ finalSchema = readDataSchema()
+ }
+ new KyuubiParquetScan(
+ sparkSession,
+ hadoopConf,
+ fileIndex,
+ dataSchema,
+ finalSchema,
+ readPartitionSchema(),
+ pushedDataFilters,
+ options,
+ pushedAggregations,
+ partitionFilters,
+ dataFilters,
+ catalogTable)
+ }
+
+ override def pushDataFilters(dataFilters: Array[Filter]): Array[Filter] = {
+ if (sparkSession.sessionState.conf.parquetFilterPushDown) {
+ HiveBridgeHelper.parquetConvertibleFilters(readDataSchema(), dataFilters.toSeq).toArray
+ } else {
+ Array.empty[Filter]
+ }
+ }
+
+ override def pushAggregation(aggregation: Aggregation): Boolean = {
+ if (!sparkSession.sessionState.conf.parquetAggregatePushDown) {
+ return false
+ }
+
+ AggregatePushDownUtils.getSchemaForPushedAggregation(
+ aggregation,
+ schema,
+ partitionNameSet,
+ dataFilters) match {
+
+ case Some(schema) =>
+ finalSchema = schema
+ this.pushedAggregations = Some(aggregation)
+ true
+ case _ => false
+ }
+ }
+}
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/spark/sql/hive/kyuubi/connector/HiveBridgeHelper.scala b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/spark/sql/hive/kyuubi/connector/HiveBridgeHelper.scala
index 2993a0f12d..afe9a7a6d1 100644
--- a/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/spark/sql/hive/kyuubi/connector/HiveBridgeHelper.scala
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/spark/sql/hive/kyuubi/connector/HiveBridgeHelper.scala
@@ -22,9 +22,14 @@ import scala.collection.mutable
import org.apache.spark.SparkContext
import org.apache.spark.sql.catalyst.catalog.{BucketSpec, ExternalCatalogEvent}
import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Literal}
+import org.apache.spark.sql.catalyst.util.RebaseDateTime.RebaseSpec
import org.apache.spark.sql.catalyst.util.quoteIfNeeded
import org.apache.spark.sql.connector.expressions.{BucketTransform, FieldReference, IdentityTransform, Transform}
import org.apache.spark.sql.connector.expressions.LogicalExpressions.{bucket, reference}
+import org.apache.spark.sql.execution.datasources.orc.OrcFilters
+import org.apache.spark.sql.execution.datasources.parquet.{ParquetFilters, SparkToParquetSchemaConverter}
+import org.apache.spark.sql.internal.{LegacyBehaviorPolicy, SQLConf}
+import org.apache.spark.sql.sources.Filter
import org.apache.spark.sql.types.{DataType, DoubleType, FloatType, StructType}
object HiveBridgeHelper {
@@ -119,4 +124,38 @@ object HiveBridgeHelper {
implicit class NamespaceHelper(namespace: Array[String]) {
def quoted: String = namespace.map(quoteIfNeeded).mkString(".")
}
+
+ def orcConvertibleFilters(
+ schema: StructType,
+ caseSensitive: Boolean,
+ dataFilters: Seq[Filter]): Seq[Filter] = {
+ val dataTypeMap = OrcFilters.getSearchableTypeMap(schema, caseSensitive)
+ OrcFilters.convertibleFilters(dataTypeMap, dataFilters)
+ }
+
+ def parquetConvertibleFilters(
+ readDataSchema: StructType,
+ dataFilters: Seq[Filter]): Seq[Filter] = {
+ val sqlConf = SQLConf.get
+ val pushDownDate = sqlConf.parquetFilterPushDownDate
+ val pushDownTimestamp = sqlConf.parquetFilterPushDownTimestamp
+ val pushDownDecimal = sqlConf.parquetFilterPushDownDecimal
+ val pushDownStringPredicate = sqlConf.parquetFilterPushDownStringPredicate
+ val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
+ val isCaseSensitive = sqlConf.caseSensitiveAnalysis
+ val parquetSchema = new SparkToParquetSchemaConverter(sqlConf).convert(readDataSchema)
+ val rebaseSpec = RebaseSpec(LegacyBehaviorPolicy.CORRECTED)
+ val parquetFilters = new ParquetFilters(
+ parquetSchema,
+ pushDownDate,
+ pushDownTimestamp,
+ pushDownDecimal,
+ pushDownStringPredicate,
+ pushDownInFilterThreshold,
+ isCaseSensitive,
+ // The rebase mode doesn't matter here because the filters are used to determine
+ // whether they is convertible.
+ rebaseSpec)
+ parquetFilters.convertibleFilters(dataFilters)
+ }
}
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/DynamicPartitionPruningSuite.scala b/extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/DynamicPartitionPruningSuite.scala
index 602c8cc1fc..b6bcad7028 100644
--- a/extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/DynamicPartitionPruningSuite.scala
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/DynamicPartitionPruningSuite.scala
@@ -19,46 +19,51 @@ package org.apache.kyuubi.spark.connector.hive
import scala.annotation.tailrec
-import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.Row
import org.apache.spark.sql.catalyst.expressions.DynamicPruningExpression
+import org.apache.spark.sql.connector.read.Scan
import org.apache.spark.sql.execution.SparkPlan
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec
import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.spark.sql.internal.SQLConf
-import org.apache.kyuubi.spark.connector.hive.read.HiveScan
+import org.apache.kyuubi.spark.connector.hive.read.{HiveScan, KyuubiOrcScan, KyuubiParquetScan}
class DynamicPartitionPruningSuite extends KyuubiHiveTest {
- private def findBatchScanExec(
- spark: SparkSession,
- sql: String,
- tableNameHint: String): BatchScanExec = {
- // Match on `HiveScan.catalogTable` rather than the node's `toString` because
- // `BatchScanExec.toString` shape differs across Spark versions.
- def matchesHint(b: BatchScanExec): Boolean = b.scan match {
- case h: HiveScan => h.catalogTable.identifier.table == tableNameHint
- case _ => false
+ private def findBatchScanExec(plan: SparkPlan, tableNameHint: String): BatchScanExec = {
+ // Match on the underlying Hive `catalogTable` rather than the node's `toString`
+ // because `BatchScanExec.toString` shape differs across Spark versions.
+ def hiveTableName(b: BatchScanExec): Option[String] = b.scan match {
+ case h: HiveScan => Some(h.catalogTable.identifier.table)
+ case o: KyuubiOrcScan => Some(o.catalogTable.identifier.table)
+ case p: KyuubiParquetScan => Some(p.catalogTable.identifier.table)
+ case _ => None
}
@tailrec
- def findBatchScan(plan: SparkPlan): Option[BatchScanExec] = plan match {
+ def findBatchScan(p: SparkPlan): Option[BatchScanExec] = p match {
case aqe: AdaptiveSparkPlanExec => findBatchScan(aqe.inputPlan)
- case _ => plan.collectFirst {
- case b: BatchScanExec if matchesHint(b) => b
+ case _ => p.collectFirst {
+ case b: BatchScanExec if hiveTableName(b).contains(tableNameHint) => b
}
}
- val exec = findBatchScan(spark.sql(sql).queryExecution.executedPlan)
+ val exec = findBatchScan(plan)
assert(exec.isDefined)
exec.get
}
- test("HiveScan supports DPP runtime filtering on partition columns") {
+ private def runDppCase(storedAs: String): Unit = {
+ // Collect the number of input partitions actually planned under DPP on / off
+ // and later assert a strict reduction.
+ val plannedPartitions = scala.collection.mutable.Map.empty[Boolean, Int]
+
Seq(true, false).foreach { enabled =>
withSparkSession(Map(
"hive.exec.dynamic.partition.mode" -> "nonstrict",
- "spark.sql.optimizer.dynamicPartitionPruning.enabled" -> enabled.toString)) { spark =>
- val suffix = if (enabled) "on" else "off"
+ SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> enabled.toString)) { spark =>
+ val suffix = s"${storedAs.toLowerCase}_${if (enabled) "on" else "off"}"
val fact = s"hive.default.dpp_fact_$suffix"
val dim = s"hive.default.dpp_dim_$suffix"
@@ -66,8 +71,8 @@ class DynamicPartitionPruningSuite extends KyuubiHiveTest {
spark.sql(
s"""
| CREATE TABLE $fact (id INT, v STRING) PARTITIONED BY (dt STRING)
- | STORED AS TEXTFILE
- |""".stripMargin).collect()
+ | STORED AS $storedAs
+ |""".stripMargin)
spark.sql(s"INSERT INTO $fact PARTITION (dt='2026-01-01') VALUES (1, 'a'), (2, 'b')")
spark.sql(s"INSERT INTO $fact PARTITION (dt='2026-05-01') VALUES (3, 'c'), (4, 'd')")
spark.sql(s"INSERT INTO $fact PARTITION (dt='2026-09-01') VALUES (5, 'e'), (6, 'f')")
@@ -75,30 +80,160 @@ class DynamicPartitionPruningSuite extends KyuubiHiveTest {
spark.sql(
s"""
| CREATE TABLE $dim (dt STRING, tag STRING)
- | STORED AS TEXTFILE
- |""".stripMargin).collect()
+ | STORED AS $storedAs
+ |""".stripMargin)
spark.sql(s"INSERT INTO $dim VALUES ('2026-05-01', 'target')")
- val sql =
+ val df = spark.sql(
s"""
| SELECT f.id, f.v, f.dt
| FROM $fact f JOIN $dim d ON f.dt = d.dt
| WHERE d.tag = 'target'
- |""".stripMargin
-
+ |""".stripMargin)
checkAnswer(
- spark.sql(sql),
+ df,
Seq(
Row(3, "c", "2026-05-01"),
Row(4, "d", "2026-05-01")))
// DPP being actually applied is observable as a `DynamicPruningExpression`
// injected into `BatchScanExec.runtimeFilters`.
- val exec = findBatchScanExec(spark, sql, fact.split('.').last)
+ val exec = findBatchScanExec(df.queryExecution.executedPlan, fact.split('.').last)
val hasDpp = exec.runtimeFilters.exists(_.isInstanceOf[DynamicPruningExpression])
assert(hasDpp == enabled)
+
+ val planned = exec.scan.toBatch.planInputPartitions().length
+ plannedPartitions(enabled) = planned
+
+ exec.scan match {
+ case _: KyuubiOrcScan | _: KyuubiParquetScan =>
+ assert(exec.scan.columnarSupportMode() == Scan.ColumnarSupportMode.SUPPORTED)
+ case _: HiveScan =>
+ assert(exec.scan.columnarSupportMode() == Scan.ColumnarSupportMode.UNSUPPORTED)
+ case other =>
+ fail(s"unexpected scan type: ${other.getClass.getName}")
+ }
+ }
+ }
+ }
+
+ val planOn = plannedPartitions(true)
+ val planOff = plannedPartitions(false)
+ assert(
+ planOn < planOff,
+ s"DPP ($storedAs) should plan fewer partitions when enabled")
+ }
+
+ test("HiveScan supports DPP runtime filtering on partition columns") {
+ runDppCase(storedAs = "TEXTFILE")
+ }
+
+ test("KyuubiOrcScan supports DPP runtime filtering on partition columns") {
+ runDppCase(storedAs = "ORC")
+ }
+
+ test("KyuubiParquetScan supports DPP runtime filtering on partition columns") {
+ runDppCase(storedAs = "PARQUET")
+ }
+
+ /**
+ * Build a fact table and assert `columnarSupportMode()` matches
+ * `expectedColumnarMode` under the given `extraConf`.
+ */
+ private def runColumnarModeCase(
+ storedAs: String,
+ extraConf: Map[String, String],
+ expectedColumnarMode: Scan.ColumnarSupportMode): Unit = {
+ withSparkSession(extraConf ++ Map(
+ "hive.exec.dynamic.partition.mode" -> "nonstrict")) { spark =>
+ val fact = s"hive.default.mode_fact_${storedAs.toLowerCase}"
+ dropTableAfter(fact) {
+ spark.sql(
+ s"""
+ | CREATE TABLE $fact (id INT, v STRING) PARTITIONED BY (dt STRING)
+ | STORED AS $storedAs
+ |""".stripMargin)
+ spark.sql(s"INSERT INTO $fact PARTITION (dt='2026-05-01') VALUES (1, 'a')")
+
+ val df = spark.sql(s"SELECT id, v, dt FROM $fact")
+ val exec = findBatchScanExec(df.queryExecution.executedPlan, fact.split('.').last)
+ exec.scan match {
+ case _: KyuubiOrcScan | _: KyuubiParquetScan =>
+ assert(exec.scan.columnarSupportMode() == expectedColumnarMode)
+ case other =>
+ fail(s"unexpected scan type: ${other.getClass.getName}")
+ }
+ }
+ }
+ }
+
+ test("KyuubiOrcScan returns UNSUPPORTED when orc vectorized reader is disabled") {
+ runColumnarModeCase(
+ storedAs = "ORC",
+ extraConf = Map(SQLConf.ORC_VECTORIZED_READER_ENABLED.key -> "false"),
+ expectedColumnarMode = Scan.ColumnarSupportMode.UNSUPPORTED)
+ }
+
+ test("KyuubiParquetScan returns UNSUPPORTED when parquet vectorized reader is disabled") {
+ runColumnarModeCase(
+ storedAs = "PARQUET",
+ extraConf = Map(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false"),
+ expectedColumnarMode = Scan.ColumnarSupportMode.UNSUPPORTED)
+ }
+
+ private def runAllPrunedCase(storedAs: String): Unit = {
+ withSparkSession(Map(
+ "hive.exec.dynamic.partition.mode" -> "nonstrict",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true")) { spark =>
+ val suffix = storedAs.toLowerCase
+ val fact = s"hive.default.pruned_fact_$suffix"
+ val dim = s"hive.default.pruned_dim_$suffix"
+
+ dropTableAfter(fact, dim) {
+ spark.sql(
+ s"""
+ | CREATE TABLE $fact (id INT, v STRING) PARTITIONED BY (dt STRING)
+ | STORED AS $storedAs
+ |""".stripMargin)
+ spark.sql(s"INSERT INTO $fact PARTITION (dt='2026-01-01') VALUES (1, 'a')")
+ spark.sql(s"INSERT INTO $fact PARTITION (dt='2026-05-01') VALUES (2, 'b')")
+
+ spark.sql(
+ s"""
+ | CREATE TABLE $dim (dt STRING, tag STRING)
+ | STORED AS $storedAs
+ |""".stripMargin)
+ // Dim key matches no fact partition, so DPP prunes every fact partition.
+ spark.sql(s"INSERT INTO $dim VALUES ('1999-12-31', 'target')")
+
+ val df = spark.sql(
+ s"""
+ | SELECT f.id, f.v, f.dt
+ | FROM $fact f JOIN $dim d ON f.dt = d.dt
+ | WHERE d.tag = 'target'
+ |""".stripMargin)
+ // Trigger full execution so `BatchScanExec.filteredPartitions` pushes
+ // runtime filters into the wrapped scan (via `SupportsRuntimeFiltering`)
+ // and `planInputPartitions()` below reflects DPP-pruned partitions.
+ assert(df.collect().isEmpty)
+
+ val exec = findBatchScanExec(df.queryExecution.executedPlan, fact.split('.').last)
+ exec.scan match {
+ case _: KyuubiOrcScan | _: KyuubiParquetScan =>
+ assert(exec.scan.toBatch.planInputPartitions().isEmpty)
+ assert(exec.scan.columnarSupportMode() == Scan.ColumnarSupportMode.SUPPORTED)
+ case other =>
+ fail(s"unexpected scan type: ${other.getClass.getName}")
}
}
}
}
+
+ test("KyuubiOrcScan returns SUPPORTED when DPP prunes every partition") {
+ runAllPrunedCase(storedAs = "ORC")
+ }
+
+ test("KyuubiParquetScan returns SUPPORTED when DPP prunes every partition") {
+ runAllPrunedCase(storedAs = "PARQUET")
+ }
}
diff --git a/extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/HiveCatalogSuite.scala b/extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/HiveCatalogSuite.scala
index 0e276bdef7..bd39844c62 100644
--- a/extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/HiveCatalogSuite.scala
+++ b/extensions/spark/kyuubi-spark-connector-hive/src/test/scala/org/apache/kyuubi/spark/connector/hive/HiveCatalogSuite.scala
@@ -32,15 +32,14 @@ import org.apache.spark.sql.catalyst.catalog.CatalogTableType
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.apache.spark.sql.connector.catalog.{Identifier, SupportsNamespaces, TableCatalog}
import org.apache.spark.sql.connector.expressions.Transform
-import org.apache.spark.sql.execution.datasources.v2.orc.OrcScan
-import org.apache.spark.sql.execution.datasources.v2.parquet.ParquetScan
import org.apache.spark.sql.hive.kyuubi.connector.HiveBridgeHelper._
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
import org.apache.spark.sql.util.CaseInsensitiveStringMap
import org.apache.kyuubi.spark.connector.hive.HiveTableCatalog.IdentifierHelper
import org.apache.kyuubi.spark.connector.hive.KyuubiHiveConnectorConf.{READ_CONVERT_METASTORE_ORC, READ_CONVERT_METASTORE_PARQUET}
-import org.apache.kyuubi.spark.connector.hive.read.HiveScan
+import org.apache.kyuubi.spark.connector.hive.read.{HiveScan, KyuubiOrcScan, KyuubiParquetScan}
class HiveCatalogSuite extends KyuubiHiveTest {
@@ -509,9 +508,9 @@ class HiveCatalogSuite extends KyuubiHiveTest {
val parScan = value match {
case "true" =>
assert(
- scan.isInstanceOf[ParquetScan],
- s"Expected ParquetScan, got ${scan.getClass.getSimpleName}")
- scan.asInstanceOf[ParquetScan]
+ scan.isInstanceOf[KyuubiParquetScan],
+ s"Expected KyuubiParquetScan, got ${scan.getClass.getSimpleName}")
+ scan.asInstanceOf[KyuubiParquetScan]
case "false" =>
assert(
scan.isInstanceOf[HiveScan],
@@ -538,9 +537,9 @@ class HiveCatalogSuite extends KyuubiHiveTest {
val orcScan = value match {
case "true" =>
assert(
- scan.isInstanceOf[OrcScan],
- s"Expected OrcScan, got ${scan.getClass.getSimpleName}")
- scan.asInstanceOf[OrcScan]
+ scan.isInstanceOf[KyuubiOrcScan],
+ s"Expected KyuubiOrcScan, got ${scan.getClass.getSimpleName}")
+ scan.asInstanceOf[KyuubiOrcScan]
case "false" =>
assert(
scan.isInstanceOf[HiveScan],
@@ -554,4 +553,40 @@ class HiveCatalogSuite extends KyuubiHiveTest {
}
}
}
+
+ test("KyuubiParquetScan and KyuubiOrcScan pick up mid-session confs and catalog overlay") {
+ // Confs set in the session must take effect for the reader.
+ val fieldIdRead = SQLConf.PARQUET_FIELD_ID_READ_ENABLED.key
+ val fieldIdIgnoreMissing = SQLConf.IGNORE_MISSING_PARQUET_FIELD_ID.key
+ // Injected in `newCatalog` as camelCase, overlay writes it as lowercase.
+ val overlayKey = "javax.jdo.option.connectionurl"
+
+ Seq("orc", "parquet").foreach { provider =>
+ withSparkSession() { spark =>
+ val props = new util.HashMap[String, String]()
+ props.put(TableCatalog.PROP_PROVIDER, provider)
+ val ident = Identifier.of(testNs, s"scan_conf_$provider")
+
+ try {
+ val table = catalog.createTable(ident, schema, Array.empty[Transform], props)
+ spark.sessionState.conf.setConfString(fieldIdRead, "true")
+ spark.sessionState.conf.setConfString(fieldIdIgnoreMissing, "true")
+
+ val hadoopConf = table.asInstanceOf[HiveTable]
+ .newScanBuilder(CaseInsensitiveStringMap.empty()).build() match {
+ case s: KyuubiParquetScan => s.hadoopConf
+ case s: KyuubiOrcScan => s.hadoopConf
+ case other => fail(s"unexpected scan type: ${other.getClass.getName}")
+ }
+ // Mid-session confs reach the reader (no snapshot freeze).
+ assert(hadoopConf.get(fieldIdRead) == "true")
+ assert(hadoopConf.get(fieldIdIgnoreMissing) == "true")
+ // Per-catalog overlay applied (lowercase key).
+ assert(hadoopConf.get(overlayKey) != null)
+ } finally {
+ catalog.dropTable(ident)
+ }
+ }
+ }
+ }
}