You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "dwsmith1983 (via GitHub)" <gi...@apache.org> on 2026/08/15 12:48:14 UTC
[PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
dwsmith1983 opened a new pull request, #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365
## Which issue does this PR close?
Part of #174 (Explore integration with Delta Lake). It does not close #174, that issue also tracks writes, CDF, and broader integration; this PR delivers the native read path.
**Supersedes** two earlier efforts, and deliberately builds on both (both given co-authored by since ideas were learned and borrowed):
- #4366 (@schenksj) — the delta-kernel-rs-based contrib. This PR keeps its contrib module shape, build gating, decline-gate philosophy, and test scenario catalog, but replaces the kernel data-read path: kernel's parquet reader has no page-index pruning, and its planning duplicates work delta-spark has already done by the time the DSv1 scan reaches Comet.
- #4669 (@adityavaish) — the minimal plain-table native scan. This PR takes the same core insight (reuse Comet's existing native parquet scan; no kernel needed) and extends it to deletion vectors, column mapping, DPP, and the full Spark matrix.
## Rationale for this change
Comet currently falls back to Spark's reader for all Delta tables (`isFileFormatSupported` requires exact `ParquetFileFormat`, and `DeltaParquetFileFormat` is a subclass). That forfeits native execution and all of Comet's parquet pruning on one of the most common table formats.
Key observation: delta-spark has already done log replay, snapshot resolution, time travel, and partition pruning by the time `CometScanRule` sees the `FileSourceScanExec`. So no Delta planning is needed on the native side at all, the scan can route through the exact same DataFusion `ParquetSource` path as `CometNativeScanExec`, inheriting row-group stats pruning, page-index pruning (#5142), and filter pushdown (#4722) for free. The only genuinely Delta-specific native code is deletion-vector decoding: DV bitmaps are decoded into per-file `ParquetAccessPlan`s, which DataFusion *intersects* with page-index pruning, so deleted rows are skipped in-scan and DV skips compose with page skips.
Local benchmark (20M rows, selective predicate): 1.44x faster than stock Spark 9.4% of bytes read; DV tables at time parity with in-scan DV application.
## What changes are included in this PR?
- **`contrib/delta/`** new Maven module behind a `-Pdelta` profile: scan rule, decline gates, serde, `CometDeltaNativeScanExec` (split-mode partition serialization, DPP via derived scan helper), ServiceLoader registrations, differential test suites, Delta own-suite regression harness, benchmark script.
- **Core (Delta-free, generic)** new `CometScanRuleExtension` SPI + ServiceLoader hook at the top of `transformV1Scan`; `CometNativeScan.convert body extracted into reusable `buildNativeScanCommon`.
- **Native** `delta` cargo feature: `DeltaScan` proto + planner arm delegati the shared parquet scan builder; `delta_dv.rs` for DV blob unframing (CRC verified), roaring decode (portable + native magic), and access-plan construction.
- **Supported** plain/partitioned tables, deletion vectors (inline + on-disk), column mapping name mode, time travel, checkpoints, OPTIMIZE'd tables, schema evolution + defaults, DPP, INT96, special-character paths.
- **Declines (fall back to Spark, tagged for EXPLAIN)** CDC reads, column mapping id mode, row-index-consuming plans, unknown reader features, generated columns, encryption, `input_file_name()`.
- **Spark matrix**: 3.5 / 4.0 / 4.1 (Delta 3.3.2 / 4.0.1 / 4.3.1 — Delta 4.1.0/4.2.0 are binary-incompatible with Spark 4.1.3). Spark 3.4 declines (Delta 2.4 needs shims); spark-4.2 profile is dormant until Delta ships support.
## How are these changes tested?
- 36-test differential suite (comet-on, comet-off) covering the DV, column mapping, DPP, schema evolution, and decline-gate matrices green on Spark 3.5/4.0/4.1 cells. Pruning is asserted as hard metrics checks (`page_index_rows_pruned > 0`, `row_groups_pruned_statistics > 0`), not benchmark notes.
- Delta's own test suites run with Comet injected (`dev/run-delta-regression.s DeletionVectorsSuite 29/29, TimeTravel/ColumnMapping/DeleteSQL/UpdateSQL 197/197, MergeIntoSQLSuite 664/665 (the one failure is a scan-telemetry count assertio plan-shape artifact; data assertions pass).
- Rust unit tests for DV unframing/decode/access-plan edge cases and the feature-off error path.
- A DML repro suite proving DELETE writes DVs (not rewrites) under the claimed scan, in both `useMetadataRowIndex` modes.
---
Co-authored-by: Scott Schenkein <sc...@yahoo.com>
Co-authored-by: Aditya Vaish <ad...@microsoft.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3861298389
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,981 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.ListBuffer
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands. Anything else on the protocol declines the
+ * table. Note `deletionVectors` and `columnMapping` are declined separately (below) so their
+ * fallback reasons are specific.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: this is
+ * the first gate on every V1 scan and must stay inert when delta-spark is absent from the
+ * classpath, where `classOf[DeltaParquetFileFormat]` would raise NoClassDefFoundError inside
+ * CometScanRule and take down every parquet scan in the session. A name match proves
+ * delta-spark is present, so Delta types used past this gate are safe.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Returns the first reason this Delta scan cannot go native, or None when claimable. Only
+ * called when [[isDeltaScan]] is true. `scanHelper` is the same [[CometScanExec]] the caller
+ * builds to drive [[CometDeltaNativeScan.convert]] on a claim, reused here to resolve the
+ * scan's selected files for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Descriptor deserialization is expensive; hoisted once so it runs at most once per claim
+ // attempt. `lazy` since most scans are not DV-shaped and gates that return earlier should not
+ // pay for it.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported (the same type checker core's own
+ // FileSourceScanExec path runs) so scan-time type gates -- notably the default-on
+ // COMET_PARQUET_UNSIGNED_SMALL_INT_CHECK safety fallback for ShortType, plus the
+ // collation and shredded-variant-struct gates -- apply identically whether the scan is
+ // claimed through core or through this contrib. A pure in-memory schema check with no
+ // file or store I/O, so it runs first, ahead of every other gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated.
+ val cmMode = metadata.columnMappingMode.name
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder uses the
+ // required schema verbatim as output: name-sensitive expressions (e.g. to_json) would leak
+ // physical names into results. Needs a rename adapter for the logical schema; decline until
+ // then.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bound native's memory for expanded DV row selectors (delta_dv.rs), bounded above by
+ // 2*cardinality + #row-groups; the descriptor's cardinality is a sound, pessimistic upper
+ // bound. The conf below makes an over-pessimistic decline recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities; the native builder resolves the
+ // scan's ObjectStoreUrl from the FIRST selected file only, so a later file under a different
+ // store would silently read through the wrong handle. Force file listing (scanHelper is
+ // already built for the claim path) and decline rather than risk it.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // Credentials that resolve ONLY through a Hadoop credential provider (JCEKS et al.) are
+ // invisible to the plain-conf extraction forwarded to the native S3 client; reuses hadoopConf
+ // from the encryption gate above.
+ val credentialReason = credentialAliasReason(hadoopConf, dataFileUris ++ dvUris)
+ if (credentialReason.isDefined) {
+ return credentialReason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, deserialized once and
+ * normalized to absolute on-disk paths via `copyWithAbsolutePath`. Returns `Seq.empty` for the
+ * plain shape ([[CometDeltaNativeScan.isDvShape]] false). Shared by the DV cardinality gate and
+ * [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset. Hoisted so [[declineReason]]'s
+ * root-path and selected-file scheme gates share one parsed set.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged: such a URI cannot come from a Hadoop-backed source. Factored out of
+ * [[declineReason]]'s two scheme gates so both share one predicate, unit-testable without a
+ * Spark session.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt). See [[declineReason]]'s call site for ordering vs. the authority gates
+ * below.
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme plus the raw
+ * authority -- userinfo, host, port -- lowercased so `S3A://Bucket:1234` and
+ * `s3a://bucket:1234` collapse), or `None` when they share one. `file://` paths carry no
+ * authority, so local scans across many directories are unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the individually-parsed host/port/userinfo fields: `getAuthority` already
+ * includes userinfo, so containers on the same storage account don't collapse together; and
+ * `getHost` (and `getUserInfo`/`getPort`) return `null` for the whole authority when it doesn't
+ * conform to RFC 3986's `reg-name` syntax (e.g. an underscore in a GCS bucket name,
+ * `gs://my_bucket`), silently collapsing distinct buckets into one empty-host key. A `null`
+ * authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`: as with [[uriAuthority]], the structured getters return
+ * `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an underscore
+ * in a GCS bucket name), hiding a real userinfo component. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme` + `://` + a literal `***` + `@host[:port]`, for
+ * embedding in a decline reason. NEVER interpolate `uri.getAuthority` or [[uriUserInfo]]
+ * directly into a reason string: doing so would leak credentials embedded as URI userinfo (e.g.
+ * `s3a://AKIA...:secret@bucket`) into the SQL plan's explain output, fallback-reason logging,
+ * or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle -- normally a missing-object
+ * error, but silently wrong data if a same-named object exists under both.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted by [[credentialAliasReason]] below. `hadoop-aws` is
+ * NOT on this module's runtime classpath (`spark-hadoop-cloud` is test-scope only), so
+ * `org.apache.hadoop.fs.s3a.Constants` must never be referenced here -- doing so would raise
+ * `NoClassDefFoundError` and take down every Delta scan in a session with no S3 dependency at
+ * all, not just S3 ones.
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key
+ * (`fs.s3a.bucket.B.<base key minus its "fs.s3a." prefix>`) -- see [[s3aCredentialAliases]] for
+ * why both must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ /**
+ * The aliases the native S3 credentials provider chain tries per bucket (short bucket form,
+ * then global), PLUS the LONG bucket form Hadoop's `S3AUtils#lookupPassword` also consults.
+ * Hadoop resolves long before short before global, keeping a non-empty long value over the
+ * short/global ones (`S3AUtils#getPassword` returns a non-empty `val` unchanged); native reads
+ * only short+global. So a JCEKS entry set ONLY under the long alias is resolved by Hadoop but
+ * invisible to native -- a shadowed value under ANY alias here means the caller must decline.
+ * List order only affects which alias name appears in the reason.
+ */
+ private def s3aCredentialAliases(bucket: String): Seq[String] =
+ Seq(
+ s"fs.s3a.bucket.$bucket.fs.s3a.access.key",
+ s"fs.s3a.bucket.$bucket.fs.s3a.secret.key",
+ s"fs.s3a.bucket.$bucket.fs.s3a.session.token",
+ s"fs.s3a.bucket.$bucket.access.key",
+ s"fs.s3a.bucket.$bucket.secret.key",
+ s"fs.s3a.bucket.$bucket.session.token",
+ "fs.s3a.access.key",
+ "fs.s3a.secret.key",
+ "fs.s3a.session.token")
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority (host, minus any
+ * userinfo or port), or `None` when `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority
+ * manually (last `@`, then last `:`) rather than using `URI#getHost`: the same RFC 3986
+ * `reg-name` pitfall as [[uriAuthority]] applies (an underscore, valid in an S3 bucket name,
+ * makes `getHost` return `null` for the whole authority).
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ /**
+ * The three per-bucket S3A credential base keys the native S3 client's `get_config` (`s3.rs`)
+ * resolves: short bucket key first, then global (native never reads the long bucket key).
+ */
+ private val PlainCredentialBaseKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket`. Used by the credential-provider-class gates below, which read general S3A options
+ * the same way native does.
+ */
+ private def effectiveOptionValue(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ private def plainLongFormCredentialDivergenceReason(bucket: String, longKey: String): String =
+ "Native Delta scan cannot forward long-form bucket credentials for " +
+ s"$bucket ($longKey is set but the native S3 client only reads the short bucket and " +
+ "global keys, so its credentials would differ from Hadoop's)"
+
+ /**
+ * Zero-I/O plain-value divergence check (see [[s3aCredentialAliases]] for the long-before-short
+ * precedence). Native's `get_config` never reads the long form, so it can silently diverge from
+ * Hadoop's effective value. Declines when the long form is set and the two sides disagree
+ * (short set to the SAME value as long passes). Never interpolates a resolved value, only the
+ * key name.
+ *
+ * This applies to credentials ONLY. Hadoop resolves them via `S3AUtils#lookupPassword`, where
+ * the long bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) WINS when set -- a real, silent
+ * divergence from native, which never reads it. Every OTHER `fs.s3a` option instead flows
+ * through `S3AUtils#propagateBucketOptions`, which strips only ONE `fs.s3a.bucket.B.` prefix
+ * layer: a long-form key there folds into `fs.s3a.fs.s3a.<key>`, a key nothing else in Hadoop
+ * ever reads, so the long form is inert for those options -- Hadoop's own effective value
+ * already reduces to short.orElse(global), matching native. No gate is needed (or present) for
+ * that wider set; see DeltaScanContribSuite's pinned control test for the long-form endpoint
+ * case.
+ */
+ private def plainLongFormCredentialReason(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ PlainCredentialBaseKeys.foldLeft(Option.empty[String]) { (declined, baseKey) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val long = plainValue(hadoopConf, longKey)
+ if (long.isEmpty) {
+ None
+ } else {
+ val short = plainValue(hadoopConf, shortKey)
+ val global = plainValue(hadoopConf, baseKey)
+ val hadoopEffective = long.orElse(short).orElse(global)
+ val nativeEffective = short.orElse(global)
+ if (hadoopEffective != nativeEffective) {
+ Some(plainLongFormCredentialDivergenceReason(bucket, longKey))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * String-literal mirror of every credential-provider class name s3.rs's
+ * `build_aws_credential_provider_metadata` and `is_anonymous_credential_provider` recognize
+ * (Hadoop S3A names plus AWS SDK v1/v2 names). `hadoop-aws` is NOT on this module's runtime
+ * classpath (see the note above [[HadoopCredentialProviderPathKey]]), so these are string
+ * literals, never `org.apache.hadoop.fs.s3a.auth.*` or AWS SDK provider `classOf` references.
+ */
+ private val SupportedCredentialProviderClasses: Set[String] = Set(
+ "org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider",
+ "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider",
+ "com.amazonaws.auth.ContainerCredentialsProvider",
+ "com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper",
+ "software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider",
+ "com.amazonaws.auth.InstanceProfileCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider",
+ "com.amazonaws.auth.EnvironmentVariableCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider",
+ "com.amazonaws.auth.WebIdentityTokenCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider",
+ "com.amazonaws.auth.profile.ProfileCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider",
+ "com.amazonaws.auth.AnonymousAWSCredentials")
+
+ private val AnonymousCredentialProviderClasses: Set[String] = Set(
+ "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider",
+ "com.amazonaws.auth.AnonymousAWSCredentials")
+
+ private val HadoopAssumedRoleProviderClass =
+ "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider"
+
+ private val AwsCredentialsProviderKey = "fs.s3a.aws.credentials.provider"
+ private val AssumedRoleCredentialsProviderKey = "fs.s3a.assumed.role.credentials.provider"
+
+ /**
+ * Splits a Hadoop-style comma-separated credential-provider-class list the same way s3.rs's
+ * `parse_credential_provider_names` does: split on comma, trim each entry, drop empties.
+ */
+ private def parseProviderClassNames(value: String): Seq[String] =
+ value.split(",").map(_.trim).filter(_.nonEmpty).toSeq
+
+ private def unsupportedProviderReason(bucket: String, key: String, className: String): String =
+ s"Native Delta scan does not support the credential provider class $className " +
+ s"configured via $key for $bucket (the native S3 client only supports a fixed set of " +
+ "provider classes; an unsupported class would fail at scan execution time, after the " +
+ "scan was already claimed, rather than at planning time)"
+
+ private def mixedAnonymousProviderReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key for $bucket naming an anonymous credential " +
+ "provider together with any other provider (the native S3 client rejects this " +
+ "combination at scan execution time)"
+
+ private def anonymousAssumedRoleProviderReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support an anonymous credential provider in $key for " +
+ s"$bucket (the native S3 client does not allow an anonymous provider as the base " +
+ "credentials for an assumed-role chain)"
+
+ private def unsupportedProviderNameReason(
+ bucket: String,
+ key: String,
+ names: Seq[String]): Option[String] =
+ names
+ .find(name => !SupportedCredentialProviderClasses.contains(name))
+ .map(unsupportedProviderReason(bucket, key, _))
+
+ /**
+ * Decline reason when `bucket`'s effective (short-bucket-then-global; see
+ * [[effectiveOptionValue]]) `assumed.role.credentials.provider` names an unsupported class, or
+ * an anonymous one (native's `build_assume_role_credential_provider_metadata` rejects ANY
+ * anonymous entry here, not just a mix). Unset defaults to native's own hardcoded
+ * `[SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider]` fallback, both
+ * always supported, so `None` is safe.
+ */
+ private def assumedRoleProviderClassReason(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ effectiveOptionValue(hadoopConf, bucket, AssumedRoleCredentialsProviderKey) match {
+ case None => None
+ case Some(value) =>
+ val names = parseProviderClassNames(value)
+ unsupportedProviderNameReason(bucket, AssumedRoleCredentialsProviderKey, names).orElse {
+ if (names.exists(AnonymousCredentialProviderClasses.contains)) {
+ Some(anonymousAssumedRoleProviderReason(bucket, AssumedRoleCredentialsProviderKey))
+ } else {
+ None
+ }
+ }
+ }
+ }
+
+ /**
+ * Decline reason when `bucket`'s effective `aws.credentials.provider` names a class native's
+ * `build_aws_credential_provider_metadata` (s3.rs) does not recognize, mixes an anonymous
+ * provider with any other provider (native's `build_credential_provider` rejects this
+ * combination), or -- when `AssumedRoleCredentialProvider` is among the names -- its
+ * `assumed.role.credentials.provider` sub-chain has the same problem. An unset/empty value is
+ * fine: native falls back to its own default AWS SDK provider chain. Checked so an unsupported
+ * class or invalid combination declines at planning time instead of erroring during scan
+ * execution, after the scan was already claimed.
+ */
+ private def providerClassReason(hadoopConf: Configuration, bucket: String): Option[String] = {
+ effectiveOptionValue(hadoopConf, bucket, AwsCredentialsProviderKey).flatMap { value =>
+ val names = parseProviderClassNames(value)
+ unsupportedProviderNameReason(bucket, AwsCredentialsProviderKey, names)
+ .orElse {
+ if (names.length > 1 && names.exists(AnonymousCredentialProviderClasses.contains)) {
+ Some(mixedAnonymousProviderReason(bucket, AwsCredentialsProviderKey))
+ } else {
+ None
+ }
+ }
+ .orElse {
+ if (names.contains(HadoopAssumedRoleProviderClass)) {
+ assumedRoleProviderClassReason(hadoopConf, bucket)
+ } else {
+ None
+ }
+ }
+ }
+ }
+
+ /**
+ * Returns the first reason any bucket among `uris` names an unsupported (or invalidly combined)
+ * credential-provider class, or `None` when every bucket's provider configuration is one native
+ * can build. See [[providerClassReason]].
+ */
+ private[delta] def providerClassGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) declined else providerClassReason(hadoopConf, bucket)
+ }
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def shadowedCredentialAliasReason(bucket: String, alias: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($alias resolves through $HadoopCredentialProviderPathKey but is not present " +
+ "as a plain configuration value, so the native S3 client would have no credentials)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * Compares each [[s3aCredentialAliases]] alias's plain `Configuration#get` value against
+ * `Configuration#getPassword` (providers first, plain conf as fallback); a resolved value that
+ * differs means the alias is invisible or wrong to native's plain-conf extraction, so decline.
+ * Only called once the GLOBAL provider path is established as the sole path for `bucket`.
+ *
+ * Runs inside try/catch: `getPassword` performs real keystore I/O, and a corrupt or unreadable
+ * store must decline this bucket rather than abort planning for the whole session.
+ */
+ private def verifyGlobalProviderAliases(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ try {
+ s3aCredentialAliases(bucket).foldLeft(Option.empty[String]) { (declined, alias) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ val resolved =
+ Option(hadoopConf.getPassword(alias)).map(new String(_)).filter(_.nonEmpty)
+ resolved match {
+ case Some(value) if !Option(hadoopConf.get(alias)).contains(value) =>
+ Some(shadowedCredentialAliasReason(bucket, alias))
+ case _ => None
+ }
+ }
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableCredentialProviderReason(bucket, e))
+ }
+ }
+
+ /**
+ * The decline reason for `bucket` alone, or `None` when its credentials are safe to forward.
+ * `globalPathSet`/`s3aPathSet` are hoisted by the caller since they don't vary per bucket.
+ *
+ * Checked FIRST, regardless of provider-path config: [[plainLongFormCredentialReason]] is a
+ * zero-I/O, provider-unrelated check that is decisive on its own.
+ *
+ * Zero-I/O precheck: `None` immediately when none of the four provider-path keys are set -- a
+ * config-map lookup only, no keystore access.
+ *
+ * Arm A: the S3A-scoped provider-path keys point S3A's own resolution at a provider
+ * `Configuration#getPassword` does not consult, so any being set declines with no keystore
+ * read.
+ *
+ * Arm B: only the global path is set, which `getPassword` DOES consult; delegates to
+ * [[verifyGlobalProviderAliases]].
+ */
+ private def bucketCredentialAliasReason(
+ hadoopConf: Configuration,
+ bucket: String,
+ globalPathSet: Boolean,
+ s3aPathSet: Boolean): Option[String] = {
+ plainLongFormCredentialReason(hadoopConf, bucket).orElse {
+ val bucketPathKey = s3aBucketProviderPathKey(bucket)
+ val bucketLongPathKey = s3aBucketLongProviderPathKey(bucket)
+ val bucketPathSet = nonEmptyConf(hadoopConf, bucketPathKey)
+ val bucketLongPathSet = nonEmptyConf(hadoopConf, bucketLongPathKey)
+ if (!globalPathSet && !s3aPathSet && !bucketPathSet && !bucketLongPathSet) {
+ None
+ } else if (s3aPathSet || bucketPathSet || bucketLongPathSet) {
+ val offendingKey =
+ if (s3aPathSet) S3aCredentialProviderPathKey
+ else if (bucketPathSet) bucketPathKey
+ else bucketLongPathKey
+ Some(s3aScopedProviderPathReason(bucket, offendingKey))
+ } else {
+ verifyGlobalProviderAliases(hadoopConf, bucket)
+ }
+ }
+ }
+
+ /**
+ * Returns the first reason a native S3 scan cannot faithfully forward this table's Hadoop
+ * credentials, or `None` when claimable. `uris` is the scan's data-file and DV URIs; only
+ * `s3`/`s3a` authorities matter here (ABFS/WASB mooted by the userinfo gate, GCS out of scope).
+ * `Configuration#getPassword` checks every provider FIRST, falling back to plain conf only when
+ * unset, so a keystore-only or shadowing alias is invisible/wrong to native's plain-conf
+ * extraction; when in doubt, decline. Never interpolates a resolved value, only key names.
+ */
+ private[delta] def credentialAliasReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ if (buckets.isEmpty) {
+ return None
Review Comment:
Added the conservative gate: any scan whose data files or DVs touch gs:// declines when a fs.gs.auth.* key is set, since the native resolver forwards no GCS options. ADC-only configs still claim because both engines resolve credentials the same way there. Tests cover your mixed local-data plus gs DV sidecar shape and the scheme scoping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3858563591
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
Review Comment:
**[P2] Keep mapped partition names in the physical namespace**
This mixes physical data names with logical partition names. A valid table history is: write `(a BIGINT, p BIGINT)` partitioned by `p`, enable name mapping, rename data column `a` to `b`, then rename partition column `p` to `a`. Delta preserves the old physical names, so this call now supplies both a physical data field named `a` and a partition constant named `a`. DataFusion's name-based partition rewrite replaces the data projection with that constant. On this head with Spark 4.0.4 / Delta 4.0.1 and an asserted native Delta scan, `SELECT b, a` returned `(100,100), (100,100)` instead of Spark's `(1,100), (2,100)`. Could we physicalize the partition schema consistently here and in the DV builder, or decline such ambiguous scans, and cover this upgrade-plus-rename history?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3831648406
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,587 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::ParquetAccessPlan;
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::ObjectStoreExt;
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::PageIndexPolicy;
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::parquet::parquet_support::prepare_object_store_with_configs;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
Review Comment:
**[P2] Also account for the reader's combined-selection allocation**
Construction admission and the initial reader clone are now covered at `bc98657f`. One later allocation is still outside the reservation: DataFusion 54.1's `build_stream` calls `ParquetAccessPlan::prepare` -> `into_overall_row_selection`, which collects another `RowSelection` while the attached original and the consumed clone's backing vector remain live. The resize at `delta_dv.rs:504-507` has already reduced the reservation to twice the retained selector bytes by then.
I verified this using the unchanged current attachment code and the real locked DataFusion/Parquet conversion. For one 2,000,000-row group with exactly 1,000,000 alternating deletions (permitted by the default cap):
- Construction reserves 96,000,048 bytes, then attachment reduces the reservation to 64,000,000 bytes.
- The attached selectors plus reader-normalization allocations peak at 97,554,457 bytes and retain 65,554,432 bytes afterward, while the reservation remains 64,000,000.
These are allocator-requested bytes, not RSS or a reproduced executor OOM; the dependency conversion was invoked directly, with its production call path and allocation lifetimes checked in source, rather than running a complete Comet scan. Reservation release still works. Could we account for normalization and vector capacity before it allocates, or transfer ownership to avoid the extra buffer? Simply changing the steady-state factor to 3 would still fall below this measured peak. This is a remaining part of the existing memory P2, not a separate finding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4048085990
##########
dev/ci/compute-changes.py:
##########
@@ -387,6 +387,26 @@
".mvn/**",
"mvnw",
],
+ "delta": [
Review Comment:
> Could you put the measured runner-minutes in the description? If it lands where I think it does, it may be worth trimming to one Spark profile in the queue with the other two on the label or nightly
Trimmed in e320d96f5 to the Iceberg shape: Spark 3.5 runs in the queue, 4.0 and 4.1 run nightly, and `run-delta-tests` opts a pull request into all three. The reusable workflow takes the profile as an input, and the MinIO, feature-off and dev-script jobs run once, on the 3.5 call. Measured minutes have to come from the labelled run, since the suites have not executed in CI on this PR yet; your 8m6s of test time for the 3.5 leg matches what I see locally for the three suites, and I will put the runner minutes in the description once that run exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3869537329
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme (finding 8).
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
Review Comment:
The variable-reference case declines now too. The native side receives raw values (the extraction iterates the conf without substitution), so any credential alias whose raw value differs from its substituted value falls back to Spark, wrapped in the same guard the keystore checks use. Your ${review.access} shape declines naming only the key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5377257164
On the macOS scans failure: pulled the hs_err from the run artifact. The crashing thread is a native thread (not a Java thread) that was exiting: the stack is pthread_start into pthread_exit into pthread TSD cleanup, then a jump through a corrupted destructor slot whose value is ASCII string bytes, at 119s elapsed, immediately after ParquetReadFromFakeHadoopFsSuite, the only suite in the group that exercises the libhdfs bridge and its JNI-attached native threads. The Delta code in this PR is structurally unreachable in those suites (native side is dispatch-gated on an operator those plans never emit, and the contrib jar is not on that build's classpath), and the Linux scans group passed on the same commit. My working hypothesis is a teardown race in the libhdfs bridge or a runner flake rather than anything this PR executes; the falsifying experiment would be rebuilding the dylib without the delta feature and re-running, since the same crash would exonerate it by construction. Coul
d someone re-run the job? Happy to file the hs_err as an issue either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5618418055
> point layer 1 at `cargo tree -p datafusion-comet` with no flag, and reword the header and the `.github/workflows/delta_build_gate.yml` comment to the invariant that actually holds
Done. Layer 1 now runs `cargo tree -p datafusion-comet` as the primary check and `--no-default-features` as a second case through the same helper, each with the anti-vacuous root-crate guard. The script header, the section headers and the workflow comment now state the invariant the checks prove: the kernel-backed `comet-contrib-delta` crate stays out of every shipped build, while the default-on `delta` feature is deliberately in.
> Pinning the default-on surface near the 82 KB you measured would make that layer say something the grep cannot drift away from.
The symbol layer now asserts three things on the default library: zero contrib/kernel symbols, at least one symbol from the `delta` feature's own modules (`delta_dv`, `delta_scan`, `delta_spark_scan`), and, where `nm -S` reports sizes, their total below a cap. In a Linux debug build that footprint is 84 KB across 372 symbols, 63 KB of it the decoder in `delta_dv` and the rest the planner arm, so the cap is 512 KB: room for toolchain drift, none for a kernel. Mach-O `nm` reports every size as zero, so on macOS the layer checks presence and skips the cap; CI runs on Linux where it applies. A default build that dropped the feature fails the presence check, and one that pulled something kernel-sized in through the feature fails the cap, which is what the old grep could not say.
The size comparison between the default and contrib libraries is gone from the same section. On an unstripped debug library the contrib code is a few hundred kilobytes against 1.4 GB, so layout noise decided that assertion, and it has failed on changes that never touched the contrib crate. The sizes are still printed, and `nm` is now required rather than silently skipped, since the symbol table is the only evidence the section relies on.
> it should go in the `Cargo.toml` comment next to the #5411 pointer
Added to the `delta` feature comment: `roaring` and `crc32fast` are already in the default tree through iceberg and the shuffle crate, so the feature adds no crate and only promotes two transitive dependencies to direct ones.
`dev/verify-contrib-delta-gate.sh` passes end to end on this head on macOS, and the symbol functions run against a Linux debug library report 0 contrib symbols and the 84 KB footprint under the cap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4006936471
##########
pom.xml:
##########
@@ -95,6 +95,11 @@ under the License.
<guava.version>33.2.1-jre</guava.version>
<testcontainers.version>1.21.4</testcontainers.version>
<amazon-awssdk-v2.version>2.31.51</amazon-awssdk-v2.version>
+ <!-- Delta Lake pairing for the contrib/delta-spark module (-Pdelta). Each Spark profile
+ overrides these with its matching Delta release; the defaults match the default
+ spark-4.1 profile. Delta 2.x ships as artifact delta-core, 3.x/4.x as delta-spark. -->
+ <delta.artifact>delta-spark</delta.artifact>
+ <delta.version>4.3.1</delta.version>
Review Comment:
> Could the first declaration and its comment go, or the two contribs use distinct property names so a reader does not have to work out which wins?
The first declaration and its comment are gone; the Spark profiles set the value and the top-level default is the remaining one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4006991954
##########
native/core/src/parquet/datetime_rebase.rs:
##########
@@ -0,0 +1,2662 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Per-file datetime calendar-rebase handling for the parquet scan.
+//!
+//! Spark 2.4 and earlier wrote dates and timestamps in the hybrid Julian + Gregorian calendar;
+//! Spark 3.0+ uses the proleptic Gregorian calendar and records the calendar policy of every
+//! file it writes in the parquet footer's key-value metadata (`org.apache.spark.version`,
+//! `org.apache.spark.legacyDateTime`, `org.apache.spark.legacyINT96`,
+//! `org.apache.spark.timeZone`). Spark's reader resolves the rebase policy from EACH FILE's
+//! writer metadata (`DataSourceUtils.datetimeRebaseSpec` / `int96RebaseSpec`) -- the session's
+//! `spark.sql.parquet.datetimeRebaseModeInRead` conf only applies to files whose metadata does
+//! not decide the policy on its own -- so a reader that ignores the metadata silently returns
+//! values shifted by up to ten days for dates before 1582-10-15 (e.g. `1500-01-01` reads as
+//! `1500-01-10`).
+//!
+//! This module mirrors that per-file resolution: [`resolve_file_rebase_policies`] computes the
+//! date / INT64-timestamp / INT96-timestamp policies from a file's arrow schema metadata (the
+//! parquet key-value pairs survive the parquet -> arrow schema conversion), and
+//! [`wrap_datetime_rebase`] wraps the per-file rewritten expressions' column references in a
+//! [`SparkDatetimeRebaseExpr`] that rebases values exactly where that is possible without the
+//! JVM's historical timezone tables (dates always; timestamps for a fixed UTC writer zone) and
+//! refuses -- rather than silently corrupting -- ancient values it cannot rebase. Nested
+//! columns are rebuilt leaf by leaf (struct / list / map / fixed-size list / dictionary), each
+//! leaf under its own policy, with nulls and offsets preserved. Modern values are always the
+//! identity under every policy: from 1582-10-15 onward for dates, and from
+//! [`LAST_SWITCH_JULIAN_TS_SECONDS`] (1900-01-01T00:00:00Z, Spark's
+//! `RebaseDateTime.lastSwitchJulianTs`) onward for timestamps.
+//!
+//! Spark applies `datetimeRebaseSpec` to INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns
+//! and `int96RebaseSpec` to INT96 columns. The two physical types are indistinguishable in the
+//! arrow schema DataFusion hands the expression adapter (both surface as `Timestamp(us, "UTC")`
+//! after INT96 coercion), so Comet's parquet reader factory stamps the file's INT96 leaf
+//! ordinals -- taken from the parquet footer's own `SchemaDescriptor` -- into the key-value
+//! metadata under [`INT96_LEAVES_METADATA_KEY`] before the arrow schema is derived (see
+//! [`stamp_int96_leaves`] and `eager_page_index_reader_factory.rs`), and the adapter attributes
+//! every timestamp leaf to its spec from that stamp. Without a stamp, the two specs are merged:
+//! agreement decides, disagreement degrades to [`RebasePolicy::CheckAncient`].
+//!
+//! The wrapper sits BENEATH the schema adapter's nested narrowing (the struct -> struct convert
+//! that keeps only the requested children), which is what keeps those ordinals physical -- but
+//! it means the wrapper sees every physical child, requested or not. Spark only ever decodes
+//! the requested nested schema, so [`FileRebasePolicies::restrict_to_requested`] marks the
+//! physical leaves the narrowing drops as the identity: an unrequested ancient `s.ts` never
+//! blocks `select s.d`, exactly as in Spark.
+//!
+//! Currently only enabled by the Delta scan arms via
+//! `SparkParquetOptions::rebase_from_file_metadata`, which also carries the session read modes
+//! ([`SessionRebaseModes`], forwarded from the JVM) that decide the policy for files without
+//! Spark writer metadata; the plain NativeScan keeps its documented no-rebase behavior (see
+//! the compatibility guide and issue #5010).
+
+use std::collections::HashMap;
+use std::fmt::{self, Display};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Date32Array, FixedSizeListArray, GenericListArray, MapArray,
+ OffsetSizeTrait, PrimitiveArray, RecordBatch, StructArray,
+};
+use arrow::datatypes::{
+ ArrowTimestampType, DataType, Date32Type, FieldRef, Schema, SchemaRef, TimeUnit,
+ TimestampMicrosecondType, TimestampMillisecondType, TimestampNanosecondType,
+ TimestampSecondType,
+};
+use arrow::error::ArrowError;
+use datafusion::common::tree_node::{Transformed, TreeNode};
+use datafusion::common::{DataFusionError, Result as DataFusionResult};
+use datafusion::physical_expr::expressions::Column;
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::ColumnarValue;
+use parquet::basic::Type as ParquetPhysicalType;
+use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaData};
+use parquet::schema::types::SchemaDescriptor;
+
+use super::name_fold::fold_names;
+use super::schema_adapter::parse_field_id;
+
+/// Footer key naming the Spark release that wrote the file; absent for non-Spark writers.
+const SPARK_VERSION_METADATA_KEY: &str = "org.apache.spark.version";
+/// Present (empty value) when the file's dates and INT64 timestamps were written with
+/// `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_DATETIME_KEY: &str = "org.apache.spark.legacyDateTime";
+/// Present (empty value) when the file's INT96 timestamps were written with
+/// `spark.sql.parquet.int96RebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_INT96_KEY: &str = "org.apache.spark.legacyINT96";
+/// The writer session's time zone, stamped alongside either legacy flag.
+const SPARK_TIMEZONE_KEY: &str = "org.apache.spark.timeZone";
+
+/// Key-value metadata entry Comet's parquet reader factory adds to a file's footer metadata
+/// (in memory only, never written back) so the expression adapter can tell INT96 timestamp
+/// columns from INT64 ones after both have been coerced to the same arrow type. Value:
+/// `"<leaf count>:<comma-separated INT96 leaf ordinals>"`, where leaves are the file's
+/// primitive columns in `SchemaDescriptor::columns()` order -- the same depth-first order
+/// parquet-rs assigns arrow leaves, so an arrow-side depth-first walk lines up with it. The
+/// leaf count lets the reader detect a stamp that does not describe the schema it is paired
+/// with (see [`Int96Attribution::from_schema`]).
+pub(crate) const INT96_LEAVES_METADATA_KEY: &str = "comet.int96_leaf_columns";
+
+/// Day of the Gregorian cutover (1582-10-15) as days since the epoch; rebasing is the identity
+/// from this day onward. Same value as Spark's `RebaseDateTime.lastSwitchJulianDay`.
+const LAST_SWITCH_JULIAN_DAY: i32 = -141427;
+
+/// Spark's `RebaseDateTime.lastSwitchJulianTs` (and `lastSwitchGregorianTs`) in seconds since
+/// the epoch: 1900-01-01T00:00:00Z. Spark derives it as the latest switch instant across every
+/// zone in its `julian-gregorian-rebase-micros.json` table (`getLastSwitchTs`, which also
+/// asserts the calendars' difference is zero for every zone from then on): most zones ran on
+/// local mean time before 1900, so the last instant at which rebasing changes a value in ANY
+/// zone is 1900-01-01T00:00:00Z, not the 1582 cutover. `createTimestampRebaseFuncInRead`
+/// under `EXCEPTION` throws exactly for `micros < lastSwitchJulianTs` (after converting
+/// `TIMESTAMP_MILLIS` to micros), and `rebaseJulianToGregorianMicros` is the identity from it
+/// onward in every zone. The value is in seconds so it scales exactly to any timestamp unit.
+pub(crate) const LAST_SWITCH_JULIAN_TS_SECONDS: i64 = -2_208_988_800;
+
+/// The per-century differences between the Julian and proleptic Gregorian calendars, and the
+/// Julian-calendar switch days at which each difference starts to apply. Copied verbatim from
+/// Spark's `RebaseDateTime.julianGregDiffs` / `julianGregDiffSwitchDay` (which Spark generated
+/// from `localRebaseJulianToGregorianDays`); `rebase_julian_to_gregorian_days` must stay
+/// value-for-value equal to Spark's `rebaseJulianToGregorianDays`.
+const JULIAN_GREG_DIFFS: [i32; 14] = [2, 1, 0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, 0];
+const JULIAN_GREG_DIFF_SWITCH_DAY: [i32; 14] = [
+ -719164, -682945, -646420, -609895, -536845, -500320, -463795, -390745, -354220, -317695,
+ -244645, -208120, -171595, -141427,
+];
+
+/// Proleptic-Gregorian days since 1970-01-01 for a nominal civil date, via Howard Hinnant's
+/// `days_from_civil`. `d` may exceed the month's length; the excess rolls into the following
+/// month exactly like `LocalDate.of(y, m, 1).plusDays(d - 1)` in Spark's
+/// `localRebaseJulianToGregorianDays` (how the non-existent proleptic date `1000-02-29`,
+/// valid in the Julian calendar, lands on `1000-03-01`).
+fn days_from_civil(y: i64, m: i64, d: i64) -> i64 {
+ let y = if m <= 2 { y - 1 } else { y };
+ let era = y.div_euclid(400);
+ let yoe = y - era * 400; // [0, 399]
+ let mp = (m + 9) % 12; // [0, 11], March = 0
+ let doy = (153 * mp + 2) / 5 + d - 1;
+ let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
+ era * 146097 + doe - 719468
+}
+
+/// Julian-calendar civil date `(year, month, day)` for a day count since 1970-01-01 that labels
+/// days in the Julian calendar (astronomical year numbering: 1 BCE is year 0). Standard
+/// Julian-day-number conversion (E.G. Richards' algorithm), exact for any day.
+fn julian_day_to_civil(days: i64) -> (i64, i64, i64) {
+ // Integer (noon) Julian Day Number of this civil day: 1970-01-01 is JDN 2440588.
+ let jdn = days + 2_440_588;
+ let f = jdn + 1401;
+ let e = 4 * f + 3;
+ let g = e.rem_euclid(1461) / 4;
+ let h = 5 * g + 2;
+ let day = h.rem_euclid(153) / 5 + 1;
+ let month = (h / 153 + 2).rem_euclid(12) + 1;
+ let year = e.div_euclid(1461) - 4716 + (14 - month) / 12;
+ (year, month, day)
+}
+
+/// Exact port of Spark's `RebaseDateTime.rebaseJulianToGregorianDays`: reinterprets a day count
+/// written in the hybrid Julian + Gregorian calendar as the proleptic Gregorian day count of the
+/// same nominal civil date. Identity for days from 1582-10-15 onward. Days before the tables'
+/// range (before Julian `0001-01-01`) take the calendar-arithmetic path, mirroring Spark's
+/// `localRebaseJulianToGregorianDays` fallback.
+pub(crate) fn rebase_julian_to_gregorian_days(days: i32) -> i32 {
+ if days < JULIAN_GREG_DIFF_SWITCH_DAY[0] {
+ let (y, m, d) = julian_day_to_civil(days as i64);
+ (days_from_civil(y, m, 1) + (d - 1)) as i32
+ } else {
+ // Spark's rebaseDays: linear search from the most recent switch day.
+ let mut i = JULIAN_GREG_DIFF_SWITCH_DAY.len();
+ loop {
+ i -= 1;
+ if i == 0 || days >= JULIAN_GREG_DIFF_SWITCH_DAY[i] {
+ break;
+ }
+ }
+ days + JULIAN_GREG_DIFFS[i]
+ }
+}
+
+/// Timezone strings from `org.apache.spark.timeZone` that denote a fixed zero-offset zone in
+/// both `java.util.TimeZone` and `java.time`. Only for these is timestamp rebasing the pure
+/// nominal-date shift [`SparkDatetimeRebaseExpr::rebase_timestamp_utc`] computes; any other (or
+/// absent) zone needs the JVM's historical timezone tables and stays on the
+/// refuse-ancient-values path.
+const UTC_EQUIVALENT_TIMEZONES: [&str; 6] = ["UTC", "Etc/UTC", "GMT", "Etc/GMT", "Z", "+00:00"];
+
+/// How the writer's session time zone (if recorded) affects timestamp rebasing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum WriterTimeZone {
+ /// A fixed zero-offset zone: rebasing reduces to the exact nominal-date shift.
+ Utc,
+ /// Any other zone, or none recorded (pre-3.0 files): ancient values cannot be rebased
+ /// without the JVM's historical timezone data.
+ OtherOrUnknown,
+}
+
+/// One session-level datetime rebase read mode (a `LegacyBehaviorPolicy` value of
+/// `spark.sql.parquet.datetimeRebaseModeInRead` / `int96RebaseModeInRead`), consulted by
+/// [`resolve_file_rebase_policies`] ONLY for files whose footer metadata does not decide the
+/// policy on its own -- exactly the `getOrElse` fallback in Spark's
+/// `DataSourceUtils.getRebaseSpec`. Files that carry `org.apache.spark.version` ignore these
+/// modes entirely, on every Spark version.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) enum RebaseReadMode {
+ /// Refuse ancient values (Spark raises `SparkUpgradeException`); maps to
+ /// [`RebasePolicy::CheckAncient`]. The default mirrors the conservative posture used
+ /// before the conf was plumbed through (and Spark 3.x's own conf default).
+ #[default]
+ Exception,
+ /// Read values as proleptic Gregorian without rebasing.
+ Corrected,
+ /// Rebase from the hybrid Julian + Gregorian calendar.
+ Legacy,
+}
+
+impl RebaseReadMode {
+ /// Parses a `LegacyBehaviorPolicy` conf value. `SQLConf` validates and upper-cases the
+ /// session conf, but a per-relation `datetimeRebaseMode` option arrives verbatim, so the
+ /// match is case-insensitive. Anything unrecognized -- including the empty string a proto
+ /// producer that predates the field sends -- falls back to [`RebaseReadMode::Exception`],
+ /// which refuses ancient values rather than silently corrupting them.
+ pub(crate) fn from_conf_value(value: &str) -> Self {
+ match value.to_ascii_uppercase().as_str() {
+ "CORRECTED" => RebaseReadMode::Corrected,
+ "LEGACY" => RebaseReadMode::Legacy,
+ _ => RebaseReadMode::Exception,
+ }
+ }
+}
+
+/// The session's effective datetime rebase read modes, one per spec class (INT64
+/// dates/timestamps vs INT96 timestamps), forwarded from the JVM at planning time.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) struct SessionRebaseModes {
+ /// `spark.sql.parquet.datetimeRebaseModeInRead` (or the relation's `datetimeRebaseMode`).
+ pub datetime: RebaseReadMode,
+ /// `spark.sql.parquet.int96RebaseModeInRead` (or the relation's `int96RebaseMode`).
+ pub int96: RebaseReadMode,
+}
+
+/// Calendar policy of one file's date or timestamp columns, resolved from writer metadata the
+/// same way Spark's `DataSourceUtils.getRebaseSpec` resolves it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum RebasePolicy {
+ /// Written in the proleptic Gregorian calendar; values pass through untouched.
+ Corrected,
+ /// Written in the hybrid Julian + Gregorian calendar; values must be rebased.
+ Legacy(WriterTimeZone),
+ /// Policy could not be pinned down (contradictory flags, or a non-Spark writer under the
+ /// `EXCEPTION` read mode): modern values -- identical under either calendar -- pass,
+ /// ancient values raise. Mirrors Spark's `EXCEPTION` behavior (`SparkUpgradeException`).
+ CheckAncient,
+}
+
+/// Which of a file's leaf columns are physically INT96, from the stamp the parquet reader
+/// factory adds under [`INT96_LEAVES_METADATA_KEY`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) enum Int96Attribution {
+ /// No stamp, or a stamp whose leaf count does not match the schema it arrived with: the
+ /// INT64 and INT96 timestamp specs cannot be told apart per column and are merged.
+ Unknown,
+ /// Sorted leaf ordinals (depth-first over the file schema's primitive columns) that are
+ /// INT96; every other timestamp leaf is INT64.
+ Known(Vec<usize>),
+}
+
+impl Int96Attribution {
+ /// Parses the stamp out of `schema`'s metadata and validates its leaf count against the
+ /// schema's own depth-first leaf count, so a stamp that does not describe this schema (a
+ /// crafted footer key, or a cached-metadata mismatch) degrades to [`Self::Unknown`].
+ fn from_schema(schema: &Schema) -> Self {
+ let Some(stamp) = schema.metadata().get(INT96_LEAVES_METADATA_KEY) else {
+ return Int96Attribution::Unknown;
+ };
+ let Some((count, ordinals)) = stamp.split_once(':') else {
+ return Int96Attribution::Unknown;
+ };
+ let schema_leaves: usize = schema
+ .fields()
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ if count.parse::<usize>().ok() != Some(schema_leaves) {
+ return Int96Attribution::Unknown;
+ }
+ let parsed: Option<Vec<usize>> = if ordinals.is_empty() {
+ Some(Vec::new())
+ } else {
+ ordinals
+ .split(',')
+ .map(|o| o.parse::<usize>().ok().filter(|o| *o < schema_leaves))
+ .collect()
+ };
+ match parsed {
+ Some(mut leaves) => {
+ leaves.sort_unstable();
+ Int96Attribution::Known(leaves)
+ }
+ None => Int96Attribution::Unknown,
+ }
+ }
+
+ /// `Some(true)` / `Some(false)` when the leaf is known to be INT96 / INT64, `None` when
+ /// the attribution is unknown.
+ fn is_int96(&self, leaf: usize) -> Option<bool> {
+ match self {
+ Int96Attribution::Unknown => None,
+ Int96Attribution::Known(leaves) => Some(leaves.binary_search(&leaf).is_ok()),
+ }
+ }
+}
+
+/// The [`INT96_LEAVES_METADATA_KEY`] value describing `schema`: its leaf count and the
+/// ordinals of its INT96 primitive columns.
+pub(crate) fn int96_leaf_stamp(schema: &SchemaDescriptor) -> String {
+ let ordinals: Vec<String> = schema
+ .columns()
+ .iter()
+ .enumerate()
+ .filter(|(_, column)| column.physical_type() == ParquetPhysicalType::INT96)
+ .map(|(ordinal, _)| ordinal.to_string())
+ .collect();
+ format!("{}:{}", schema.num_columns(), ordinals.join(","))
+}
+
+/// Returns a copy of `metadata` whose key-value metadata carries the [`int96_leaf_stamp`] of
+/// its own schema, or `None` when it already does (the common case after the first open of a
+/// file, since the caller caches the stamped copy). Any pre-existing entry under the key --
+/// a file cannot legitimately carry one -- is replaced, never trusted. Only the file-level
+/// key-value list changes; row groups and page indexes are carried over as-is. The parquet
+/// API cannot carry a file decryptor, nor `FileMetaData`'s crate-private encryption fields
+/// (encryption algorithm, footer signing key metadata), across this rebuild, so callers must
+/// not stamp opens that supply decryption properties -- and the only consumer, the Delta
+/// scan, declines every encrypted-parquet configuration before planning, so a parquet
+/// modular encryption file never reaches this path with or without those properties.
+pub(crate) fn stamp_int96_leaves(metadata: &ParquetMetaData) -> Option<ParquetMetaData> {
+ let file_metadata = metadata.file_metadata();
+ let stamp = int96_leaf_stamp(file_metadata.schema_descr());
+ let existing = file_metadata
+ .key_value_metadata()
+ .and_then(|kvs| kvs.iter().find(|kv| kv.key == INT96_LEAVES_METADATA_KEY))
+ .and_then(|kv| kv.value.as_deref());
+ if existing == Some(stamp.as_str()) {
+ return None;
+ }
+ let mut key_values: Vec<KeyValue> = file_metadata
+ .key_value_metadata()
+ .map(|kvs| {
+ kvs.iter()
+ .filter(|kv| kv.key != INT96_LEAVES_METADATA_KEY)
+ .cloned()
+ .collect()
+ })
+ .unwrap_or_default();
+ key_values.push(KeyValue::new(INT96_LEAVES_METADATA_KEY.to_string(), stamp));
+ let stamped_file_metadata = FileMetaData::new(
+ file_metadata.version(),
+ file_metadata.num_rows(),
+ file_metadata.created_by().map(str::to_string),
+ Some(key_values),
+ file_metadata.schema_descr_ptr(),
+ file_metadata.column_orders().cloned(),
+ );
+ Some(
+ ParquetMetaData::new(stamped_file_metadata, metadata.row_groups().to_vec())
+ .into_builder()
+ .set_column_index(metadata.column_index().cloned())
+ .set_offset_index(metadata.offset_index().cloned())
+ .build(),
+ )
+}
+
+/// Per-file rebase policies for the three affected column classes, plus the INT96
+/// attribution that selects between the two timestamp specs per leaf.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct FileRebasePolicies {
+ /// `DATE` columns, governed by `org.apache.spark.legacyDateTime` alone.
+ pub date: RebasePolicy,
+ /// INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns: the datetime spec (same
+ /// resolution as `date`), as Spark's `ParquetVectorUpdaterFactory` selects for INT64.
+ pub int64_timestamp: RebasePolicy,
+ /// INT96 columns: the INT96 spec (`org.apache.spark.legacyINT96`, min version 3.1.0).
+ pub int96_timestamp: RebasePolicy,
+ /// Which timestamp leaves are INT96. See [`Int96Attribution`].
+ pub int96_leaves: Int96Attribution,
+ /// Sorted depth-first leaf ordinals -- over the physical file schema, the same ordinals
+ /// `int96_leaves` uses -- that the query does not read: nested children the schema
+ /// adapter's struct narrowing drops before any value leaves the scan. Spark never decodes
+ /// them either, so their policy is the identity whatever the file's calendar. Empty until
+ /// [`Self::restrict_to_requested`] runs (every leaf requested).
+ pub unrequested_leaves: Vec<usize>,
+}
+
+impl FileRebasePolicies {
+ /// True when some policy is not the plain proleptic-Gregorian pass-through, i.e. when the
+ /// per-column wrap in [`wrap_datetime_rebase`] can install anything at all.
+ pub(crate) fn any_rebase_needed(&self) -> bool {
+ self.date != RebasePolicy::Corrected
+ || self.int64_timestamp != RebasePolicy::Corrected
+ || self.int96_timestamp != RebasePolicy::Corrected
+ }
+
+ fn is_requested(&self, leaf: usize) -> bool {
+ self.unrequested_leaves.binary_search(&leaf).is_err()
+ }
+
+ /// The policy of the `Date32` leaf at depth-first ordinal `leaf`: the file's date policy,
+ /// or the identity when the query does not read that leaf.
+ fn date_policy(&self, leaf: usize) -> RebasePolicy {
+ if self.is_requested(leaf) {
+ self.date
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+
+ /// The policy of the timezone-carrying timestamp leaf at depth-first ordinal `leaf`: the
+ /// identity when the query does not read it; otherwise its physical type's spec when the
+ /// attribution is known, or else the two specs merged -- agreement decides, disagreement
+ /// degrades to [`RebasePolicy::CheckAncient`], which still passes every modern value and
+ /// refuses only ancient ones.
+ fn timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ Some(false) => self.int64_timestamp,
+ None if self.int64_timestamp == self.int96_timestamp => self.int64_timestamp,
+ None => RebasePolicy::CheckAncient,
+ }
+ }
+
+ /// These policies with every physical leaf the query does not read marked the identity.
+ /// `requested` pairs each top-level field of `physical_schema` (by position) with the type
+ /// of the logical field the schema adapter narrows it to -- `None` for a column without a
+ /// logical counterpart, whose leaves are left as they are (no expression reads it anyway).
+ /// Nested children pair the way the adapter's struct convert selects them (see
+ /// [`push_unrequested_leaves`]); the INT96 attribution is untouched, since the ordinals
+ /// stay physical. `requested` is parallel to the schema's fields; should a caller pass a
+ /// shorter slice, the trailing columns simply keep every leaf (the safe direction).
+ pub(crate) fn restrict_to_requested(
+ mut self,
+ physical_schema: &Schema,
+ requested: &[Option<&DataType>],
+ case_sensitive: bool,
+ use_field_id: bool,
+ ) -> Self {
+ debug_assert_eq!(requested.len(), physical_schema.fields().len());
+ let matching = FieldMatching {
+ case_sensitive,
+ use_field_id,
+ };
+ let mut next_leaf = 0;
+ let mut unrequested = Vec::new();
+ for (field, requested) in physical_schema.fields().iter().zip(requested) {
+ match requested {
+ Some(logical) => push_unrequested_leaves(
+ field.data_type(),
+ logical,
+ &mut next_leaf,
+ matching,
+ &mut unrequested,
+ ),
+ None => next_leaf += leaf_count(field.data_type()),
+ }
+ }
+ // Emitted in depth-first order, so already sorted for `is_requested`'s binary search.
+ self.unrequested_leaves = unrequested;
+ self
+ }
+}
+
+/// The field-matching rules of the schema adapter's nested narrowing
+/// (`parquet_convert_struct_to_struct`): names fold per `case_sensitive`, and Parquet field ids
+/// select fields when `use_field_id` is set.
+#[derive(Debug, Clone, Copy)]
+struct FieldMatching {
+ case_sensitive: bool,
+ use_field_id: bool,
+}
+
+/// Appends to `out` the depth-first leaf ordinals of `physical` (counting from `next_leaf`,
+/// which advances past every leaf of `physical`) that reading it as `requested` drops.
+///
+/// Recurses through exactly the pairings `parquet_convert_array` narrows, and no others: a
+/// struct child is dropped only when NO requested child selects it by either rule the struct
+/// convert uses -- folded name, or Parquet field id when ids are in play -- and an ambiguous
+/// child (several requested children select it) is kept; `List` pairs with `List` by element
+/// type, and `Map` with a `Map` of the same key ordering by its entries, positionally. Any
+/// other pairing -- a leaf, a `LargeList` / `FixedSizeList` / dictionary, a map whose ordering
+/// differs, or a shape mismatch -- is handed to arrow's cast or passed through whole by the
+/// convert, so it keeps every leaf. Keeping a superset of what the narrowing reads is always
+/// safe (a spurious check at worst); dropping a leaf the narrowing reads would skip its
+/// rebase, so every doubt resolves to "requested".
+fn push_unrequested_leaves(
+ physical: &DataType,
+ requested: &DataType,
+ next_leaf: &mut usize,
+ matching: FieldMatching,
+ out: &mut Vec<usize>,
+) {
+ match (physical, requested) {
+ (DataType::Struct(physical_fields), DataType::Struct(requested_fields)) => {
+ let names: Vec<&str> = physical_fields
+ .iter()
+ .chain(requested_fields.iter())
+ .map(|f| f.name().as_str())
+ .collect();
+ let folded = fold_names(&names, matching.case_sensitive);
+ let (physical_folded, requested_folded) = folded.split_at(physical_fields.len());
+ for (i, child) in physical_fields.iter().enumerate() {
+ let child_id = if matching.use_field_id {
+ parse_field_id(child)
+ } else {
+ None
+ };
+ let mut selectors = requested_fields.iter().enumerate().filter(|(j, r)| {
+ requested_folded[*j] == physical_folded[i]
+ || (child_id.is_some() && parse_field_id(r) == child_id)
+ });
+ match (selectors.next(), selectors.next()) {
+ (None, _) => {
+ let n = leaf_count(child.data_type());
+ out.extend(*next_leaf..*next_leaf + n);
+ *next_leaf += n;
+ }
+ (Some((_, requested_child)), None) => push_unrequested_leaves(
+ child.data_type(),
+ requested_child.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (Some(_), Some(_)) => *next_leaf += leaf_count(child.data_type()),
+ }
+ }
+ }
+ (DataType::List(physical_item), DataType::List(requested_item)) => push_unrequested_leaves(
+ physical_item.data_type(),
+ requested_item.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (
+ DataType::Map(physical_entries, physical_sorted),
+ DataType::Map(requested_entries, requested_sorted),
+ ) if physical_sorted == requested_sorted => {
+ match (physical_entries.data_type(), requested_entries.data_type()) {
+ (DataType::Struct(physical_kv), DataType::Struct(requested_kv))
+ if physical_kv.len() == requested_kv.len() =>
+ {
+ for (p, r) in physical_kv.iter().zip(requested_kv.iter()) {
+ push_unrequested_leaves(
+ p.data_type(),
+ r.data_type(),
+ next_leaf,
+ matching,
+ out,
+ );
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+}
+
+/// The writer time zone recorded in `metadata`, classified for timestamp rebasing. Mirrors the
+/// `Option(lookupFileMeta(SPARK_TIMEZONE_METADATA_KEY))` lookup Spark's `getRebaseSpec` performs
+/// for every LEGACY resolution, conf-fallback included; Spark substitutes the JVM default zone
+/// when the key is absent (`RebaseSpec.timeZone`), which is unavailable natively, so an absent or
+/// non-UTC zone classifies as [`WriterTimeZone::OtherOrUnknown`] (dates still rebase fully --
+/// the day rebase is zone-free -- while ancient timestamps refuse rather than guess).
+fn writer_time_zone(metadata: &HashMap<String, String>) -> WriterTimeZone {
+ match metadata.get(SPARK_TIMEZONE_KEY) {
+ Some(tz) if UTC_EQUIVALENT_TIMEZONES.contains(&tz.as_str()) => WriterTimeZone::Utc,
+ _ => WriterTimeZone::OtherOrUnknown,
+ }
+}
+
+/// One spec resolution, mirroring Spark's `DataSourceUtils.getRebaseSpec` exactly: a Spark
+/// version below `min_version` (lexicographic comparison, same as the Scala `String.<`) or a
+/// present legacy flag means LEGACY; a Spark version at/after `min_version` without the flag
+/// means CORRECTED; no Spark version at all falls back to `conf_mode`, the session read conf
+/// forwarded from the JVM (`getRebaseSpec`'s `modeByConfig` fallback, its ONLY use of the
+/// conf): CORRECTED passes values through, LEGACY rebases (with the writer zone from the
+/// file's `org.apache.spark.timeZone` key, same lookup as the metadata-driven LEGACY path),
+/// and EXCEPTION refuses ancient values as [`RebasePolicy::CheckAncient`].
+fn resolve_spec(
+ metadata: &HashMap<String, String>,
+ min_version: &str,
+ legacy_key: &str,
+ conf_mode: RebaseReadMode,
+) -> RebasePolicy {
+ match metadata.get(SPARK_VERSION_METADATA_KEY) {
+ None => match conf_mode {
+ RebaseReadMode::Corrected => RebasePolicy::Corrected,
+ RebaseReadMode::Legacy => RebasePolicy::Legacy(writer_time_zone(metadata)),
+ RebaseReadMode::Exception => RebasePolicy::CheckAncient,
+ },
+ Some(version) => {
+ if version.as_str() < min_version || metadata.contains_key(legacy_key) {
+ RebasePolicy::Legacy(writer_time_zone(metadata))
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+ }
+}
+
+/// Resolves the per-file rebase policies from a file's arrow schema: the parquet footer's
+/// key-value pairs in its metadata decide the specs (the datetime spec uses min version
+/// `3.0.0` and the INT96 spec `3.1.0`, matching `DataSourceUtils.datetimeRebaseSpec` /
+/// `int96RebaseSpec`; `session_modes` supplies the per-spec conf fallback for files without
+/// Spark writer metadata), and the reader factory's INT96 stamp -- validated against the
+/// schema's leaf structure -- attributes each timestamp leaf to its spec.
+pub(crate) fn resolve_file_rebase_policies(
+ physical_file_schema: &Schema,
+ session_modes: SessionRebaseModes,
+) -> FileRebasePolicies {
+ let metadata = physical_file_schema.metadata();
+ let datetime_spec = resolve_spec(
+ metadata,
+ "3.0.0",
+ SPARK_LEGACY_DATETIME_KEY,
+ session_modes.datetime,
+ );
+ let int96_spec = resolve_spec(
+ metadata,
+ "3.1.0",
+ SPARK_LEGACY_INT96_KEY,
+ session_modes.int96,
+ );
+ FileRebasePolicies {
+ date: datetime_spec,
+ int64_timestamp: datetime_spec,
+ int96_timestamp: int96_spec,
+ int96_leaves: Int96Attribution::from_schema(physical_file_schema),
+ unrequested_leaves: Vec::new(),
+ }
+}
+
+/// Number of primitive leaves `dt` contains in a depth-first walk -- the same count and order
+/// parquet-rs uses when it maps the file's `SchemaDescriptor` columns onto the arrow schema, so
+/// arrow-side leaf ordinals line up with [`int96_leaf_stamp`]'s.
+fn leaf_count(dt: &DataType) -> usize {
+ match dt {
+ DataType::Struct(fields) => fields.iter().map(|f| leaf_count(f.data_type())).sum(),
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_count(f.data_type()),
+ DataType::Dictionary(_, value) => leaf_count(value),
+ DataType::RunEndEncoded(_, value) => leaf_count(value.data_type()),
+ DataType::Union(fields, _) => fields.iter().map(|(_, f)| leaf_count(f.data_type())).sum(),
+ _ => 1,
+ }
+}
+
+/// Appends the policy of every leaf of `dt`, in depth-first order, to `out`, consuming leaf
+/// ordinals from `next_leaf` (exactly [`leaf_count`] of them). Only `Date32` and
+/// timezone-carrying timestamps have a policy to apply, and only when the query reads the
+/// leaf; timezone-free timestamps are `TIMESTAMP_NTZ`, which Spark never rebases, and every
+/// other leaf is the identity ([`RebasePolicy::Corrected`]).
+fn leaf_policies(
+ dt: &DataType,
+ next_leaf: &mut usize,
+ policies: &FileRebasePolicies,
+ out: &mut Vec<RebasePolicy>,
+) {
+ match dt {
+ DataType::Date32 => {
+ out.push(policies.date_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Timestamp(_, Some(_)) => {
+ out.push(policies.timestamp_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Struct(fields) => {
+ for f in fields {
+ leaf_policies(f.data_type(), next_leaf, policies, out);
+ }
+ }
+ // Mirrors `leaf_count` variant for variant, so a rebase-affected leaf inside a nested
+ // type `rebase_array` cannot rebuild (views, run-end, union -- never produced from a
+ // parquet schema) still gets its real policy and makes `rebase_array` refuse loudly
+ // instead of being stamped the identity.
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_policies(f.data_type(), next_leaf, policies, out),
+ DataType::Dictionary(_, value) => leaf_policies(value, next_leaf, policies, out),
+ DataType::RunEndEncoded(_, value) => {
+ leaf_policies(value.data_type(), next_leaf, policies, out)
+ }
+ DataType::Union(fields, _) => {
+ for (_, f) in fields.iter() {
+ leaf_policies(f.data_type(), next_leaf, policies, out);
+ }
+ }
+ _ => {
+ *next_leaf += 1;
+ out.push(RebasePolicy::Corrected);
+ }
+ }
+}
+
+/// Wraps every column reference in `expr` whose physical file type contains a rebase-affected
+/// leaf under a policy that needs handling with a [`SparkDatetimeRebaseExpr`] carrying that
+/// column's per-leaf policies, so both the per-file projection and the pushed-down predicate
+/// evaluate rebased values. Columns whose leaves are all the identity -- unaffected types,
+/// affected types under [`RebasePolicy::Corrected`], or leaves the query does not read (see
+/// [`FileRebasePolicies::restrict_to_requested`]) -- pass through unwrapped. (The pruning
+/// predicates derived from the wrapped predicate treat the wrapper as an opaque expression and
+/// skip pruning on those columns -- conservative, since file-level statistics are in the
+/// file's own calendar.)
+pub(crate) fn wrap_datetime_rebase(
+ expr: Arc<dyn PhysicalExpr>,
+ physical_schema: &SchemaRef,
+ policies: &FileRebasePolicies,
+) -> DataFusionResult<Arc<dyn PhysicalExpr>> {
+ expr.transform(|e| {
+ let Some(col) = e.downcast_ref::<Column>() else {
+ return Ok(Transformed::no(e));
+ };
+ // Missing columns were already replaced with literals; any surviving reference is
+ // physical-schema-indexed. Out-of-range means a non-file column (defensive): skip.
+ let Some(field) = physical_schema.fields().get(col.index()) else {
+ return Ok(Transformed::no(e));
+ };
+ // This column's first leaf ordinal: the leaves of every preceding top-level field.
+ let mut next_leaf: usize = physical_schema.fields()[..col.index()]
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ let mut column_leaf_policies = Vec::with_capacity(leaf_count(field.data_type()));
+ leaf_policies(
+ field.data_type(),
+ &mut next_leaf,
+ policies,
+ &mut column_leaf_policies,
+ );
+ if column_leaf_policies
+ .iter()
+ .all(|p| *p == RebasePolicy::Corrected)
+ {
+ return Ok(Transformed::no(e));
+ }
+ Ok(Transformed::yes(Arc::new(SparkDatetimeRebaseExpr {
+ child: e,
+ field: Arc::clone(field),
+ leaf_policies: column_leaf_policies,
+ }) as Arc<dyn PhysicalExpr>))
+ })
+ .map(|t| t.data)
+}
+
+/// Applies a file's calendar-rebase policies to one column: rebases exactly where possible,
+/// raises on ancient values it cannot rebase, and passes modern values (the identity under
+/// every policy) through untouched. Nested columns are rebuilt leaf by leaf with nulls and
+/// offsets preserved. See the module doc for the policy table.
+#[derive(Debug, Eq)]
+struct SparkDatetimeRebaseExpr {
+ child: Arc<dyn PhysicalExpr>,
+ /// The physical file field this expression reads (type preserved by the rebase).
+ field: FieldRef,
+ /// One policy per primitive leaf of `field`'s type, in depth-first order (a single entry
+ /// for a flat column). At least one is not [`RebasePolicy::Corrected`].
+ leaf_policies: Vec<RebasePolicy>,
+}
+
+impl SparkDatetimeRebaseExpr {
+ /// The refusal error, as an [`ArrowError`] so `try_unary` closures can raise it directly;
+ /// it converts into a `DataFusionError` at the `?` in `evaluate`.
+ fn rebase_error(&self, detail: &str) -> ArrowError {
+ ArrowError::ComputeError(format!(
+ "Native scan cannot rebase ancient values in column '{}': the file was written \
+ with the legacy (hybrid Julian/Gregorian) calendar, or does not declare which \
+ calendar it used, and {detail}. Reading it natively would return silently \
+ shifted values; disable the native Delta scan \
+ (spark.comet.scan.delta.enabled=false) to let Spark read this table",
+ self.field.name(),
+ ))
+ }
+
+ fn internal_error(&self, detail: impl Display) -> DataFusionError {
+ DataFusionError::Internal(format!(
+ "SparkDatetimeRebaseExpr on column '{}': {detail}",
+ self.field.name()
+ ))
+ }
+
+ /// Rebases a timestamp column written at a fixed zero-offset zone: shift the nominal day
+ /// with the exact date table, keep the time of day. Matches Spark's
+ /// `rebaseJulianToGregorianMicros` for UTC, where the hybrid calendar's day boundaries sit
+ /// exactly on multiples of a day and no timezone transition can apply (UTC's last switch
+ /// instant in Spark's rebase table is the 1582-10-15 cutover itself).
+ fn rebase_timestamp_utc(&self, v: i64, units_per_day: i64) -> Result<i64, ArrowError> {
+ // Compare in days, not units: the cutover day times a nanosecond day does not fit i64.
+ let day = v.div_euclid(units_per_day);
+ if day >= LAST_SWITCH_JULIAN_DAY as i64 {
+ return Ok(v);
+ }
+ let time_of_day = v - day * units_per_day;
+ let day = i32::try_from(day).map_err(|_| {
+ self.rebase_error("the value is outside the rebaseable timestamp range")
+ })?;
+ let rebased = rebase_julian_to_gregorian_days(day) as i64;
+ rebased
+ .checked_mul(units_per_day)
+ .and_then(|d| d.checked_add(time_of_day))
+ .ok_or_else(|| self.rebase_error("the rebased value overflows the timestamp range"))
+ }
+
+ /// The refuse-ancient-values policy for timestamps: values from
+ /// [`LAST_SWITCH_JULIAN_TS_SECONDS`] onward are identical under both calendars in every
+ /// zone (Spark's `createTimestampRebaseFuncInRead` under `EXCEPTION` accepts exactly
+ /// these, and `rebaseJulianToGregorianMicros` is the identity on them for any zone);
+ /// older values raise.
+ fn check_ancient_timestamp(
+ &self,
+ v: i64,
+ units_per_second: i64,
+ detail: &str,
+ ) -> Result<i64, ArrowError> {
+ if v >= LAST_SWITCH_JULIAN_TS_SECONDS * units_per_second {
+ Ok(v)
+ } else {
+ Err(self.rebase_error(detail))
+ }
+ }
+
+ fn rebase_timestamp_array<T: ArrowTimestampType>(
+ &self,
+ array: &PrimitiveArray<T>,
+ policy: RebasePolicy,
+ units_per_second: i64,
+ ) -> DataFusionResult<ArrayRef> {
+ let tz = array.timezone().map(Arc::<str>::from);
+ let rebased: PrimitiveArray<T> = match policy {
+ RebasePolicy::Corrected => return Ok(Arc::new(array.clone())),
+ RebasePolicy::Legacy(WriterTimeZone::Utc) => arrow::compute::try_unary(array, |v| {
+ self.rebase_timestamp_utc(v, units_per_second * 86_400)
+ })?,
+ RebasePolicy::Legacy(WriterTimeZone::OtherOrUnknown) => {
Review Comment:
> A validity-aware `all(v >= cutoff)` over `values()` followed by `Ok(Arc::clone(array))` would make them allocation-free
Done for timestamps and dates. A batch with nothing before the cutover returns the input Arc under every policy, including `Legacy(Utc)`, and the check is validity-aware so null slots holding ancient values do not force a copy. Tests assert pointer equality for the pass-through, that the checking policies still reject an ancient non-null value, and that the legacy UTC rebase still produces a new buffer when one is needed. The `try_unary` helper for the checking arms is gone since nothing calls it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5682113862
CI on 6039c304a failed in two places. `Lint Java` caught a redundant `s` interpolator in the new discovery test, fixed at b238a9811 together with two older ones in the contrib suite; the CI-style semantic scalafix (`package -DskipTests scalafix:scalafix -Dscalafix.mode=CHECK -Psemanticdb`) passes locally on that head. `Lint Scala (syntactic)` failed before running anything, on three attempts to download scalafix from Maven Central, so that one needs a rerun.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5442357496
@andygrove, a few factual clarifications against `7a0f2a59` for [these questions](https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5441843040):
- The default native feature was [discussed here](https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5344589643) and [accepted here](https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5346521797) to avoid requiring a custom native build. The separate `contrib-delta` comment says “zero kernel surface,” not zero Delta surface. Runtime opt-in does not imply zero binary cost; I have not verified the binary-size cost.
- Coverage includes [Delta DELETE-produced DVs](https://github.com/apache/datafusion-comet/blob/7a0f2a59a575625094a9b07729b84c9c30bf0582/contrib/delta-spark/src/test/scala/org/apache/comet/contrib/delta/CometDeltaNativeScanSuite.scala#L447), not only Rust round trips. The [decoder](https://github.com/apache/datafusion-comet/blob/7a0f2a59a575625094a9b07729b84c9c30bf0582/native/core/src/execution/delta_dv.rs#L57) has explicit truncation/length/CRC error paths; this is not a comprehensive malformed-input or no-panic proof.
- [DeltaScanSupport](https://github.com/apache/datafusion-comet/blob/7a0f2a59a575625094a9b07729b84c9c30bf0582/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala#L79) uses typed Delta APIs, not an IcebergReflection-style layer. The hook catches `NonFatal`; linkage errors deliberately propagate, so arbitrary version skew is not guaranteed to decline cleanly.
- [CI is path-gated](https://github.com/apache/datafusion-comet/blob/7a0f2a59a575625094a9b07729b84c9c30bf0582/dev/ci/compute-changes.py#L248), including shared Spark/Common/build changes. Current-head workflows are approval-blocked, so they do not establish wall-clock or runner-minute cost.
- Benchmark numbers are present in the description, but the 1-TB figures are author-reported at older commit `7fd81aa9`, with a row-reader control. A [DV plus page-pruning regression](https://github.com/apache/datafusion-comet/blob/7a0f2a59a575625094a9b07729b84c9c30bf0582/contrib/delta-spark/src/test/scala/org/apache/comet/contrib/delta/CometDeltaNativeScanSuite.scala#L568) already checks native execution, 90 surviving rows and positive pruning metrics; that is correctness coverage, not a pruning-on/off performance comparison.
These are source/discussion clarifications; I did not rerun the tests or benchmarks for this follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3910365988
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1714 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Zero-I/O in the common case, like the encryption gate above: native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ /**
+ * `CommonConfigurationKeysPublic.HADOOP_SECURITY_CREDENTIAL_CLEAR_TEXT_FALLBACK`, default
+ * `true`, verified via `javap` against `hadoop-common` 3.3.4's
+ * `Configuration#getPasswordFromConfig`: `getPassword` only falls back to reading a plaintext
+ * conf value once `getBoolean(<this key>, true)` holds -- with the flag off, a plaintext value
+ * is invisible to every `getPassword`-based resolver, even when no credential provider is
+ * configured at all.
+ */
+ private val ClearTextFallbackKey = "hadoop.security.credential.clear-text-fallback"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * How Hadoop's OWN consumer reads one of the keys compared by [[s3ConfigDivergenceReason]],
+ * which decides how [[s3KeyDivergenceReason]] computes the Hadoop-effective side of its
+ * equality check. Exactly two consumer families exist among [[AllS3ConfigKeys]] in `hadoop-aws`
+ * 3.3.4, each verified via `javap`/CFR against the real call sites (cited per key on
+ * [[S3ConfigKeyConsumers]]). The tier must mirror the key's ACTUAL consumer: resolving a
+ * [[PropagatedOptionConsumer]] key through the wider `lookupPassword` cascade is NOT fail-safe
+ * for a value-EQUALITY comparator -- a long-form alias value Hadoop itself never reads can
+ * EQUAL native's resolution while Hadoop's true propagate-then-plain-get value differs, turning
+ * a real divergence into a wrongly-claimed scan (the endpoint `${...}`-redirect shape pinned in
+ * `DeltaScanContribSuite`).
+ */
+ private[delta] sealed trait S3ConfigConsumer
+
+ /**
+ * Read via `S3AUtils#lookupPassword(bucket, conf, baseKey)`, verified via `javap` against
+ * `hadoop-aws` 3.3.4: builds `longBucketKey = "fs.s3a.bucket." + bucket + "." + baseKey` (the
+ * FULL, already-`fs.s3a`-prefixed base key appended after the bucket segment) and reads it via
+ * `Configuration#getPassword` BEFORE the short-bucket key, keeping the long value whenever
+ * `getPassword` returns non-empty and only falling through to short-then-global otherwise.
+ * `getPassword` is Hadoop-credential-provider-aware and skips plaintext conf entirely when
+ * [[ClearTextFallbackKey]] is false. Modeled by [[hadoopLookupPasswordEffective]].
+ */
+ private[delta] case object LookupPasswordConsumer extends S3ConfigConsumer
+
+ /**
+ * Read via `S3AUtils#propagateBucketOptions` followed by a plain `Configuration#get`-family
+ * call (`getTrimmed`/`getBoolean`/`getClasses`) against the propagated view: the short bucket
+ * form wins only by having overwritten the global key during propagation, the long bucket form
+ * folds into an unread `fs.s3a.fs.s3a.*` key, and neither a credential provider nor
+ * [[ClearTextFallbackKey]] is ever consulted. Modeled as a plain `Configuration#get` on the
+ * [[propagateBucketOptions]] result, which also expands `${...}` references under that
+ * propagated view exactly like the real consumer.
+ */
+ private[delta] case object PropagatedOptionConsumer extends S3ConfigConsumer
+
+ /**
+ * Every `fs.s3a.*` base key that governs whether a claimed native scan actually behaves like
+ * Hadoop's own reader would, paired with the consumer family Hadoop resolves it through -- ONE
+ * list, with each key's resolution tier declared beside it, so a key can never sit in the
+ * comparator without a deliberate classification (adding one without picking a tier does not
+ * compile). The entries are every per-bucket `fs.s3a.*` base key native's S3 client's
+ * `get_config` (s3.rs) resolves, verified directly against its call sites:
+ * `extract_s3_config_options` (endpoint.region, path.style.access, endpoint,
+ * requester.pays.enabled), `lookup_provider_class` (the Comet-specific
+ * credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name).
+ *
+ * Tier assignments, each verified via `javap`/CFR against `hadoop-aws` 3.3.4:
+ * - access.key/secret.key/session.token: `S3AUtils#getAWSAccessKeys` and
+ * `MarshalledCredentialBinding#fromFileSystem` (reached from
+ * `TemporaryAWSCredentialsProvider`) resolve all three via `S3AUtils#lookupPassword` --
+ * [[LookupPasswordConsumer]].
+ * - aws.credentials.provider and assumed.role.credentials.provider:
+ * `S3AUtils#buildAWSProviderList` -> `loadAWSProviderClasses` -> plain
+ * `Configuration#getClasses` -- [[PropagatedOptionConsumer]].
+ * - assumed.role.arn/session.name: `AssumedRoleCredentialProvider`'s constructor reads both
+ * via plain `Configuration#getTrimmed` -- [[PropagatedOptionConsumer]].
+ * - endpoint (`S3AFileSystem`: `getTrimmed`), endpoint.region (`DefaultS3ClientFactory`:
+ * `getTrimmed`), path.style.access (`S3AFileSystem`: `getBoolean`) --
+ * [[PropagatedOptionConsumer]].
+ * - requester.pays.enabled: not read anywhere in `hadoop-aws` 3.3.4 (the constant does not
+ * even exist in its `Constants` class); later releases read it via plain `getBoolean`
+ * against the propagated conf, so the plain tier is both the faithful forward model and
+ * inert on 3.3.4 -- [[PropagatedOptionConsumer]].
+ * - comet.credential.provider.class: Comet's own activation key, plain conf read on both
+ * sides, never a Hadoop key at all -- [[PropagatedOptionConsumer]].
+ *
+ * SYNC NOTE: the key list must stay a superset of native's `NATIVE_S3A_CONFIG_PROPERTIES`
+ * constant (`native/core/src/parquet/objectstore/s3.rs`, property suffixes without the
+ * `fs.s3a.` prefix) -- `DeltaScanContribSuite`'s discovery-harness test asserts this
+ * mechanically against [[AllS3ConfigKeys]]. Literal strings, not the
+ * [[AwsCredentialsProviderKey]] / [[AssumedRoleCredentialsProviderKey]] vals declared below,
+ * purely to avoid a forward reference inside this `object` body; kept textually identical to
+ * those two constants.
+ */
+ private[delta] val S3ConfigKeyConsumers: Seq[(String, S3ConfigConsumer)] = Seq(
+ "fs.s3a.access.key" -> LookupPasswordConsumer,
+ "fs.s3a.secret.key" -> LookupPasswordConsumer,
+ "fs.s3a.session.token" -> LookupPasswordConsumer,
+ "fs.s3a.aws.credentials.provider" -> PropagatedOptionConsumer,
+ "fs.s3a.assumed.role.arn" -> PropagatedOptionConsumer,
+ "fs.s3a.assumed.role.session.name" -> PropagatedOptionConsumer,
+ "fs.s3a.assumed.role.credentials.provider" -> PropagatedOptionConsumer,
+ "fs.s3a.endpoint" -> PropagatedOptionConsumer,
+ "fs.s3a.endpoint.region" -> PropagatedOptionConsumer,
+ "fs.s3a.path.style.access" -> PropagatedOptionConsumer,
+ "fs.s3a.requester.pays.enabled" -> PropagatedOptionConsumer,
+ "fs.s3a.comet.credential.provider.class" -> PropagatedOptionConsumer)
+
+ /** The compared keys alone, in [[S3ConfigKeyConsumers]] order (discovery-harness surface). */
+ private[delta] val AllS3ConfigKeys: Seq[String] = S3ConfigKeyConsumers.map(_._1)
+
+ /**
+ * The short-bucket-then-global value resolved for `baseKey` under `bucket` from `hadoopConf`,
+ * skipping an empty value at either alias exactly like [[plainValue]]. NOT used by
+ * [[s3ConfigDivergenceReason]]/[[s3KeyDivergenceReason]] any more -- every key checked there
+ * resolves per its declared [[S3ConfigKeyConsumers]] tier (see [[s3KeyDivergenceReason]]),
+ * neither of which matches this function's read. This function's one remaining caller is
+ * [[shortThenGlobalOrReason]], which reads provider-CLASS strings (from the ORIGINAL,
+ * unpropagated conf) for name-support validation in [[providerClassReason]]/
+ * [[assumedRoleProviderClassReason]] -- by the time those run, [[s3ConfigDivergenceReason]] has
+ * already proven Hadoop's and native's effective values agree for the same key, so whichever of
+ * the two (equal) values this narrower read returns does not affect correctness there. NEVER
+ * used to compute native's own effective value -- see [[nativeShortThenGlobal]] for that.
+ */
+ private def shortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket` from the ORIGINAL, unpropagated `hadoopConf` -- `NativeConfig
+ * .extractObjectStoreOptions` forwards `Configuration#get`'s substituted value for every
+ * `fs.s3a.*` entry with no bucket-option propagation step of its own, so the original conf is
+ * the right input here. Unlike [[shortThenGlobal]]/[[plainValue]], this mirrors `get_config`
+ * faithfully: PRESENCE of the short-bucket key alone -- never its emptiness -- decides whether
+ * native falls back to the global key (`get_config` is a plain `HashMap::get`, which returns
+ * `Some` for a key explicitly set to `""`), so an explicitly empty or whitespace-only
+ * short-bucket value resolves to `Some("")` here and never falls through to global -- the
+ * OPPOSITE of Hadoop's own `getPassword`/`lookupPassword` semantics (see
+ * [[hadoopLookupPasswordEffective]]), which treat empty as absent and keep trying the next
+ * alias. The ONLY function used to compute native's effective value in
+ * [[s3KeyDivergenceReason]].
+ *
+ * Deliberately does NOT apply `get_config_trimmed`'s `.trim()` here: [[s3KeyDivergenceReason]]
+ * trims both this value and Hadoop's effective value together, symmetrically, at the point they
+ * are compared, rather than one-sidedly here -- trimming only the native side would flag a
+ * spurious divergence for a value neither side's whitespace actually changes the behavior of
+ * once each side's own downstream parsing normalizes it (e.g. Hadoop's own multi-line
+ * `fs.s3a.aws.credentials.provider` default, which both Hadoop and native additionally trim per
+ * comma-separated entry after splitting), while a one-sided trim would make an
+ * otherwise-identical default value look diverged for every bucket, never claiming natively at
+ * all.
+ */
+ private def nativeShortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ Option(hadoopConf.get(shortKey)).orElse(Option(hadoopConf.get(baseKey)))
+ }
+
+ /**
+ * Faithful in-memory replica of `S3AUtils#propagateBucketOptions` (`hadoop-aws`), which
+ * `S3AFileSystem#initialize` calls FIRST, before any option or credential is read:
+ * `Configuration conf = propagateBucketOptions(originalConf, bucket); ...; setConf(conf);` --
+ * every subsequent `conf.get`/`getPassword` call in that filesystem instance, including
+ * `${...}` variable substitution, resolves against this propagated view, not the original conf.
+ * `hadoop-aws` is not on this module's runtime classpath (see the string-literal-keys note
+ * above), so `S3AUtils#propagateBucketOptions` cannot be called directly; this reproduces its
+ * logic verbatim using only `hadoop-common`'s `Configuration`:
+ * {{{
+ * public static Configuration propagateBucketOptions(Configuration source, String bucket) {
+ * final String bucketPrefix = FS_S3A_BUCKET_PREFIX + bucket + '.';
+ * final Configuration dest = new Configuration(source);
+ * for (Map.Entry<String, String> entry : source) {
+ * final String key = entry.getKey();
+ * final String value = entry.getValue(); // the (unexpanded) value
+ * if (!key.startsWith(bucketPrefix) || bucketPrefix.equals(key)) continue;
+ * final String stripped = key.substring(bucketPrefix.length());
+ * if (stripped.startsWith("bucket.") || "impl".equals(stripped)) {
+ * // ignored
+ * } else {
+ * final String generic = FS_S3A_PREFIX + stripped;
+ * dest.set(generic, value, ...); // overwrites any existing global value
+ * }
+ * }
+ * return dest;
+ * }
+ * }}}
+ * Note the LONG bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) folds to an unread
+ * `fs.s3a.fs.s3a.<key>` key here too, exactly like the real method -- `stripped` already starts
+ * with `fs.s3a.` in that case, so prepending `fs.s3a.` again produces a key nothing ever reads.
+ */
+ private def propagateBucketOptions(hadoopConf: Configuration, bucket: String): Configuration = {
+ val bucketPrefix = s"fs.s3a.bucket.$bucket."
+ val dest = new Configuration(hadoopConf)
+ hadoopConf.iterator().asScala.foreach { entry =>
+ val key = entry.getKey
+ if (key.startsWith(bucketPrefix) && key != bucketPrefix) {
+ val stripped = key.substring(bucketPrefix.length)
+ if (!stripped.startsWith("bucket.") && stripped != "impl") {
+ dest.set(s"fs.s3a.$stripped", entry.getValue)
+ }
+ }
+ }
+ dest
+ }
+
+ /**
+ * Canonical and deprecated Hadoop S3A encryption-algorithm config keys, verified via `javap`
+ * against `hadoop-aws` 3.3.4's `org.apache.hadoop.fs.s3a.Constants`: `S3_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.encryption.algorithm"` (canonical) and `SERVER_SIDE_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.server-side-encryption-algorithm"` (DEPRECATED -- note the hyphen before "algorithm",
+ * unlike the corresponding `*.key` constants below, which both use a `.key` suffix).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, same
+ * rationale as [[HadoopCredentialProviderPathKey]].
+ */
+ private val S3EncryptionAlgorithmKey = "fs.s3a.encryption.algorithm"
+ private val DeprecatedS3EncryptionAlgorithmKey = "fs.s3a.server-side-encryption-algorithm"
+
+ /**
+ * The exact strings `S3AEncryptionMethods#getMethod` accepts, verified via `javap`/CFR against
+ * `hadoop-aws` 3.3.4's `S3AEncryptionMethods` enum: `NONE("")`, `SSE_S3("AES256", serverSide =
+ * true, requiresSecret = false)`, `SSE_KMS("SSE-KMS", serverSide = true, requiresSecret =
+ * false)`, `SSE_C("SSE-C", serverSide = true, requiresSecret = true)`, `CSE_KMS("CSE-KMS",
+ * serverSide = false, requiresSecret = true)`, `CSE_CUSTOM("CSE-CUSTOM", serverSide = false,
+ * requiresSecret = true)`. `getMethod` parses case-insensitively
+ * (`values().find(_.getMethod.equalsIgnoreCase(algorithm))`), matched below the same way.
+ *
+ * ALLOWLIST, not a blocklist (replaces the former SSE-C-only blocklist): only the algorithms S3
+ * decrypts transparently on GET/HEAD given read permission alone, with NO extra request header
+ * and NO client-side step, are safe for a native scan that forwards none of Hadoop's
+ * `fs.s3a.encryption.*`/`fs.s3a.server-side-encryption*` options --
+ * - `AES256` (SSE_S3, `serverSide = true`): plain server-side encryption, transparent on GET.
+ * - `SSE-KMS` (SSE_KMS, `serverSide = true`): server-side, KMS-managed key, transparent on
+ * GET given KMS decrypt permission (no header).
+ * - `DSSE-KMS`: NOT present in this enum on `hadoop-aws` 3.3.4 (confirmed by the six values
+ * listed above) -- `S3AEncryptionMethods.getMethod("DSSE-KMS")` throws
+ * `IOException("Unknown encryption algorithm DSSE-KMS")` on this version, so
+ * `S3AUtils#buildEncryptionSecrets` (and therefore Hadoop's own reader) already fails
+ * before ever reading such a table under 3.3.4, meaning this string can never actually be
+ * the resolved value on the declared target version -- admitting it here is inert there.
+ * Included anyway, forward-compatible, for a newer `hadoop-aws` on the runtime classpath (a
+ * later Hadoop release; this module has no compile-time `hadoop-aws` dependency, see the
+ * string-literal-keys note above) where DSSE-KMS is a real, dual-layer, server-side
+ * algorithm decrypted transparently on GET the same way SSE-KMS is. Every other value
+ * declines: `SSE-C` (SSE_C is `serverSide = true` in Hadoop's own enum, but `requiresSecret
+ * \= true` -- S3 rejects a GET/HEAD for an SSE-C object outright (400 Bad Request) unless
+ * the customer key is resent as a request header on every call, so a native scan that never
+ * learns the key cannot succeed at all, where Hadoop's own reader -- whose request factory
+ * attaches the key -- would), `CSE-KMS`/`CSE-CUSTOM` (`serverSide = false`: client-side
+ * encryption decrypts object bytes locally in the SDK layer, which the native Parquet
+ * reader has no equivalent of -- it would read raw ciphertext), and any future/unknown
+ * value (a value `S3AEncryptionMethods.getMethod` itself would reject is certainly not one
+ * of the three confirmed-transparent algorithms above; declining is the only safe default
+ * for anything this gate cannot positively confirm).
+ */
+ private val AllowedEncryptionAlgorithms: Set[String] = Set("AES256", "SSE-KMS", "DSSE-KMS")
+
+ /**
+ * `bucket`'s effective encryption-algorithm key and value under `hadoopConf`, or `None` when
+ * neither the canonical nor deprecated key is set anywhere consulted. Mirrors
+ * `S3AUtils#buildEncryptionSecrets`'s real resolution order, verified via `javap`/CFR
+ * decompilation of `hadoop-aws` 3.3.4's `S3AUtils.class`:
+ * {{{
+ * String algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.server-side-encryption-algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.server-side-encryption-algorithm");
+ * }}}
+ * i.e. bucket-tier (canonical, then deprecated), THEN global-tier (canonical, then deprecated)
+ * -- the two tiers are never interleaved key-by-key, so this must stay two explicit bucket-tier
+ * lookups followed by two explicit global-tier lookups, not a single
+ * [[hadoopLookupPasswordEffective]] call per key (which would let an unset canonical bucket key
+ * fall through straight to the canonical GLOBAL value ahead of a SET deprecated bucket key, the
+ * wrong answer).
+ *
+ * THE FIX for the SSE-C long-bucket-alias gap is entirely inside the bucket tier:
+ * `lookupBucketSecret` itself is long-then-short, decompiled from `hadoop-aws` 3.3.4's
+ * `S3AUtils.class`:
+ * {{{
+ * // longBucketKey = fs.s3a.bucket.B.fs.s3a.<key>
+ * String longBucketKey = String.format(BUCKET_PATTERN, bucket, baseKey);
+ * String initialVal = getPassword(conf, longBucketKey, null, null);
+ * // shortBucketKey = fs.s3a.bucket.B.<key>
+ * String shortBucketKey = String.format(BUCKET_PATTERN, bucket, subkey);
+ * // keeps initialVal (the LONG value) if non-empty
+ * return getPassword(conf, shortBucketKey, initialVal, null);
+ * }}}
+ * i.e. the SAME long-bucket-key construction and long-wins-if-nonempty semantics as
+ * `S3AUtils#lookupPassword` (see [[LookupPasswordConsumer]]/[[hadoopLookupPasswordEffective]])
+ * -- the encryption algorithm is NOT one of the keys that flows through
+ * `S3AUtils#propagateBucketOptions` (which folds an unrelated per-bucket LONG form into an
+ * unread key). An earlier version of this function modeled the bucket tier as SHORT-only,
+ * documented as "the LONG bucket form is genuinely never consulted for this key" -- that
+ * documentation was wrong (this decompilation supersedes it): a bucket configured only via
+ * `fs.s3a.bucket.B.fs.s3a.encryption.algorithm=SSE-C` bypassed the SSE-C gate entirely, because
+ * Hadoop's own reader DOES read that long form (and picks SSE-C), while this function reported
+ * `None` (nothing set) and the allowlist check below never even ran.
+ *
+ * The canonical-vs-deprecated distinction below is frequently moot in practice: `hadoop-aws`'s
+ * `S3AFileSystem.addDeprecatedKeys()` statically registers `fs.s3a.server-side-encryption-*` as
+ * `Configuration`-level deprecated aliases of `fs.s3a.encryption.*` (verified via `javap`), a
+ * registration that lives in a static field on Hadoop's `Configuration` class -- process-wide
+ * once `S3AFileSystem`'s class has loaded anywhere in the JVM, which a real scan has always
+ * already done by the time this gate runs, since reading the S3 table at all requires loading
+ * that class. Once active, `Configuration#get` resolves either literal key to the identical
+ * value transparently, making the two-key cascade below redundant (but harmless) for that case;
+ * it remains the operative path only when nothing else in the process has loaded
+ * `S3AFileSystem` yet.
+ *
+ * ALSO walks the Hadoop-credential-provider (JCEKS) path via [[resolveViaCredentialAliases]]
+ * for each of the four lookups below, matching `lookupBucketSecret`/`lookupPassword`'s real
+ * per-alias `getPassword` calls (quoted above) exactly: both are `getPassword`, not plain
+ * `Configuration#get`, so a bucket storing the algorithm name ONLY in a JCEKS keystore is
+ * exactly as real a Hadoop deployment shape for this key as it is for the credential keys
+ * [[hadoopLookupPasswordEffective]] already covers -- there is nothing algorithm-specific that
+ * makes JCEKS storage implausible here, so an earlier version of this function skipping it
+ * (documented at the time as "the algorithm NAME is not credential-sensitive data, so storing
+ * it in a keystore is not a realistic Hadoop deployment pattern") was an unjustified, narrower
+ * read than Hadoop's own resolver actually performs, under-declining a bucket whose algorithm
+ * is keystore-only. [[resolveViaCredentialAliases]]'s Arm B/C split still means this is zero
+ * extra I/O for the common case: keystore I/O only happens when a Hadoop credential-provider
+ * path is actually configured for the bucket, contained in that function's own try/catch.
+ * `bucketTier`/`globalTier` return `Left` (propagated straight through by [[orElseTier]]) when
+ * [[resolveViaCredentialAliases]] cannot safely verify a tier at all (an S3A-scoped provider
+ * path, or a corrupt/unreadable global keystore) -- correctly short-circuiting the whole
+ * cascade with a decline rather than silently falling through to a later tier that might look
+ * unset only because the true value was unverifiable.
+ */
+ private def effectiveEncryptionAlgorithm(
+ hadoopConf: Configuration,
+ bucket: String): Either[String, Option[(String, String)]] = {
+ def bucketTier(baseKey: String): Either[String, Option[(String, String)]] = {
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ resolveViaCredentialAliases(
+ hadoopConf,
+ bucket,
+ Seq(longKey, shortKey),
+ honorClearTextFallback = true)
+ .map(_.map(baseKey -> _))
+ }
+ def globalTier(baseKey: String): Either[String, Option[(String, String)]] =
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(baseKey), honorClearTextFallback = true)
+ .map(_.map(baseKey -> _))
+
+ // Short-circuits on Left (unverifiable tier) or Right(Some(_)) (resolved); only Right(None)
+ // (tier definitively unset) falls through to `next`, mirroring buildEncryptionSecrets's
+ // sequential `if (algorithm == null) algorithm = ...` cascade exactly.
+ def orElseTier(
+ current: Either[String, Option[(String, String)]],
+ next: => Either[String, Option[(String, String)]])
+ : Either[String, Option[(String, String)]] =
+ current match {
+ case Left(reason) => Left(reason)
+ case Right(Some(value)) => Right(Some(value))
+ case Right(None) => next
+ }
+
+ orElseTier(
+ bucketTier(S3EncryptionAlgorithmKey),
+ orElseTier(
+ bucketTier(DeprecatedS3EncryptionAlgorithmKey),
+ orElseTier(
+ globalTier(S3EncryptionAlgorithmKey),
+ globalTier(DeprecatedS3EncryptionAlgorithmKey))))
+ }
+
+ private def unsupportedEncryptionAlgorithmDeclineReason(
+ bucket: String,
+ algorithmKey: String,
+ algorithm: String): String =
+ s"Native Delta scan does not support $algorithmKey=$algorithm for $bucket " +
+ "(the native S3 client only supports unencrypted objects and S3's transparent " +
+ "server-side algorithms -- AES256/SSE-S3, SSE-KMS, and DSSE-KMS decrypt on GET/HEAD given " +
+ "read permission alone, with no extra request header; SSE-C additionally requires the " +
+ "customer-provided key resent as a header on every GET/HEAD request, which the native S3 " +
+ "client's extract_s3_config_options never forwards, and CSE-KMS/CSE-CUSTOM decrypt object " +
+ "bytes client-side, a layer the native Parquet reader does not have -- any of these would " +
+ "fail outright or silently read ciphertext where Hadoop's own reader succeeds)"
+
+ /**
+ * First reason any bucket among `uris` is configured for an encryption algorithm the native S3
+ * client cannot safely read, or `None` when claimable. Allowlist-based (see
+ * [[AllowedEncryptionAlgorithms]]): only `AES256`/`SSE-KMS`/`DSSE-KMS` (and unset/empty) pass;
+ * every other resolved value -- `SSE-C`, `CSE-KMS`, `CSE-CUSTOM`, or any unrecognized future
+ * algorithm string -- declines. Deliberately NOT a blocklist keyed on `SSE-C` alone: an
+ * allowlist is safe by construction against a Hadoop release adding a new encryption method
+ * this gate has never heard of, where a blocklist would silently admit it. Never interpolates a
+ * resolved key value, only key names, the bucket, and the (non-secret) algorithm name. Declines
+ * on a `Left` from [[effectiveEncryptionAlgorithm]] too (an unverifiable credential-provider
+ * arm, e.g. an S3A-scoped provider path or a corrupt/unreadable global keystore) -- the
+ * algorithm cannot be ruled safe when it cannot be read at all.
+ */
+ private[delta] def unsupportedEncryptionAlgorithmReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ effectiveEncryptionAlgorithm(hadoopConf, bucket) match {
+ case Left(reason) => Some(reason)
+ case Right(None) => None
+ case Right(Some((key, value))) =>
+ if (!AllowedEncryptionAlgorithms.exists(_.equalsIgnoreCase(value))) {
+ Some(unsupportedEncryptionAlgorithmDeclineReason(bucket, key, value))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Canonical Hadoop S3A HTTP-proxy host config key, verified via CFR decompilation of
+ * `hadoop-aws` 3.3.4's `S3AUtils.class` (`initProxySupport`):
+ * {{{
+ * String proxyHost = conf.getTrimmed("fs.s3a.proxy.host", "");
+ * int proxyPort = conf.getInt("fs.s3a.proxy.port", -1);
+ * if (!proxyHost.isEmpty()) {
+ * ...
+ * String proxyUsername =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.username", null, null);
+ * String proxyPassword =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.password", null, null);
+ * ...
+ * }
+ * }}}
+ * `fs.s3a.proxy.host`/`fs.s3a.proxy.port` resolve via a PLAIN, non-bucket-scoped, non-JCEKS
+ * `Configuration#getTrimmed`/`getInt` call -- NOT `lookupPassword` -- against whatever conf
+ * `S3AFileSystem#initialize` already ran through `propagateBucketOptions` before
+ * `createAwsConf`/`initProxySupport` ever runs; only the SIBLING `fs.s3a.proxy.username`/
+ * `fs.s3a.proxy.password` keys go through `lookupPassword` (bucket long/short/global,
+ * JCEKS-aware). So the host is bucket-aware only via `propagateBucketOptions`'s short-bucket-
+ * form folding, never the long-bucket form, and never a credential-provider read -- the SAME
+ * shape as `endpoint`/`path.style.access` ([[PropagatedOptionConsumer]], see
+ * [[S3ConfigKeyConsumers]]'s doc), not the credential family.
+ *
+ * Resolved here through [[hadoopLookupPasswordEffective]]'s same long-then-short-then-global,
+ * credential-provider-aware cascade anyway (via [[resolveViaCredentialAliases]], with
+ * `honorClearTextFallback = false` since the host's real plain `getTrimmed` never checks
+ * [[ClearTextFallbackKey]] -- a plaintext host must stay visible to this gate even with the
+ * flag off): checking MORE than Hadoop's real, narrower resolution can only ever find a value
+ * Hadoop itself ignores, which for a PRESENCE gate can only turn an already-necessary decline
+ * into one found slightly more eagerly -- never a missed one. A missed decline is exactly the
+ * failure mode this gate exists to close (native has NO HTTP-proxy support of any kind -- no
+ * `fs.s3a.proxy.*` key is read anywhere in `s3.rs`), so over-approximating is the conservative
+ * direction here, unlike a value-EQUALITY comparator (e.g. [[s3KeyDivergenceReason]]) where a
+ * wider read than the key's real consumer can produce a false equality as easily as a false
+ * divergence -- which is exactly why the comparator classifies per key while this gate does not
+ * need to.
+ */
+ private val S3ProxyHostKey = "fs.s3a.proxy.host"
+
+ private def unsupportedProxyReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key configured for $bucket (the native S3 client has " +
+ "no HTTP proxy support at all -- no fs.s3a.proxy.* key is read anywhere in its object " +
+ "store layer -- so a claimed scan would connect to S3 directly instead of routing through " +
+ "the configured proxy, either bypassing an egress/network-segmentation policy or simply " +
+ "failing to reach the endpoint)"
+
+ /**
+ * First reason any bucket among `uris` has an HTTP proxy configured via [[S3ProxyHostKey]], or
+ * `None` when claimable. Ordered alongside [[unsupportedEncryptionAlgorithmReason]] among the
+ * other conf-only gates that are zero-I/O in the common case, ahead of
+ * [[s3ConfigDivergenceReason]]: [[hadoopLookupPasswordEffective]] only performs real keystore
+ * I/O, inside its own try/catch, when a Hadoop credential-provider path is actually configured
+ * for this bucket (see that function's doc for the Arm A/B/C dispatch). Never interpolates a
+ * resolved value: the proxy HOST is not secret, but naming it here would be a strange place to
+ * first surface it, and proxy CREDENTIALS (`fs.s3a.proxy.username`/`fs.s3a.proxy.password`, not
+ * read by this gate at all -- the whole point of gating on the host is that a non-empty host
+ * declines before any proxy credential would ever need to be forwarded) must never appear in a
+ * decline reason regardless.
+ */
+ private[delta] def proxyGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI],
+ propagatedConfCache: MutableMap[String, Configuration] = MutableMap.empty)
+ : Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ try {
+ val propagatedConf =
+ propagatedConfCache.getOrElseUpdate(
+ bucket,
+ propagateBucketOptions(hadoopConf, bucket))
+ resolveViaCredentialAliases(
+ propagatedConf,
+ bucket,
+ longThenShortThenGlobalAliases(bucket, S3ProxyHostKey),
+ honorClearTextFallback = false) match {
Review Comment:
Right again, and this one collapsed the gate rather than patching it: the host has exactly one real consumer, a plain getTrimmed on the propagated conf (initProxySupport in 3.3.4, AWSClientConfig in 3.4.x, both checked in the bytecode), so the provider-aware cascade never belonged there at all. The gate now does that single read and nothing else. Your shape declines; a host living only in a keystore now admits, since getTrimmed can never see it; and the lone long-form alias case now admits too, folding into the unread key the same way endpoint does. Each behavior flip carries the consumer fact in its test, and both profiles run green (the 4.1 cell included, where the class moved).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5505737394
> cc @andygrove @comphead @parthchandra @viirya @peterxcli as well, since this is a fairly large PR and would benefit from a few extra sets of eyes.
I will take a look. I may need some time because this is a large PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3942085900
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1789 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // An opted-in S3-compliant alias scheme (fs.comet.s3Compliant.schemes) is declined before the
+ // generic scheme gate below so the reason says why: core's native scan reads it through the
+ // S3 client, but the S3 divergence gates further down model Hadoop's S3AFileSystem only.
+ val aliasReason =
+ s3CompliantAliasSchemeReason(hadoopConf, scanExec.relation.location.rootPaths.map(_.toUri))
+ if (aliasReason.isDefined) {
+ return aliasReason
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Always zero-I/O (plain propagated-conf read, no keystore): native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Zero-I/O, conf-only, like the proxy gate above: Hadoop's AssumedRoleCredentialProvider
+ // sends fs.s3a.assumed.role.policy as the session policy of its STS AssumeRole request,
+ // while native's assumed-role provider never reads the key -- a claimed scan would assume
+ // the role WITHOUT the configured session restriction, silently widening permissions.
+ val rolePolicyReason =
+ assumedRolePolicyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (rolePolicyReason.isDefined) {
+ return rolePolicyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * parsed exactly like core's scan gate (`NativeConfig.parseSchemeSet`: split on commas,
+ * trimmed, lowercased) and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) => NativeConfig.parseSchemeSet(s)
+ case None => Set("hdfs")
+ }
+
+ /**
+ * Decline reason when any of `uris` uses a scheme opted in as an S3-compliant alias through
+ * `fs.comet.s3Compliant.schemes` (e.g. `blob`), or `None`. Core's native Parquet scan admits
+ * such a scheme and reads it through its S3 client, with `NativeConfig` translating the vendor
+ * `fs.<scheme>.<authority>.*` keys into `fs.s3a.bucket.*` options. Spark, however, reads the
+ * same table through the vendor's own Hadoop FileSystem, not `S3AFileSystem`, and every S3
+ * divergence gate in this object ([[s3ConfigDivergenceReason]] and its siblings) is verified
+ * against `S3AFileSystem`'s consumers only. With no model of how the vendor filesystem resolves
+ * its configuration, whether native and Spark would agree cannot be decided, so the scan is
+ * declined rather than claimed on a guess. Selected data-file and deletion-vector URIs under an
+ * alias scheme are declined by the generic scheme gates, which never admit an alias (see
+ * [[unsupportedSchemes]]).
+ */
+ private[delta] def s3CompliantAliasSchemeReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val aliases = NativeConfig.resolveS3CompliantSchemes(hadoopConf)
+ if (aliases.isEmpty) {
+ return None
+ }
+ val found = uris
+ .flatMap(uri => Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)))
+ .filter(aliases.contains)
+ .distinct
+ if (found.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support S3-compliant alias filesystem scheme(s) " +
+ s"${found.sorted.mkString(", ")} (${CometConf.COMET_S3_COMPLIANT_SCHEMES_KEY}): " +
+ "Spark reads them through a vendor filesystem whose S3 configuration resolution the " +
+ "native scan's S3AFileSystem divergence model cannot verify")
+ }
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source. The alias
+ * set handed to core's gate is deliberately empty: an `fs.comet.s3Compliant.schemes` alias is
+ * never admitted here (see [[s3CompliantAliasSchemeReason]]), even though core admits it.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri, Set.empty)
Review Comment:
### Correctness
[P2] Preserve the real-path fallback for Delta scans
`isNativelyReadableScheme` now probes a synthetic URL such as `file:///`, so this call no longer checks whether object_store accepts the actual path. Core added a separate `objectStoreAcceptsPath` root gate, but Delta returns from the contrib hook before that gate, and `CometNativeScan.isSupported` does not apply it. With native Delta enabled, an otherwise supported local table under a newline-containing directory (URI such as `file:///tmp/dir%0A/data`) is claimed here; the encoded filename reaches native planning, where object_store 0.13.2 decodes `%0A` and rejects the control character. Spark's filesystem reader would have remained usable.
The newly introduced case is a fresh JVM with a cold file-scheme cache: the previous helper tested that actual URI and declined it. A previously warmed cache could already hide the old path rejection. Please mirror core's actual-root-path fallback, respecting libhdfs exemptions, and add a Delta regression asserting Spark fallback and the correct answer. This finding is verified from the exact source and locked parser; no full query reproduction was executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3826287506
##########
native/core/src/execution/planner/delta_spark_scan.rs:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! JVM-planned Delta handler for the generic `OpStruct::ContribScan` dispatcher, feature-gated
+//! behind `delta`.
+//!
+//! delta-spark has already done log replay, snapshot resolution, and partition pruning by the
+//! time the scan reaches Comet, so the envelope carries a concrete file list (plus deletion
+//! vector descriptors) and the read path reuses the exact same shared parquet scan builder as
+//! `NativeScan` -- inheriting row-group stats pruning, page-index pruning, and filter pushdown.
+//! Sibling of the kernel-planned handler in `delta_scan.rs`; the two claim different
+//! `type_url`s within the same `ContribScan` envelope.
+
+use std::collections::HashMap;
+
+use datafusion_comet_proto::spark_operator::{
+ ContribScan, DeltaSparkScan, Operator, SparkFilePartition,
+};
+use prost::Message;
+
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::execution::planner::PhysicalPlanner;
+use crate::execution::planner::PlanCreationResult;
+
+/// Type name the JVM-planned Delta contrib claims within the `ContribScan` envelope. The
+/// contrib jar packs a `DeltaSparkScan` with a `type_url` of
+/// `type.googleapis.com/comet.contrib.delta_spark.DeltaSparkScan`; dispatch keys on the
+/// contrib-owned suffix, same convention as the kernel path's `delta_scan.rs`.
+const DELTA_SPARK_SCAN_TYPE_NAME: &str = "comet.contrib.delta_spark.DeltaSparkScan";
+
+/// Contrib entry point for the `OpStruct::ContribScan` dispatcher. Returns `Some(result)` when
+/// the envelope carries a JVM-planned Delta scan, or `None` when the `type_url` belongs to some
+/// other contrib.
+pub(crate) fn try_plan_contrib_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ contrib: &ContribScan,
+) -> Option<PlanCreationResult> {
+ if !contrib.type_url.ends_with(DELTA_SPARK_SCAN_TYPE_NAME) {
+ return None;
+ }
+ Some(
+ DeltaSparkScan::decode(contrib.value.as_slice())
+ .map_err(|e| {
+ GeneralError(format!(
+ "Failed to decode DeltaSparkScan from contrib_scan: {e}"
+ ))
+ })
+ .and_then(|scan| plan_delta_spark_scan(planner, spark_plan, &scan)),
+ )
+}
+
+fn plan_delta_spark_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ scan: &DeltaSparkScan,
+) -> PlanCreationResult {
+ // Delta data files are plain parquet; the read path deliberately reuses
+ // the same shared parquet scan builder as NativeScan so Delta inherits
+ // row-group stats pruning, page-index pruning, and filter pushdown. Only
+ // the file list arrives in Delta-specific form. Note delta_common's
+ // column_mapping_mode is informational in M1: the actual field-id
+ // matching switch is common.use_field_id, same as the Iceberg path.
+ let common = scan
+ .common
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing common data".into()))?;
+
+ let delta_partition = scan
+ .file_partition
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing file_partition".into()))?;
+
+ let spark_partition = SparkFilePartition {
+ partitioned_file: delta_partition
+ .partitioned_file
+ .iter()
+ .map(|f| {
+ f.file
+ .clone()
+ .ok_or_else(|| GeneralError("DeltaSparkPartitionedFile missing inner file".into()))
+ })
+ .collect::<Result<Vec<_>, _>>()?,
+ };
+
+ let (object_store_url, mut files) =
+ planner.prepare_scan_store_and_files(common, &spark_partition)?;
Review Comment:
Went with decline rather than per-file routing. The claim gate now compares the full lowercased URI authority (userinfo included, so abfss containers count as distinct) across all selected files, with a native check as backstop, and file order is never changed on the native side. One residual worth flagging: the shared object-store cache key in core drops userinfo, so a DV sidecar on a different container of the same account could still resolve through the wrong store. That's in the shared store layer and predates this PR, so I left it as a core follow-up rather than changing cache semantics here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3826305642
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
Review Comment:
Went with derive rather than decline: options are now unioned over the data-file authority, the table root, and every on-disk DV authority, deduped per authority. The extracted prefixes are scheme-disjoint so the union can't collide. Cross-provider coverage is a unit test (S3 data plus ABFS sidecar, asserting both prefixes appear and that an S3-only set gets no azure keys), since a same-scheme MinIO scenario wouldn't prove anything, option extraction is scheme-global.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4157738644
##########
native/core/src/parquet/objectstore/s3.rs:
##########
@@ -1002,10 +1037,97 @@ impl CredentialProviderMetadata {
#[cfg(test)]
mod tests {
+ use std::collections::BTreeSet;
use std::sync::atomic::{AtomicI32, Ordering};
use super::*;
+ /// Discovery-harness test (see `NATIVE_S3A_CONFIG_PROPERTIES`'s doc): mechanically re-derives
+ /// the set of `fs.s3a.*` property suffixes this file actually resolves by scanning this
+ /// file's OWN source text (via `include_str!`) for every `get_config(configs, bucket, ...)`/
+ /// `get_config_trimmed(configs, bucket, ...)` call site, resolving an identifier argument
+ /// (e.g. `PROVIDER_CLASS_PROPERTY`) through its own `const NAME: &str = "..."` definition, and
+ /// asserts the result is EXACTLY `NATIVE_S3A_CONFIG_PROPERTIES`. This fails loudly the moment
+ /// a call site is added, removed, or its literal changes without updating that constant --
+ /// which is exactly the class of bug (a config key silently added to one side of the
+ /// Scala/Rust boundary but not the other) that let a Hadoop-side resolution rule diverge
+ /// unnoticed in the round-15 SSE-C finding.
+ ///
+ /// The `configs, property` call inside `get_config_trimmed`'s own body (a passthrough of its
+ /// own `property` parameter, not a call site naming a fixed config key) is deliberately
+ /// excluded by name.
+ #[test]
+ fn native_s3a_config_properties_matches_call_sites() {
Review Comment:
> Would it make sense to run that one discovery-harness test in the PR tier whenever `native/core/src/parquet/objectstore/**` changes, so the first signal is not the queue? Or could the contrib comparator be made advisory for keys it does not need to compare?
I went a third way. The comparator has to stay strict, since a key native reads but the claim gate doesn't compare is how a claimed scan ends up with a different endpoint or credentials than Hadoop. And running the Scala test on PRs means a `-Pdelta` build for every objectstore change.
32eb96066 puts the check in the Rust tests instead. `delta_contrib_compares_every_native_s3a_property` reads `S3ConfigKeyConsumers` from `DeltaScanSupport.scala` as text and fails when a key in `NATIVE_S3A_CONFIG_PROPERTIES` is missing. The PR run that asks for the constant update now names the Delta list too, with no new job. For the other direction, a suite test checks that the same text parse equals the compiled `AllS3ConfigKeys`, so a Delta-only reformat of that block fails in its own queue run. What text can't check is whether a new key got the right consumer tier, so the failure message says how to pick it.
Separately, `claim runs before core's metadata-column guard` fails on this head with `column types must match schema types ... "dataChange"`. That's #6334: it passes twice with #6458 applied and fails twice without it. So `delta_3_5` in the queue needs #6458 to land first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3998601057
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ *
+ * `private[delta]` (not `private`): [[DeltaScanSupport.declineReason]]'s non-ASCII
+ * case-insensitive name gate reuses this exact conversion to compute the names native sees
+ * under column mapping, rather than re-deriving physical names with separate logic.
+ */
+ private[delta] def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. Strip the parquet.field.id metadata
+ // createPhysicalSchema also stamps: files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations.
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully. `memo` is the same claim-memo instance
+ * [[DeltaScanSupport.declineReason]] populated on this claim; its `hadoopConf` and
+ * `dvDescriptors` are reused here rather than recomputed.
+ */
+ def convert(
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaScanSupport.DeltaClaimMemo): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = memo.hadoopConf
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = toPhysical(scanExec, relation.partitionSchema),
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Already forced by declineReason on this claim; reused rather than deserialized again.
+ val dvDescriptors = memo.dvDescriptors
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme.
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
+
+ val common = commonBuilder.build()
+ // Effective session rebase read modes, resolved through ParquetOptions exactly as
+ // ParquetFileFormat.buildReaderWithPartitionValues resolves them (per-relation
+ // `datetimeRebaseMode` / `int96RebaseMode` options win over the session conf, whose
+ // per-Spark-version default -- EXCEPTION on 3.x, CORRECTED on 4.0 -- SQLConf supplies).
+ // Native consults them only for files whose footer metadata does not decide the rebase
+ // policy on its own, mirroring DataSourceUtils.getRebaseSpec's modeByConfig fallback.
+ val parquetReadOptions =
+ new org.apache.spark.sql.execution.datasources.parquet.ParquetOptions(
+ relation.options,
+ scanExec.conf)
+ val deltaCommon = OperatorOuterClass.DeltaSparkScanCommon
+ .newBuilder()
+ .setTableRoot(tableRoot)
+ .setColumnMappingMode(columnMappingMode(scanExec))
+ .setSourceKey(DeltaPlanDataInjector.sourceKey(tableRoot, common))
+ .setDatetimeRebaseModeInRead(parquetReadOptions.datetimeRebaseModeInRead)
+ .setInt96RebaseModeInRead(parquetReadOptions.int96RebaseModeInRead)
+ .build()
+ val deltaScan = OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaCommon)
+ Operator
+ .newBuilder()
+ .setPlanId(scanExec.id)
+ .setContribScan(DeltaSparkScanEnvelope.pack(deltaScan.build()))
+ .build()
+ }
+ }
+
+ /**
+ * One representative store URI per distinct object-store authority this scan's partitions may
+ * need options for: the data-file authority (`firstFileUri`), the table root unconditionally
+ * (UUID-relative DV sidecars resolve against it), and every distinct on-disk DV authority from
+ * `descriptors` (inline DVs carry no external URI and are filtered out). Deduping by authority
+ * rather than full URI keeps this O(distinct authorities) instead of O(files), keeping the
+ * FIRST URI seen per authority so `firstFileUri`/the table root win over a same-authority DV
+ * path.
+ */
+ private[delta] def storeUris(
+ descriptors: Seq[DeletionVectorDescriptor],
+ tableRootPath: Path,
+ firstFileUri: Option[java.net.URI]): Seq[java.net.URI] = {
+ val dvAuthorityUris = descriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(tableRootPath).toUri)
+ val candidates = firstFileUri.toSeq ++ Seq(tableRootPath.toUri) ++ dvAuthorityUris
+ val byAuthority = scala.collection.mutable.LinkedHashMap.empty[String, java.net.URI]
+ candidates.foreach(uri =>
+ byAuthority.getOrElseUpdate(DeltaScanSupport.uriAuthority(uri), uri))
+ byAuthority.values.toSeq
+ }
+
+ /**
+ * Unions `NativeConfig.extractObjectStoreOptions` over every `uris` authority. Safe to union
+ * rather than pick one: extracted keys are scheme-disjoint prefixes (`fs.s3a.*` vs
+ * `fs.azure.*`, ...), so options for different schemes never collide, and re-extracting the
+ * same scheme from two URIs is idempotent.
+ */
+ private[delta] def mergedObjectStoreOptions(
+ hadoopConf: org.apache.hadoop.conf.Configuration,
+ uris: Seq[java.net.URI]): Map[String, String] =
+ uris.foldLeft(Map.empty[String, String]) { (merged, uri) =>
+ merged ++ NativeConfig.extractObjectStoreOptions(hadoopConf, uri)
+ }
+
+ /**
+ * Harvest subquery-bearing predicates for this scan from its covering FilterExec. Spark 3.x
+ * strips them from a scan's `dataFilters` at planning (`FileSourceStrategy` routes them to the
+ * post-scan filter only), while Spark 4.x keeps them in `dataFilters`; collecting them here at
+ * claim time gives the execution-time resolve-and-push path the same inputs on every version,
+ * and the dedup below keeps Spark 4.x from carrying duplicates. Reference containment alone
+ * does not prove pushing a predicate down is semantics-preserving, so `spineToScan` also
+ * requires every intervening operator to commute with the push (an intervening
+ * LIMIT/Sort/Aggregate/join etc. stops the walk and leaves the filter where Spark placed it:
+ * missed pruning only).
+ */
+ def subqueryFiltersFromParent(
+ plan: org.apache.spark.sql.execution.SparkPlan,
+ scanExec: FileSourceScanExec): Seq[org.apache.spark.sql.catalyst.expressions.Expression] = {
+ import org.apache.spark.sql.catalyst.expressions.{PlanExpression, SubqueryExpression}
+ import org.apache.spark.sql.execution.{FilterExec, ProjectExec, SparkPlan}
+
+ // Whether every node from `node` down to `scanExec` is one pushdown can safely cross: a
+ // deterministic ProjectExec is 1:1 on rows and a deterministic FilterExec only removes rows,
+ // so moving a predicate over the scan's output through either preserves semantics -- mirroring
+ // Spark's own PushPredicateThroughNonJoin/CollapseProject rules. A nondeterministic node (or
+ // anything else: LIMIT/TopN, Sort, Aggregate, Window, joins, ...) can change which rows survive
+ // to matter, so it stops the walk and the filter is left uncollected (missed pruning only).
+ def spineToScan(node: SparkPlan): Boolean = node match {
+ case n if n eq scanExec => true
+ case p: ProjectExec if p.projectList.forall(_.deterministic) => spineToScan(p.child)
+ case f: FilterExec if f.condition.deterministic => spineToScan(f.child)
+ case _ => false
+ }
+
+ // Nearest FilterExec whose spine down to the scan is Project/Filter-only (the DV shape
+ // interposes such nodes between them, so do not require a direct parent-child edge).
+ val filtersAboveScan = plan.collect {
+ case f: FilterExec if spineToScan(f.child) => f
+ }
+ filtersAboveScan.lastOption
+ .map { f =>
+ splitConjunctivePredicates(f.condition)
+ .filter(_.deterministic)
+ .filter(_.references.subsetOf(scanExec.outputSet))
+ .filter(p =>
+ SubqueryExpression.hasSubquery(p) || p.exists(_.isInstanceOf[PlanExpression[_]]))
+ .filterNot(p => scanExec.dataFilters.exists(_.semanticEquals(p)))
+ }
+ .getOrElse(Seq.empty)
+ }
+
+ /**
+ * Execution-time scalar-subquery data filters of a scan. `hasResolvedFilters` is true whenever
+ * pushdown is enabled and such filters exist, whether or not they bind or serialize; `protos`
+ * holds only the ones that serialized.
+ */
+ case class ResolvedSubqueryFilters(
+ hasResolvedFilters: Boolean,
+ protos: Seq[org.apache.comet.serde.ExprOuterClass.Expr])
+
+ private val NoResolvedSubqueryFilters = ResolvedSubqueryFilters(false, Seq.empty)
+
+ /**
+ * Resolve scalar-subquery data filters at execution time and serialize them for native
+ * pushdown, mirroring `CometNativeScanExec.serializedPartitionData`. `supportedDataFilters`
+ * excludes PlanExpressions at planning time (subquery results do not exist yet), so these
+ * bounds reach the native reader only through this path. Filters that fail to serialize are
+ * skipped: Spark keeps a covering FilterExec above the scan, so this is missed pruning only.
+ * Their presence is still reported, since native keys the safe timestamp conversion on the scan
+ * being filtered at all, as the core scan does for its resolved filters.
+ *
+ * Known core-parity limitation: when fused under a parent native operator,
+ * `ensureSubqueriesResolved` has already called `updateResult()` on these subqueries and this
+ * path calls it again (`ScalarSubquery.updateResult` re-executes unconditionally); benign here
+ * since the subquery's snapshot is pinned at analysis, but wasteful. Fix belongs in core.
+ */
+ def resolvedSubqueryFilters(
+ dataFilters: Seq[org.apache.spark.sql.catalyst.expressions.Expression],
+ output: Seq[org.apache.spark.sql.catalyst.expressions.Attribute],
+ requiredSchema: StructType,
+ conf: org.apache.spark.sql.internal.SQLConf): ResolvedSubqueryFilters = {
+ if (!conf.getConf(org.apache.spark.sql.internal.SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ return NoResolvedSubqueryFilters
+ }
+ val subqueryFilters = dataFilters.filter(_.exists(_.isInstanceOf[ExecScalarSubquery]))
+ if (subqueryFilters.isEmpty) {
+ return NoResolvedSubqueryFilters
+ }
+ // Same binding guard as the DV shape's plan-time filters: references limited to the
+ // data-column prefix of the output, where positions agree with the native read schema.
+ // Guard BEFORE updateResult so discarded filters never execute their subqueries.
+ val strippedLen = requiredSchema.count(f => !internalColumnNames.contains(f.name))
+ val dataColIds = output.take(strippedLen).map(_.exprId).toSet
+ val pushableFilters =
+ subqueryFilters.filter(_.references.forall(r => dataColIds.contains(r.exprId)))
+ pushableFilters.foreach(_.foreach {
+ case s: ExecScalarSubquery => s.updateResult()
+ case _ =>
+ })
+ val protos = pushableFilters
+ .flatMap { filter =>
+ // MergeScalarSubqueries can fuse several scalar subqueries into one struct-returning
+ // subquery accessed via GetStructField; fold that whole subtree to a literal (a bare
+ // GetStructField-over-Literal would not serialize).
+ val resolved = filter.transform {
+ case g @ org.apache.spark.sql.catalyst.expressions
+ .GetStructField(_: ExecScalarSubquery, _, _) =>
+ Literal.create(g.eval(null), g.dataType)
+ case s: ExecScalarSubquery =>
+ Literal.create(s.eval(null), s.dataType)
+ }
+ val proto = exprToProto(resolved, output)
+ if (proto.isEmpty) {
+ logWarning(s"Could not serialize resolved scalar subquery filter: $resolved")
+ }
+ proto
+ }
+ ResolvedSubqueryFilters(hasResolvedFilters = true, protos)
+ }
+
+ /**
+ * Allocate the partition-schema slots for the DV shape's internal columns
+ * (`internalColumnNames`), with names collision-free against the physical data schema, the
+ * physical partition schema, and the constant-metadata slots already allocated for this scan
+ * (plus each other): DataFusion substitutes partition constants BY NAME, so an unprefixed,
+ * un-uniquified slot could collide with a real column and silently replace its data with the
+ * bookkeeping constant. `buildDvScanCommon` keys `internalIndexByName` by each field's ORIGINAL
+ * name from `requiredSchema`, so the renaming here only changes the proto's field name.
+ */
+ private[delta] def allocateUniqueInternalFields(
+ requiredSchema: StructType,
+ physicalDataSchema: StructType,
+ physicalPartitionSchema: StructType,
+ constantMetadataFields: Seq[StructField]): Seq[StructField] = {
+ val reserved = scala.collection.mutable.LinkedHashSet[String]()
+ reserved ++= physicalDataSchema.fields.map(_.name)
+ reserved ++= physicalPartitionSchema.fields.map(_.name)
+ reserved ++= constantMetadataFields.map(_.name)
+ requiredSchema.fields.toSeq
+ .filter(f => internalColumnNames.contains(f.name))
+ .map { f =>
+ var name = s"$deltaConstFieldPrefix${f.name}"
+ while (reserved.contains(name)) {
+ name = name + "_"
+ }
+ reserved += name
+ StructField(name, f.dataType, f.nullable)
+ }
+ }
+
+ /**
+ * DV shape common builder. Layout invariants (declined by DeltaScanSupport when violated): scan
+ * output = requiredSchema attrs (data columns, then the internal columns as a suffix) followed
+ * by partition and constant-metadata columns. The parquet read schema strips the internal
+ * columns; they are appended to the partition schema as per-file constants, so the projection
+ * vector routes them from the constants block.
+ */
+ private def buildDvScanCommon(
Review Comment:
### 3. Reuse ordinary-column construction in the DV builder
Could ordinary-column construction in this path share the same builder as `CometNativeScan.buildNativeScanCommon`, with an explicit input or result describing the generated columns and visible projection? Configuration flags and collision-free metadata-name allocation are already shared; filter binding, output types, schemas, and projection assembly still have a separate DV implementation.
One possible approach is to build a view without generated columns, reuse the common builder, and then append their slots and restore output order by attribute identity. A smaller extraction is also reasonable if it gives ordinary and DV scans one implementation of the shared serialization rules.
This can preserve the current suffix requirement, defaults fallback, and DML/liveness checks. Could tests compare ordinary-column construction across both paths and retain predicate-presence, projection-order, and synthetic-name collision coverage? That would let later serializer fixes reach both paths without expanding DV admission in this PR.
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/DeltaPlanDataInjector.scala:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.{OperatorOuterClass, QueryContextInterner}
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * PlanDataInjector for the Delta contrib scan, discovered by core's ServiceLoader (see the
+ * `META-INF/services` resource). Lives in this package because [[PlanDataInjector]] is
+ * `private[comet]`.
+ */
+class DeltaPlanDataInjector extends PlanDataInjector {
+
+ override val opStructCase: Operator.OpStructCase = Operator.OpStructCase.CONTRIB_SCAN
+
+ override def canInject(op: Operator): Boolean =
Review Comment:
### 6. Add focused envelope and provider contract tests
As a non-blocking maintainability improvement, could we add focused tests for the serialized Delta Spark envelope and the real `DeltaPlanDataInjector` through the registry?
These could assert operator slot 200, the distinct Spark/Kernel type URLs, and expected schema, projection, and filter values in a representative common payload. Provider tests could check common/partition assembly, preservation of outer operator fields and children, rejection of another contrib's envelope, and an empty injected partition no longer qualifying for injection.
The generic suites and successful scan tests already exercise the extension machinery, and injection completion and native feature-off rejection are implemented. These additional assertions would pin the Delta-specific boundary during refactoring without introducing a new capability API or a general mixed-version compatibility commitment.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1838 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
Review Comment:
### 2. Share common native-reader admission checks
Could the input-file-expression, vectorized-reader-setting, and nested-default checks come from a shared native-reader admission helper? These checks mirror core today, so a future fix otherwise needs a separate Delta update.
The shared entry point can leave Delta's protocol checks, DV row-index exception, liveness checks, and stricter encryption fallback explicit in the contrib. Filesystem policy also differs; its broader consolidation can stay in #5658. The aim here is to share the checks that already have the same semantics.
A parity test covering the common rejection cases for ordinary Parquet and Delta, together with supported-scan controls, would protect this boundary. Existing Delta-specific metadata fallbacks should remain covered separately.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ *
+ * `private[delta]` (not `private`): [[DeltaScanSupport.declineReason]]'s non-ASCII
+ * case-insensitive name gate reuses this exact conversion to compute the names native sees
+ * under column mapping, rather than re-deriving physical names with separate logic.
+ */
+ private[delta] def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. Strip the parquet.field.id metadata
+ // createPhysicalSchema also stamps: files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations.
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully. `memo` is the same claim-memo instance
+ * [[DeltaScanSupport.declineReason]] populated on this claim; its `hadoopConf` and
+ * `dvDescriptors` are reused here rather than recomputed.
+ */
+ def convert(
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaScanSupport.DeltaClaimMemo): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = memo.hadoopConf
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
Review Comment:
### 1. Use a stable scan identity for task injection
Could `common.source` use the original physical scan's `SparkPlan.id` in both builders, for example `s"${scanExec.nodeName} (${scanExec.id})"`? `simpleStringWithNodeId()` reads Spark's explain-local ID map and can produce `(unknown)` during ordinary planning.
Two independently converted scans of the same table, with identical common fields but partition selections `p = 1` and `p = 2`, can therefore receive the same injection key. Partition predicates are absent from the key; if both scans contribute to one collection scope, the map merge retains only one payload. Exchange boundaries isolate common self-join plans, so this is a key-contract concern rather than an established end-to-end SQL wrong answer.
Could a regression clear the explain map, use disjoint selected files, and assert distinct keys? An equivalent-scan control should preserve `sameResult` and semantic hashes. The key should remain excluded from semantic equality, as it is today.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ *
+ * `private[delta]` (not `private`): [[DeltaScanSupport.declineReason]]'s non-ASCII
+ * case-insensitive name gate reuses this exact conversion to compute the names native sees
+ * under column mapping, rather than re-deriving physical names with separate logic.
+ */
+ private[delta] def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
Review Comment:
### 9. Reuse Delta's schema preparation and retain a logical-output boundary
Could `toPhysical` delegate to `DeltaParquetFileFormat.prepareSchemaForRead` instead of duplicating `createPhysicalSchema` and name-mode field-ID stripping? That helper exists in the supported Delta versions and is used by Delta for its data, required, and partition schemas. Generated bookkeeping columns should remain excluded from the physical schema passed to it.
For the [existing nested-mapping follow-up](https://github.com/apache/datafusion-comet/pull/5365#discussion_r3826296728), could we retain a clear division between those physical read schemas and the logical output types already present in `common.fields`? Native output adaptation can restore logical nested types before parent expressions run; mapped complex predicates should remain residual until they bind safely to the physical schema.
The current ID-mode and mapped-nested fallbacks can remain until that adaptation is implemented and tested. Useful follow-up controls include upgrade/rename, `to_json` of mapped structs, missing nested children, and version-specific null-struct behavior. #5661 can document the support boundary and link the implementation work.
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,2096 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::mem::size_of;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::{ParquetAccessPlan, RowGroupAccess};
+use datafusion::execution::memory_pool::{MemoryConsumer, MemoryReservation};
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::path::Path;
+use object_store::{ObjectStore, ObjectStoreExt};
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::{PageIndexPolicy, ParquetMetaData};
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ // A corrupt footer can report a negative row count. `num_rows as u64` would otherwise
+ // wrap it into a huge positive value, silently corrupting every row-group boundary
+ // computed from `group_start`/`group_end` below (and therefore which deleted row indexes
+ // land in which row group) instead of failing loudly.
+ if num_rows < 0 {
+ return Err(GeneralError(format!(
+ "Parquet footer reports a negative row count ({num_rows}) for row group {idx}"
+ )));
+ }
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
+ }
+ cursor = row + 1;
+ }
+ if deleted_in_group == num_rows && num_rows > 0 {
+ plan.skip(idx);
+ } else if deleted_in_group > 0 {
+ if group_end > cursor {
+ selectors.push(RowSelector::select((group_end - cursor) as usize));
+ }
+ plan.scan_selection(idx, RowSelection::from(selectors));
+ }
+ group_start = group_end;
+ }
+ // A deleted index beyond the file's total row count means the DV does not
+ // belong to this file (stale or corrupted metadata); silently dropping it
+ // would under-apply deletions.
+ if let Some(&row) = deleted_iter.peek() {
+ return Err(GeneralError(format!(
+ "Deletion vector marks row {row} but the file only has {group_start} rows"
+ )));
+ }
+ Ok(plan)
+}
+
+/// Verify a decoded deletion vector's row count matches the descriptor's
+/// declared `cardinality`, mirroring Delta's JVM reader
+/// (`StoredBitmap.validateCardinality`). The CRC and framing checks catch
+/// corruption but not a stale, otherwise well-formed bitmap whose row count
+/// no longer matches the descriptor -- that would silently under- or
+/// over-delete rows.
+fn validate_cardinality(
+ file_path: &str,
+ expected: i64,
+ deleted: &RoaringTreemap,
+) -> Result<(), ExecutionError> {
+ let actual = deleted.len();
+ if actual != expected as u64 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has cardinality mismatch: descriptor says {expected}, decoded bitmap has {actual} deleted rows"
+ )));
+ }
+ Ok(())
+}
+
+/// One data file plus everything needed to apply its deletion vector. The
+/// file's size comes from `file.object_meta.size` (built by the planner from
+/// the proto's `file_size`).
+///
+/// `data_store` and `dv_store` are resolved by the caller *before* entering
+/// the async `attach_access_plans` runtime (see its doc comment): building an
+/// object store is sync I/O that, for a cold S3 authority, internally issues
+/// its own `Handle::block_on` calls, which panics if nested inside another
+/// `block_on`. Resolving up front means this module never constructs a
+/// store itself.
+pub struct DvScanFile {
+ pub file: PartitionedFile,
+ /// Full URL of the data file (proto `file_path`).
+ pub file_path: String,
+ pub dv: Option<DeltaSparkDvDescriptor>,
+ /// Object store for `file_path`, pre-resolved by the caller. Only read
+ /// when `dv` is `Some` (files without a deletion vector never open their
+ /// footer here), but every file carries one so the struct's shape
+ /// doesn't depend on whether a deletion vector is present.
+ pub data_store: Arc<dyn ObjectStore>,
+ /// Store and within-store path for an on-disk deletion vector's absolute
+ /// path, pre-resolved by the caller. `None` when the file has no
+ /// deletion vector or the deletion vector is stored inline.
+ pub dv_store: Option<(Arc<dyn ObjectStore>, Path)>,
+}
+
+/// Execution-memory-pool reservation covering one file's expanded DV row selectors across
+/// their *entire* lifetime attached to a scan -- from `build_access_plan`'s construction
+/// through DataFusion 54.1's reader normalizing the attached [`ParquetAccessPlan`]
+/// (`create_initial_plan`'s deep clone plus `into_overall_row_selection`'s combined
+/// `RowSelection`; see [`reader_peak_bytes`]) -- attached to the file's [`PartitionedFile`]
+/// extensions alongside its [`ParquetAccessPlan`]. The reservation's lifetime is tied to the
+/// `PartitionedFile` it is attached to, so it is released back to the pool exactly when the
+/// plan is dropped (query completion or an early-terminated scan), never held open longer.
+/// Newtype-wrapped so it occupies its own slot in the multi-slot, type-keyed `extensions` map
+/// (`datafusion_common::extensions::Extensions`) alongside the plan, rather than a bare
+/// `MemoryReservation` colliding with one some other extension might attach.
+pub struct DvAccessPlanReservation(pub MemoryReservation);
+
+/// Total number of [`RowSelector`]s materialized across `plan`'s per-row-group
+/// selections (`RowGroupAccess::Selection`); `Scan`/`Skip` row groups
+/// contribute none. An alternating deleted/retained bitmap produces one
+/// non-coalescing selector per row (see [`reader_peak_bytes`]'s doc comment
+/// for the worst-case accounting), so this count -- not the deletion
+/// vector's cardinality -- is the thing that must be bounded and reserved
+/// against the execution memory pool.
+fn total_selectors(plan: &ParquetAccessPlan) -> usize {
+ plan.inner()
+ .iter()
+ .map(|access| match access {
+ RowGroupAccess::Selection(selection) => selection.iter().count(),
+ _ => 0,
+ })
+ .sum()
+}
+
+/// Multiplier bounding the peak allocation live *during construction* of one
+/// file's [`RowSelection`]s, relative to the conservative selector-count
+/// bound `S = 2 * cardinality + num_row_groups` (one non-coalescing selector
+/// per deleted row in the worst-case alternating pattern, doubled, plus up to
+/// one extra boundary selector per row group). Split `S` into `r`, the
+/// selectors already retained from row groups `build_access_plan` has
+/// finished, and `c`, the selectors accumulated so far in the current row
+/// group's source `Vec`; `r` and `c` partition the selectors counted toward
+/// `S`, so `r + c <= S` always. While the current group is being built, the
+/// `Vec`'s doubling growth strategy can leave its backing allocation at up to
+/// `2 * c` (the next power-of-two capacity above `c`). Once the group
+/// finishes, `RowSelection::from(Vec)` (parquet's `FromIterator` impl,
+/// `with_capacity` + copy) builds a second, separate `Vec` of size `c` from
+/// that source while the source is still alive, so at the moment the copy
+/// begins, the retained selectors, the current group's doubled source `Vec`,
+/// and the copy are all live simultaneously: `r + 2c + c = r + 3c`. Since
+/// `r >= 0`, `r + 3c <= 3r + 3c = 3(r + c) <= 3S`. 3x covers that peak.
+const CONSTRUCTION_PEAK_FACTOR: usize = 3;
+
+/// Upper bound on how much larger a `Vec`'s backing allocation can be than its element count
+/// after being built by repeated pushes: `std`'s doubling growth strategy never leaves a `Vec`
+/// of `n` elements with a backing allocation larger than the next power of two above `n`, which
+/// is at most `2 * n` for any `n >= 1`.
+const VEC_GROWTH_CAPACITY_FACTOR: usize = 2;
+
+/// `RawVec`'s minimum non-zero capacity for element sizes `<= 1024` bytes ([`RowSelector`] is
+/// 16 bytes on 64-bit platforms: a `usize` row count plus a padded `bool`). Applied once per
+/// row group (or per contiguous run of row groups) a fresh `from_fn`/`FlatMap`-driven `Vec`
+/// gets built for (see [`reader_peak_bytes`]), so even a group or run whose true selector count
+/// is tiny still pays this floor.
+const MIN_VEC_CAPACITY_SELECTORS: usize = 4;
+
+/// Conservative upper bound, in bytes, on the peak allocation live while DataFusion 54.1's
+/// reader normalizes one file's attached [`ParquetAccessPlan`] -- the allocation this module's
+/// steady-state reservation must cover, not merely the plan's own retained selector bytes.
+/// THREE allocations can be live simultaneously by the time `into_overall_row_selection`
+/// returns, not two -- the clone is only exact when page-index pruning never touches it:
+///
+/// 1. **Attached original** (`selectors`, exact): `create_initial_plan` deep-clones the
+/// attached plan while the original remains reachable from the file's `extensions` until
+/// the scan consumes it. The ORIGINAL's own selector `Vec`s are exact -- a coalesced
+/// [`RowSelection`] built via `RowSelection::from(Vec<RowSelector>)` (what
+/// `build_access_plan` uses) has no excess capacity, because that conversion is a plain
+/// `with_capacity(len)` copy, not a `size_hint`-blind fold.
+/// 2. **The clone, possibly capacity-inflated** (`<= VEC_GROWTH_CAPACITY_FACTOR * selectors +
+/// MIN_VEC_CAPACITY_SELECTORS * num_row_groups`): if page-index pruning fires
+/// (`PagePruningAccessPlanFilter`; `access_plan.rs`'s `scan_selection` on a row group that
+/// already carries a `RowGroupAccess::Selection` calls `existing.intersection(&page_derived)`
+/// -- `RowSelection::intersection` -> `intersect_row_selections`), it replaces the CLONE's
+/// per-row-group selection with that intersection's output. `intersect_row_selections` is
+/// ANOTHER `from_fn` generator with `size_hint() == (0, None)`, so each intersected row
+/// group's backing `Vec` starts at `with_capacity(0)` and doubles as it grows, independent
+/// of whatever capacity the pre-intersection selection had. This inflated clone is still
+/// live when `into_overall_row_selection` later moves its buffer. Term 1's exactness
+/// guarantee holds for the ORIGINAL always, and for the clone only when page-index pruning
+/// never fires against it -- once it does, the clone must be charged at the SAME
+/// growth-capped bound as a fresh combined-selection `Vec` (term 3), summed once per row
+/// group rather than once per run, since each row group's `Selection` is intersected
+/// independently.
+/// 3. **Per-run combined-selection allocation** (`<= VEC_GROWTH_CAPACITY_FACTOR * (selectors +
+/// num_row_groups) + MIN_VEC_CAPACITY_SELECTORS * num_row_groups`): `into_overall_row_selection`
+/// collects each contiguous run of row groups' selectors into a *new* `RowSelection` via a
+/// `FlatMap` whose `size_hint().0 == 0`, so that run's `Vec` starts at `with_capacity(0)`
+/// and doubles as it grows -- capping its backing allocation at
+/// `max(MIN_VEC_CAPACITY_SELECTORS, next_power_of_two(len))`, which is at most
+/// `MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR * len` for a run of `len`
+/// selectors. `len` is at most that run's share of `selectors` plus one boundary selector
+/// per `RowGroupAccess::Scan` row group in the run (`Scan` always contributes exactly one
+/// `RowSelector::select(num_rows)`; see `access_plan.rs`'s `into_overall_row_selection`).
+/// Summing across at most `num_row_groups` runs (each spans >= 1 row group) bounds the total
+/// at `VEC_GROWTH_CAPACITY_FACTOR * selectors + (MIN_VEC_CAPACITY_SELECTORS +
+/// VEC_GROWTH_CAPACITY_FACTOR) * num_row_groups`.
+///
+/// Summing all three terms and converting to bytes: `((1 + 2 * VEC_GROWTH_CAPACITY_FACTOR) *
+/// selectors + (2 * MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR) * num_row_groups)
+/// * size_of::<RowSelector>()` -- with the constants above, `(5 * selectors + 10 *
+/// num_row_groups) * size_of::<RowSelector>()`. Checked against two measured worst cases:
+///
+/// - No page-index pruning (the original P2 report; term 2 stays exact): one 2,000,000-row
+/// group, 1,000,000 alternating deletions, `selectors = 2,000,000`. Measured allocator peak
+/// 97,554,457 B; the byte-for-byte accounting for the attached original plus the (here,
+/// exact) clone plus the inflated combined selection explains 97,554,432 B of that, a 25 B
+/// residue we did not attribute. This bound gives 160,000,160 B -- much looser here because
+/// it must also cover the next case, where the clone is NOT exact.
+/// - Page-index pruning fires against the clone: one 1,048,577-row group, `selectors =
+/// 1,048,577`. Measured peak 83,886,096 B; this bound gives 83,886,320 B (a 224 B, <1%
+/// margin -- deliberately tight, since this is the case that drives the bound).
+///
+/// Uses checked arithmetic throughout: a selector or row-group count large enough to overflow
+/// `usize` indicates a corrupted or malicious input, reported as a clean error rather than
+/// panicking.
+fn reader_peak_bytes(selectors: usize, num_row_groups: usize) -> Result<usize, ExecutionError> {
+ let overflow = || {
+ GeneralError(format!(
+ "Deletion vector reader-peak bound overflowed for {selectors} selectors and \
+ {num_row_groups} row groups"
+ ))
+ };
+ // Term 1: the attached original -- exact, untouched by page-index pruning (only the clone
+ // is ever intersected; see the doc comment above).
+ let attached_term = selectors;
+ // Term 2: the clone, bounded as if page-index pruning DID fire against every row group
+ // (safe even when it doesn't: term 2's bound is always >= `selectors`, so it never
+ // undershoots the exact case either).
+ let clone_growth = selectors
+ .checked_mul(VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let clone_floor = num_row_groups
+ .checked_mul(MIN_VEC_CAPACITY_SELECTORS)
+ .ok_or_else(overflow)?;
+ let clone_term = clone_growth.checked_add(clone_floor).ok_or_else(overflow)?;
+ // Term 3: into_overall_row_selection's per-run combined-selection allocation.
+ let combined_growth = selectors
+ .checked_mul(VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let combined_floor = num_row_groups
+ .checked_mul(MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let combined_term = combined_growth
+ .checked_add(combined_floor)
+ .ok_or_else(overflow)?;
+
+ let selector_bound = attached_term
+ .checked_add(clone_term)
+ .and_then(|sum| sum.checked_add(combined_term))
+ .ok_or_else(overflow)?;
+ selector_bound
+ .checked_mul(size_of::<RowSelector>())
+ .ok_or_else(overflow)
+}
+
+/// Upper bound, in [`RowSelector`]s, on how many extra selectors the parquet reader's
+/// page-index pruning can add on top of the deletion vector's own selection when normalizing
+/// one file, from that file's already-fetched [`ParquetMetaData`].
+///
+/// `intersect_row_selections` (parquet's `selection.rs`), which combines a page-pruning
+/// selection with the deletion vector's selection, is a `from_fn` generator whose
+/// `size_hint()` is `(0, None)`: for inputs of length `a` and `b`, its output can have up to
+/// `a + b` selectors -- longer than either input. Bounding the page-pruning side of that sum
+/// requires knowing how many selectors a page-index-derived selection could produce: at most
+/// two per data page (one skip, one select, in the worst case of alternating page-level
+/// pruning decisions), summed over every column of every row group.
+///
+/// Returns `0` when `metadata` carries no offset index (`metadata.offset_index()` is `None`).
+/// This is provably safe, not merely a convenient default: page-index pruning cannot produce a
+/// page-level selection without the offset index to locate pages by, so there are no
+/// page-pruning selectors to bound. The offset index is fetched with
+/// `PageIndexPolicy::Optional` from the same `FileMetadataCache` entry the scan's reader later
+/// reopens (see [`attach_access_plan`]'s footer-fetch comment), so this function observes
+/// exactly what the reader will see.
+///
+/// Uses checked arithmetic throughout for the same reason as [`admission_bound_bytes`].
+fn page_selection_bound_selectors(metadata: &ParquetMetaData) -> Result<usize, ExecutionError> {
+ let Some(offset_index) = metadata.offset_index() else {
+ return Ok(0);
+ };
+ let overflow = || {
+ GeneralError(
+ "Deletion vector page-selection bound overflowed while summing offset-index page \
+ locations"
+ .to_string(),
+ )
+ };
+ let mut total_page_locations = 0usize;
+ for row_group in offset_index {
+ for column in row_group {
+ total_page_locations = total_page_locations
+ .checked_add(column.page_locations().len())
+ .ok_or_else(overflow)?;
+ }
+ }
+ total_page_locations.checked_mul(2).ok_or_else(overflow)
+}
+
+/// Execution-memory-pool admission bound, in bytes, for one file's deletion-vector access
+/// plan -- reserved *before* calling `build_access_plan` (see [`attach_access_plan`]'s
+/// pre-reserve call site) to cover the larger of two peaks live at different points in the
+/// plan's lifetime. In practice the reader-normalization peak below dominates the construction
+/// peak unconditionally for any non-trivial input (`reader_peak_bytes(S, G) = (5S + 10G) *
+/// size_of::<RowSelector>()` always exceeds `CONSTRUCTION_PEAK_FACTOR * S *
+/// size_of::<RowSelector>() = 3S * size_of::<RowSelector>()` once `S >= 1`, since the `5S` term
+/// alone already exceeds `3S`); the construction term is retained as a documented floor rather
+/// than dropped, since it is cheap to compute and keeps this bound correct even if the reader's
+/// growth factors ever shrink below construction's.
+///
+/// - **Construction peak** (`CONSTRUCTION_PEAK_FACTOR * S`, see that constant's doc comment):
+/// live while `build_access_plan` builds the plan's `RowSelection`s. Construction's
+/// transient allocations fully unwind before `build_access_plan` returns, so this peak never
+/// overlaps the reader-normalization peak below.
+/// - **Reader-normalization peak** (`reader_peak_bytes(S + page_bound_selectors,
+/// num_row_groups)`, see that function): live later, once DataFusion's reader normalizes the
+/// attached plan. `S = 2 * cardinality + num_row_groups` is the same conservative bound on
+/// the plan's final retained selector count used for the construction peak -- it provably
+/// bounds `R = total_selectors(&plan)` (`R <= S`, from `build_access_plan`'s
+/// one-non-coalescing-selector-per-deleted-row worst case plus one boundary selector per row
+/// group), so `S + page_bound_selectors` bounds `R` after page-index inflation the same way
+/// `S` bounds `R` before it.
+///
+/// These two peaks never overlap in time, so `max` -- not `sum` -- is the correct combinator:
+/// reserving their sum would over-reserve for no safety benefit.
+///
+/// Deliberately not clamped by the file's total row count here, unlike the reader-peak target
+/// `attach_access_plan` resizes down to after construction (see that call site): `S`'s
+/// `+ num_row_groups` boundary term is a worst-case padding margin that can legitimately exceed
+/// the total row count for a small, heavily-deleted file, and admission sizing has no actual
+/// retained-selector count yet to clamp against -- only after construction, once `R` is known,
+/// is clamping to the total row count both meaningful and strictly tighter. Leaving this bound
+/// unclamped only ever makes admission more conservative, never less safe.
+///
+/// Uses checked arithmetic throughout: a cardinality, row-group count, or page bound large
+/// enough to overflow `usize` while computing this bound indicates a corrupted or malicious
+/// descriptor, reported as a clean error rather than panicking.
+fn admission_bound_bytes(
+ cardinality: i64,
+ num_row_groups: usize,
+ page_bound_selectors: usize,
+) -> Result<usize, ExecutionError> {
+ let overflow = || {
+ GeneralError(format!(
+ "Deletion vector admission bound overflowed for cardinality {cardinality}, \
+ {num_row_groups} row groups, and page bound {page_bound_selectors} selectors"
+ ))
+ };
+ let cardinality_usize = usize::try_from(cardinality).map_err(|_| overflow())?;
+ // S: the conservative bound on the plan's final *retained* selector count (what
+ // `total_selectors(&plan)` cannot exceed) -- unchanged from the pre-existing
+ // construction-only bound this function replaces.
+ let s = cardinality_usize
+ .checked_mul(2)
+ .and_then(|doubled| doubled.checked_add(num_row_groups))
+ .ok_or_else(overflow)?;
+
+ let construction_bytes = s
+ .checked_mul(size_of::<RowSelector>())
+ .and_then(|bytes| bytes.checked_mul(CONSTRUCTION_PEAK_FACTOR))
+ .ok_or_else(overflow)?;
+
+ let s_plus_page = s.checked_add(page_bound_selectors).ok_or_else(overflow)?;
+ let reader_bytes = reader_peak_bytes(s_plus_page, num_row_groups)?;
+
+ Ok(construction_bytes.max(reader_bytes))
+}
+
+/// Upper bound on concurrent DV-blob and footer fetches per partition. Both
+/// are small ranged reads, so a modest fan-out hides object-store latency
+/// without flooding the store client.
+const DV_FETCH_CONCURRENCY: usize = 8;
+
+/// Called via `block_on` at plan-creation time on the executor task: DV blobs
+/// are small ranged reads and footers are needed to learn row-group
+/// boundaries. Files are fetched concurrently (bounded by
+/// [`DV_FETCH_CONCURRENCY`]) with input order preserved. Footer fetches go
+/// through the scan's shared FileMetadataCache, so the scan's subsequent open
+/// of the same file is served from cache. That reuse relies on each input
+/// [`PartitionedFile`] being returned as-is (only `with_extension` applied),
+/// never rebuilt: the cache entry is keyed by this exact `object_meta` and the
+/// scan later looks it up through the same struct.
+///
+/// Deliberately takes no object-store options map and imports no
+/// store-construction helper: every [`DvScanFile`] arrives with its stores
+/// already resolved by the caller (see its doc comment), so this async path
+/// structurally cannot build an object store -- only `runtime_env` is still
+/// threaded through, for the shared `FileMetadataCache` and (per file) the
+/// execution `MemoryPool` each expanded access plan's row selectors are
+/// reserved against -- see [`DvAccessPlanReservation`].
+pub async fn attach_access_plans(
+ runtime_env: Arc<RuntimeEnv>,
+ files: Vec<DvScanFile>,
+) -> Result<Vec<PartitionedFile>, ExecutionError> {
+ futures::stream::iter(files)
+ .map(|scan_file| attach_access_plan(Arc::clone(&runtime_env), scan_file))
+ .buffered(DV_FETCH_CONCURRENCY)
+ .try_collect()
+ .await
+}
+
+/// Resolve one file's deletion vector into an attached [`ParquetAccessPlan`];
+/// files without a DV pass through untouched.
+async fn attach_access_plan(
+ runtime_env: Arc<RuntimeEnv>,
+ scan_file: DvScanFile,
+) -> Result<PartitionedFile, ExecutionError> {
+ let DvScanFile {
+ file,
+ file_path,
+ dv,
+ data_store,
+ dv_store,
+ } = scan_file;
+ let dv = match dv {
+ Some(dv) => dv,
+ None => return Ok(file),
+ };
+ // Delta's canonical `DeletionVectorDescriptor.EMPTY`: inline storage, empty
+ // payload, size 0, cardinality 0. Spark's reader returns all rows for it;
+ // decoding would fail (the empty payload is too short for a magic
+ // number), so pass the file through unchanged before attempting to read it.
+ if dv.cardinality == 0 && dv.size_in_bytes == 0 {
+ return Ok(file);
+ }
+ if dv.size_in_bytes < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative size {}",
+ dv.size_in_bytes
+ )));
+ }
+ if dv.cardinality < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative cardinality {}",
+ dv.cardinality
+ )));
+ }
+
+ let data: Vec<u8> = if let Some(inline) = dv.inline_data {
+ inline
+ } else if let Some(dv_path) = &dv.absolute_path {
+ let offset = dv
+ .offset
+ .ok_or_else(|| GeneralError("On-disk deletion vector missing offset".into()))?;
+ if offset < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative offset {offset}"
+ )));
+ }
+ let offset = offset as u64;
+ // [i32 BE size][data: size_in_bytes][i32 BE crc]
+ let framed_len = 4 + dv.size_in_bytes as u64 + 4;
+ let (store, dv_store_path) = dv_store.ok_or_else(|| {
+ GeneralError(format!(
+ "Deletion vector for {file_path} has an absolute path but no pre-resolved object store"
+ ))
+ })?;
+ let blob = store
+ .get_range(&dv_store_path, offset..offset + framed_len)
+ .await
+ .map_err(|e| GeneralError(format!("Failed to read deletion vector {dv_path}: {e}")))?;
+ unframe_dv_blob(&blob, dv.size_in_bytes as usize)?.to_vec()
+ } else {
+ return Err(GeneralError(
+ "Deletion vector descriptor has neither inline data nor a path".into(),
+ ));
+ };
+ let deleted = deserialize_dv_bitmap(&data)
+ .map_err(|e| GeneralError(format!("Invalid deletion vector for {file_path}: {e}")))?;
+ validate_cardinality(&file_path, dv.cardinality, &deleted)?;
+
+ // Row-group boundaries come from the data file's footer, fetched through the scan's
+ // shared FileMetadataCache with the page index loaded eagerly and the scan's metadata
+ // size hint (mirroring EagerPageIndexReaderFactory): the one fetch here also serves the
+ // subsequent data-file open, so DV files pay no extra footer round-trip. Keyed by
+ // `file.object_meta`, the exact ObjectMeta the scan's reader factory will look up.
+ let metadata_cache = runtime_env.cache_manager.get_file_metadata_cache();
+ let metadata = DFParquetMetadata::new(data_store.as_ref(), &file.object_meta)
Review Comment:
### 7. Share metadata preparation and the final scan's accounting
Could DV preparation use a shared metadata-reading boundary connected to the final scan's reader configuration and metrics? This direct `DFParquetMetadata` fetch correctly fills the shared cache, but runs before the scan creates its instrumented reader. Preparation footer/page-index I/O therefore bypasses the final scan's `scan_io_*` counters, while the later reader open can report a cache hit.
A preparation hook in the shared scan builder, or a shared metadata helper with the same metrics owner, would give preparation and normal reads one place for metadata policy and instrumentation fixes. The important part is carrying preparation accounting into the final scan; creating a separate factory whose counters are discarded would retain the gap.
Could cold-cache and warm-cache DV tests verify that preparation I/O is counted, the footer is reused, and warm metadata opens avoid storage reads? Please preserve the current INT96 stamping on cached metadata. The accounting gap follows from the source path; no runtime counter totals or performance improvement have been measured here.
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so its rewrite (via
+ * [[CometScanWithPlanData]]) survives plan copies -- a transient field would be dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning (the CometIcebergNativeScanExec lesson).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node
+ // automatically gets a helper consistent with ITS runtimeFilters, avoiding the #3510 class of
+ // bug where a stored helper field desyncs from rewritten filters. Costs one extra file listing
+ // per executed instance; correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below, and AQE's UI
+ // plan-walk calls `.metrics` on every node MID-PLANNING, including while a DPP subquery is
+ // still an adaptive placeholder or a partition filter holds an unresolved ScalarSubquery (see
+ // `hasUnevaluableSubqueryFilter` below). That's safe ONLY because constructing `scanHelper` is a
+ // cheap case-class build with no file listing, and core's `CometScanExec.metrics` touches only
+ // `wrapped.driverMetrics` (populated by Spark's own planning) plus a static metric-node
+ // constructor -- neither file listing nor subquery resolution. If core's `metrics` ever touches
+ // either, forcing `scanHelper` here would resurrect the AQE mid-planning crashes this invariant
+ // prevents.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element, so checking runtimeFilters
+ // here avoids constructing/forcing the derived scanHelper just to read partitioning. The
+ // InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep in sync. The
+ // ScalarSubquery case is probed rather than treated as permanently unevaluable: Spark exposes no
+ // public finished/updated flag on ExecSubqueryExpression, but `eval()` doubles as one -- it only
+ // reads the cached `result` behind a `require(updated, ...)` guard, while the subquery is
+ // actually run by `updateResult()` (invoked separately during prepare/AQE), never by `eval()`.
+ // Once resolved, outputPartitioning below reports the real perPartitionData.length instead of
+ // staying at zero -- a fused native parent's buildNativeContext requires that count to match.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case s: ScalarSubquery => !isScalarSubqueryResolved(s)
+ case _ => false
+ })
+
+ // `eval()` never triggers the subquery's execution: on a resolved subquery it is a pure cached
+ // read of `result` (verified against bytecode: `Predef.require(updated(), ...)` then a plain
+ // field read), so this probe is safe to call repeatedly, including from AQE's mid-planning plan
+ // walks. Pre-resolution, the ONLY throw is `require`'s `IllegalArgumentException`; catch exactly
+ // that, since anything else escaping is a genuine bug we must not mask as unpartitioned.
+ private def isScalarSubqueryResolved(s: ScalarSubquery): Boolean =
+ try {
+ s.eval()
+ true
+ } catch {
+ case _: IllegalArgumentException => false
+ }
+
+ private def isAdaptivePlaceholder(p: SparkPlan): Boolean = p match {
+ case ReusedSubqueryExec(inner) => isAdaptivePlaceholder(inner)
+ case _: CometSubqueryAdaptiveBroadcastExec => true
+ case _: SubqueryAdaptiveBroadcastExec => true
+ case _ => false
+ }
+
+ override lazy val outputOrdering: Seq[SortOrder] = originalPlan.outputOrdering
+
+ override def dynamicPruningFilters: Seq[Expression] = runtimeFilters
+
+ override def withDynamicPruningFilters(filters: Seq[Expression]): SparkPlan = {
+ // A real copy: runtimeFilters is a constructor field included in equality, so the copy
+ // survives enclosing-block rebuilds, and the derived scanHelper picks up the rewritten
+ // filters automatically.
+ copy(runtimeFilters = filters)
+ }
+
+ /**
+ * Lazy split-mode serialization, mirroring CometNativeScanExec: common data was serialized at
+ * planning; per-partition file lists serialize here, at execution time.
+ */
+ @transient private lazy val serializedPartitionData
+ : (Array[Byte], Array[Array[Byte]], Array[Seq[String]]) = {
+ // Resolve the helper's DPP subqueries: it holds its own InSubqueryExec instances that
+ // Spark's expressions walk does not see (the helper is derived, not a child).
+ scanHelper.partitionFilters.foreach {
+ case DynamicPruningExpression(e: InSubqueryExec) if e.values().isEmpty =>
+ e.updateResult()
+ case _ =>
+ }
+
+ val commonBytes = {
+ val deltaScan = DeltaSparkScanEnvelope.unpack(nativeOp)
+ // Scalar subqueries in dataFilters were unresolved at planning; resolve them now and
+ // append them as pushed filters, as CometNativeScanExec.serializedPartitionData does.
+ // has_data_filters follows their presence, not the serialized count: a filter that fails
+ // to serialize still keeps native on the safe timestamp conversion for a filtered scan.
+ val resolved = org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .resolvedSubqueryFilters(dataFilters, output, requiredSchema, conf)
+ val common = if (!resolved.hasResolvedFilters) {
+ deltaScan.getCommon
+ } else {
+ val builder = deltaScan.getCommon.toBuilder
+ builder.setHasDataFilters(true)
+ resolved.protos.foreach(builder.addDataFilters)
+ builder.build()
+ }
+ OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaScan.getDeltaCommon)
+ .build()
+ .toByteArray
+ }
+
+ val filePartitions = scanHelper.getFilePartitions()
+
+ val tableRoot = DeltaSparkScanEnvelope.unpack(nativeOp).getDeltaCommon.getTableRoot
+ val perPartitionBytes = filePartitions.map { filePartition =>
+ org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .serializePartition(filePartition, originalPlan, tableRoot)
+ }.toArray
+
+ val perPartitionPaths = filePartitions.map(_.files.map(_.filePath.toString).toSeq).toArray
+
+ (commonBytes, perPartitionBytes, perPartitionPaths)
+ }
+
+ override def commonData: Array[Byte] = serializedPartitionData._1
+
+ override def perPartitionData: Array[Array[Byte]] = serializedPartitionData._2
+
+ def perPartitionFilePaths: Array[Seq[String]] = serializedPartitionData._3
+
+ override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+ val nativeMetrics = CometMetricNode.fromCometPlan(this)
+ val serializedPlan = CometExec.serializeNativePlan(nativeOp)
+
+ new CometExecRDD(
+ sparkContext,
+ Seq.empty,
+ Map(sourceKey -> commonData),
+ Map(sourceKey -> perPartitionData),
+ serializedPlan,
+ perPartitionData.length,
+ output.length,
+ nativeMetrics,
+ Seq.empty,
+ None,
+ Seq.empty,
+ perPartitionFilePaths = perPartitionFilePaths,
+ reportScanInputMetrics = true)
+ }
+
+ override def doCanonicalize(): CometDeltaNativeScanExec = {
+ val canonOriginal = if (originalPlan != null) {
+ val stripped = originalPlan.copy(partitionFilters =
+ CometScanUtils.filterUnusedDynamicPruningExpressions(originalPlan.partitionFilters))
+ stripped.doCanonicalize()
+ } else {
+ null
+ }
+ CometDeltaNativeScanExec(
+ nativeOp,
+ output.map(QueryPlan.normalizeExpressions(_, output)),
+ requiredSchema,
+ QueryPlan.normalizePredicates(
+ CometScanUtils.filterUnusedDynamicPruningExpressions(runtimeFilters),
+ output),
+ QueryPlan.normalizePredicates(dataFilters, output),
+ relation,
+ canonOriginal,
+ SerializedPlan(None),
+ "")
+ }
+
+ override def stringArgs: Iterator[Any] = Iterator(output, runtimeFilters)
+
+ override def equals(obj: Any): Boolean = obj match {
+ case other: CometDeltaNativeScanExec =>
+ this.originalPlan == other.originalPlan &&
+ this.serializedPlanOpt == other.serializedPlanOpt &&
+ this.runtimeFilters == other.runtimeFilters &&
+ this.dataFilters == other.dataFilters
+ case _ => false
+ }
+
+ override def hashCode(): Int =
+ java.util.Objects.hash(originalPlan, serializedPlanOpt, runtimeFilters, dataFilters)
+
+ private val driverMetricKeys =
+ Set(
+ "numFiles",
+ "filesSize",
+ "numPartitions",
+ "metadataTime",
+ "staticFilesNum",
+ "staticFilesSize",
+ "pruningTime")
+
+ // Forces `scanHelper` (see its doc above for why that -- and reading `.metrics` off it -- is
+ // safe even when AQE calls `.metrics` mid-planning against an unresolved DPP/scalar subquery).
+ override lazy val metrics: Map[String, SQLMetric] = {
+ CometMetricNode.nativeScanMetrics(session.sparkContext) ++
+ scanHelper.metrics.filter { case (k, _) => driverMetricKeys.contains(k) }
+ }
+}
+
+object CometDeltaNativeScanExec {
+
+ /** File-planning helper: reuses CometScanExec's listing/splitting/DPP machinery. */
Review Comment:
### 4. Make the existing file-planning boundary explicit
`planningHelper` already reuses core's listing, splitting, and dynamic partition pruning. Could we make its contract explicitly take the original scan plus the current partition and data filters, and return the final `FilePartition` objects with their metadata?
Today partition filters are supplied separately while data filters come from `originalPlan`. Keeping both inputs explicit, and distinguishing predicates used for file selection from predicates evaluated by the reader, would localize later pruning changes. The existing reader-free helper can remain the implementation; the serializer would consume only its final result. Plan copies and filter rewrites should retain the current protection against stale cached selection.
Could regression tests compare serialized paths, order, split starts and lengths, partition values, and metadata against Spark's final partitions? Include an ordinary non-DV file with a nonzero split start and dynamic partition pruning with AQE on and off. This would extend the existing result, file-count, and unresolved-subquery coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5955524699
I would like to do a deeper review of this PR before it is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4084934139
##########
contrib/delta-spark/src/test/scala/org/apache/comet/contrib/delta/CometDeltaNativeScanSuite.scala:
##########
@@ -0,0 +1,3584 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.File
+
+import scala.collection.mutable
+import scala.collection.mutable.ListBuffer
+import scala.concurrent.duration.DurationInt
+
+import org.apache.spark.scheduler.{SparkListener, SparkListenerTaskEnd}
+import org.apache.spark.sql.{DataFrame, Row}
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, DynamicPruningExpression, NamedExpression, StructsToJson}
+import org.apache.spark.sql.comet.CometDeltaNativeScanExec
+import org.apache.spark.sql.execution.{FileSourceScanExec, QueryExecution, ScalarSubquery, SparkPlan, SubqueryExec}
+import org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec
+import org.apache.spark.sql.functions.{col, lit, to_json}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ByteType, LongType, StringType, StructField, StructType}
+import org.apache.spark.sql.util.QueryExecutionListener
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus
+import org.apache.comet.ExtendedExplainInfo
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.operator.CometNativeScan
+
+/**
+ * Differential suite: append-only Delta tables read through the native Delta scan must produce
+ * results identical to Spark's Delta reader, engage the native operator, and prune at row-group
+ * and page level.
+ */
+class CometDeltaNativeScanSuite extends CometDeltaTestBase {
+
+ test("plain delta table reads natively with identical results") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark
+ .range(0, 1000)
+ .selectExpr("id", "id * 2 as v", "cast(id as string) as s")
+ .write
+ .format("delta")
+ .save(path)
+
+ val df = spark.read.format("delta").load(path).filter(col("id") > 500)
+ checkDeltaNativeScanAnswer(df)
+ }
+ }
+
+ test("projection and filter on delta table") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark
+ .range(0, 1000)
+ .selectExpr("id", "id % 10 as bucket", "cast(id as double) as d")
+ .write
+ .format("delta")
+ .save(path)
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .select("bucket", "d")
+ .filter(col("d") < 100.0)
+ checkDeltaNativeScanAnswer(df)
+ }
+ }
+
+ test("partitioned delta table with partition filter") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark
+ .range(0, 1000)
+ .selectExpr("id", "id % 7 as p")
+ .write
+ .format("delta")
+ .partitionBy("p")
+ .save(path)
+
+ val df = spark.read.format("delta").load(path).filter(col("p") === 3)
+ checkDeltaNativeScanAnswer(df)
+ assert(df.count() > 0)
+ }
+ }
+
+ test("multi-file delta table after several appends") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ for (i <- 0 until 4) {
+ spark
+ .range(i * 100, (i + 1) * 100)
+ .selectExpr("id", "id * 3 as v")
+ .write
+ .format("delta")
+ .mode("append")
+ .save(path)
+ }
+ val df = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(df)
+ assert(df.count() == 400)
+ }
+ }
+
+ test("time travel VERSION AS OF reads natively") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark.range(0, 100).write.format("delta").save(path)
+ spark.range(100, 200).write.format("delta").mode("append").save(path)
+
+ val v0 = spark.read.format("delta").option("versionAsOf", 0).load(path)
+ checkDeltaNativeScanAnswer(v0)
+ assert(v0.count() == 100)
+ }
+ }
+
+ test("selective predicate prunes row groups and pages") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ // Small row groups + page-level stats: sorted data so min/max stats are tight. The Delta
+ // writer ignores parquet.* DataFrameWriter options, so set them on the Hadoop conf.
+ val hadoopConf = spark.sparkContext.hadoopConfiguration
+ val oldBlockSize = hadoopConf.get("parquet.block.size")
+ val oldPageSize = hadoopConf.get("parquet.page.size")
+ hadoopConf.setInt("parquet.block.size", 256 * 1024)
+ hadoopConf.setInt("parquet.page.size", 16 * 1024)
+ try {
+ spark
+ .range(0, 500000)
+ .selectExpr("id", "id * 2 as v")
+ .sort("id")
+ .coalesce(1)
+ .write
+ .format("delta")
+ .save(path)
+ } finally {
+ if (oldBlockSize == null) hadoopConf.unset("parquet.block.size")
+ else hadoopConf.set("parquet.block.size", oldBlockSize)
+ if (oldPageSize == null) hadoopConf.unset("parquet.page.size")
+ else hadoopConf.set("parquet.page.size", oldPageSize)
+ }
+
+ def query = spark.read
+ .format("delta")
+ .load(path)
+ .filter(col("id") >= 100 && col("id") < 200)
+ checkDeltaNativeScanAnswer(query)
+
+ // checkSparkAnswer re-plans the query, so read metrics from a DataFrame we execute
+ // ourselves (collect() runs THIS Dataset's queryExecution; count() would plan a new one):
+ // its executed plan holds the metric objects native execution updated.
+ val df = query
+ assert(df.collect().length == 100)
+ val scans = deltaNativeScans(df)
+ assert(scans.size == 1)
+ val metrics = scans.head.metrics
+ val rowGroupsPruned = metrics.get("row_groups_pruned_statistics").map(_.value).getOrElse(0L)
+ val pagesPruned = metrics.get("page_index_rows_pruned").map(_.value).getOrElse(0L)
+ assert(
+ rowGroupsPruned > 0,
+ s"expected row-group pruning; metrics: ${metrics.map { case (k, v) => s"$k=${v.value}" }}")
+ assert(
+ pagesPruned > 0,
+ s"expected page-index pruning; metrics: ${metrics.map { case (k, v) =>
+ s"$k=${v.value}"
+ }}")
+ }
+ }
+
+ test("scalar subquery data filter is pushed down and prunes row groups and pages") {
+ withTempPath { dir =>
+ val path = s"${dir.getAbsolutePath}/data"
+ val thresholds = s"${dir.getAbsolutePath}/thresholds"
+ // Same layout as the selective-predicate test: small row groups + tight page stats.
+ val hadoopConf = spark.sparkContext.hadoopConfiguration
+ val oldBlockSize = hadoopConf.get("parquet.block.size")
+ val oldPageSize = hadoopConf.get("parquet.page.size")
+ hadoopConf.setInt("parquet.block.size", 256 * 1024)
+ hadoopConf.setInt("parquet.page.size", 16 * 1024)
+ try {
+ spark
+ .range(0, 500000)
+ .selectExpr("id", "id * 2 as v")
+ .sort("id")
+ .coalesce(1)
+ .write
+ .format("delta")
+ .save(path)
+ } finally {
+ if (oldBlockSize == null) hadoopConf.unset("parquet.block.size")
+ else hadoopConf.set("parquet.block.size", oldBlockSize)
+ if (oldPageSize == null) hadoopConf.unset("parquet.page.size")
+ else hadoopConf.set("parquet.page.size", oldPageSize)
+ }
+ spark
+ .sql("SELECT CAST(100 AS BIGINT) AS lo, CAST(200 AS BIGINT) AS hi")
+ .write
+ .format("delta")
+ .save(thresholds)
+
+ // Scalar subqueries are PlanExpressions: unresolved at planning, so the bounds can
+ // only reach the native reader via the execution-time resolve-and-append path.
+ def query = spark.sql(
+ s"SELECT * FROM delta.`$path` WHERE id >= (SELECT lo FROM delta.`$thresholds`) " +
+ s"AND id < (SELECT hi FROM delta.`$thresholds`)")
+ checkDeltaNativeScanAnswer(query)
+
+ val df = query
+ assert(df.collect().length == 100)
+ // The thresholds table inside the subquery is also claimed natively; pick the
+ // main data-table scan by its output.
+ assertSubqueryFilterPushed(df, dataColumn = "v")
+ val scans = deltaNativeScans(df).filter(_.output.exists(_.name == "v"))
+ assert(scans.size == 1)
+ val metrics = scans.head.metrics
+ val rowGroupsPruned = metrics.get("row_groups_pruned_statistics").map(_.value).getOrElse(0L)
+ val pagesPruned = metrics.get("page_index_rows_pruned").map(_.value).getOrElse(0L)
+ assert(
+ rowGroupsPruned > 0,
+ s"expected row-group pruning from the resolved subquery bounds; metrics: ${metrics.map {
+ case (k, v) => s"$k=${v.value}"
+ }}")
+ assert(
+ pagesPruned > 0,
+ s"expected page-index pruning from the resolved subquery bounds; metrics: ${metrics.map {
+ case (k, v) => s"$k=${v.value}"
+ }}")
+ }
+ }
+
+ test("deletion vectors: scalar subquery filter composes with DV application") {
+ withTempPath { dir =>
+ val path = s"${dir.getAbsolutePath}/data"
+ val thresholds = s"${dir.getAbsolutePath}/thresholds"
+ createDvTable(path, rows = 10000)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+ spark
+ .sql("SELECT CAST(5000 AS BIGINT) AS lo")
+ .write
+ .format("delta")
+ .save(thresholds)
+
+ def query =
+ spark.sql(s"SELECT * FROM delta.`$path` WHERE id >= (SELECT lo FROM delta.`$thresholds`)")
+ checkDeltaNativeScanAnswer(query)
+ // Deleted rows must stay deleted with the pushed bound applied in-scan.
+ val df = query
+ val rows = df.collect()
+ assert(rows.length == 2500)
+ assert(rows.forall(r => r.getLong(0) % 2 == 1 && r.getLong(0) >= 5000))
+ assertSubqueryFilterPushed(df, dataColumn = "v")
+ }
+ }
+
+ test("column mapping: scalar subquery filter on a renamed column") {
+ withTempPath { dir =>
+ val path = s"${dir.getAbsolutePath}/data"
+ val thresholds = s"${dir.getAbsolutePath}/thresholds"
+ spark.range(0, 1000).selectExpr("id", "id * 2 as v").write.format("delta").save(path)
+ enableColumnMapping(path)
+ spark.sql(s"ALTER TABLE delta.`$path` RENAME COLUMN v TO w")
+ spark
+ .sql("SELECT CAST(900 AS BIGINT) AS lo")
+ .write
+ .format("delta")
+ .save(thresholds)
+
+ // The pushed filter references the renamed column: it must bind against the
+ // physical read schema, not the logical name.
+ def query =
+ spark.sql(s"SELECT * FROM delta.`$path` WHERE w >= (SELECT lo FROM delta.`$thresholds`)")
+ checkDeltaNativeScanAnswer(query)
+ val df = query
+ assert(df.collect().length == 550)
+ assertSubqueryFilterPushed(df, dataColumn = "w")
+ }
+ }
+
+ /**
+ * Assert the resolved scalar-subquery bound was actually appended to the native scan's
+ * execution-time common data (answers alone cannot show this: Spark's covering FilterExec would
+ * mask a silently-skipped pushdown). `df` must already have been executed.
+ */
+ private def assertSubqueryFilterPushed(df: DataFrame, dataColumn: String): Unit = {
+ val scans = deltaNativeScans(df).collect {
+ case s: CometDeltaNativeScanExec if s.output.exists(_.name == dataColumn) => s
+ }
+ assert(scans.size == 1)
+ val scan = scans.head
+ val planTimeFilters =
+ DeltaSparkScanEnvelope.unpack(scan.nativeOp).getCommon.getDataFiltersCount
+ val executedFilters = OperatorOuterClass.DeltaSparkScan
+ .parseFrom(scan.commonData)
+ .getCommon
+ .getDataFiltersCount
+ assert(
+ executedFilters > planTimeFilters,
+ "expected resolved subquery filters appended at execution: " +
+ s"plan-time=$planTimeFilters executed=$executedFilters " +
+ s"dataFilters=${scan.dataFilters.mkString("; ")}")
+ }
+
+ test("scalar subquery filter is NOT pushed below a limit") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark.range(0, 3).selectExpr("id").write.format("delta").save(path)
+ spark.read.format("delta").load(path).createOrReplaceTempView("t_limit_pushdown")
+
+ val df = spark.sql(
+ "SELECT id FROM (SELECT id FROM t_limit_pushdown ORDER BY id LIMIT 1) q " +
+ "WHERE id > (SELECT max(id) FROM range(1))")
+ checkSparkAnswer(df)
+ assert(df.collect().isEmpty)
+ assertNoSubqueryFilterPushed(df)
+ }
+ }
+
+ test("scalar subquery filter is NOT pushed across a nondeterministic projection") {
+ withSQLConf(CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark.range(0, 5).coalesce(1).write.format("delta").save(path)
+ spark.read.format("delta").load(path).createOrReplaceTempView("t_monotonic_id")
+
+ // A deterministic conjunct does not commute with a nondeterministic projection: the
+ // subquery bound must not be pushed into the scan below `seq`, or the surviving rows'
+ // monotonically_increasing_id() values change and the answer is wrong.
+ val df = spark.sql(
+ "SELECT id FROM (SELECT id, monotonically_increasing_id() AS seq " +
+ "FROM t_monotonic_id) q WHERE id > (SELECT max(id) FROM range(1)) AND seq = 1")
+ checkSparkAnswer(df)
+ assert(df.collect().toSeq == Seq(Row(1)))
+ assertNoSubqueryFilterPushed(df)
+ }
+ }
+ }
+
+ /**
+ * Assert no scalar-subquery filter was harvested and pushed into the native scan's
+ * execution-time common data: the scan must sit below a non-commuting operator (e.g. LIMIT /
+ * TopN), so the covering FilterExec's predicate must stay above it rather than move into the
+ * scan. Also confirms the query still engaged the native Delta scan, i.e. this exercises the
+ * commutativity guard rather than a plan that fell back to Spark entirely. `df` must already
+ * have been executed.
+ */
+ private def assertNoSubqueryFilterPushed(df: DataFrame): Unit = {
+ val scans = deltaNativeScans(df).collect { case s: CometDeltaNativeScanExec => s }
+ assert(scans.size == 1, s"expected exactly one native Delta scan; found ${scans.size}")
+ val scan = scans.head
+ val planTimeFilters =
+ DeltaSparkScanEnvelope.unpack(scan.nativeOp).getCommon.getDataFiltersCount
+ val executedFilters = OperatorOuterClass.DeltaSparkScan
+ .parseFrom(scan.commonData)
+ .getCommon
+ .getDataFiltersCount
+ assert(
+ executedFilters == planTimeFilters,
+ "expected no subquery filter pushed across the non-commuting operator between the " +
+ s"covering filter and the scan: plan-time=$planTimeFilters executed=$executedFilters " +
+ s"dataFilters=${scan.dataFilters.mkString("; ")}")
+ }
+
+ test("scalar subquery filter rejected by serde still marks the scan as filtered") {
+ withTempPath { dir =>
+ val path = s"${dir.getAbsolutePath}/data"
+ val bounds = s"${dir.getAbsolutePath}/bounds"
+ spark.range(0, 100).selectExpr("id", "id * 2 as v").write.format("delta").save(path)
+ spark.sql("SELECT CAST(42 AS BIGINT) AS lo").write.format("delta").save(bounds)
+
+ // With EqualNullSafe disabled the resolved bound cannot serialize, yet the scan must still
+ // carry has_data_filters so native treats it as a filtered read, exactly like core does.
+ withSQLConf("spark.comet.expression.EqualNullSafe.enabled" -> "false") {
+ def query =
+ spark.sql(
+ s"SELECT * FROM delta.`$path` WHERE id <=> (SELECT max(lo) FROM delta.`$bounds`)")
+ checkDeltaNativeScanAnswer(query)
+ val df = query
+ assert(df.collect().toSeq == Seq(Row(42L, 84L)))
+ assertUnserializedSubqueryFilterMarksScanFiltered(df, dataColumn = "v")
+ }
+ }
+ }
+
+ test("unserializable scalar subquery filter keeps the safe TIMESTAMP_MILLIS conversion") {
+ // Same fixture as core's "filtered TIMESTAMP_MILLIS scans do not convert values Spark can
+ // skip": a raw file whose only overflowing millisecond value Spark prunes from the footer
+ // statistics once the resolved bound is pushed, so native must not convert it either.
+ withTempPath { dir =>
+ val path = s"${dir.getAbsolutePath}/data"
+ val bounds = s"${dir.getAbsolutePath}/bounds"
+ writeRawParquetFile(
+ path,
+ """message root {
+ | optional int32 id;
+ | optional int64 ts(TIMESTAMP_MILLIS);
+ |}""".stripMargin) { factory =>
+ (1 to 16).map(id => factory.newGroup().append("id", id).append("ts", 1717243200000L)) :+
+ factory.newGroup().append("id", 17).append("ts", 9223372036854776L)
+ }
+ spark.sql(s"CONVERT TO DELTA parquet.`$path` NO STATISTICS")
+ spark.sql("SELECT timestamp_seconds(0) AS bound").write.format("delta").save(bounds)
+
+ withSQLConf(
+ "spark.comet.expression.EqualNullSafe.enabled" -> "false",
+ "spark.sql.parquet.datetimeRebaseModeInRead" -> "CORRECTED",
+ "spark.sql.parquet.int96RebaseModeInRead" -> "CORRECTED") {
+ def query = spark.sql(
+ s"SELECT id, ts FROM delta.`$path` " +
+ s"WHERE ts <=> (SELECT max(bound) FROM delta.`$bounds`)")
+ // Spark 3.x never pushes subquery filters into its parquet reader and converts the
+ // overflowing value itself, so the answer comparison is meaningful on Spark 4.0+ only.
+ if (isSpark40Plus) {
+ checkDeltaNativeScanAnswer(query)
+ }
+ val df = query
+ assert(df.collect().isEmpty)
+ assert(
+ deltaNativeScans(df).nonEmpty,
+ s"expected a native Delta scan:\n${df.queryExecution}")
+ assertUnserializedSubqueryFilterMarksScanFiltered(df, dataColumn = "ts")
+ }
+ }
+ }
+
+ /**
+ * Assert the execution-time common data of the scan producing `dataColumn` reports
+ * `has_data_filters` with no serialized data filter: the plan-time proto carries neither, and
+ * the resolved subquery filter is the only data filter, so only the execution-time path can set
+ * the bit. `df` must already have been executed.
+ */
+ private def assertUnserializedSubqueryFilterMarksScanFiltered(
+ df: DataFrame,
+ dataColumn: String): Unit = {
+ val scans = deltaNativeScans(df).collect {
+ case s: CometDeltaNativeScanExec if s.output.exists(_.name == dataColumn) => s
+ }
+ assert(scans.size == 1, s"expected exactly one native Delta scan; found ${scans.size}")
+ val scan = scans.head
+ assert(
+ scan.dataFilters.exists(_.exists(_.isInstanceOf[ScalarSubquery])),
+ s"expected a scalar subquery data filter: ${scan.dataFilters.mkString("; ")}")
+ val planTime = DeltaSparkScanEnvelope.unpack(scan.nativeOp).getCommon
+ assert(!planTime.getHasDataFilters && planTime.getDataFiltersCount == 0)
+ val executed = OperatorOuterClass.DeltaSparkScan.parseFrom(scan.commonData).getCommon
+ assert(
+ executed.getHasDataFilters,
+ "expected has_data_filters at execution even though the resolved subquery filter did " +
+ s"not serialize: dataFilters=${scan.dataFilters.mkString("; ")}")
+ assert(executed.getDataFiltersCount == 0)
+ }
+
+ test("aggregation over delta table") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark
+ .range(0, 10000)
+ .selectExpr("id", "id % 13 as g", "id * 2 as v")
+ .write
+ .format("delta")
+ .save(path)
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .groupBy("g")
+ .sum("v")
+ checkDeltaNativeScanAnswer(df)
+ }
+ }
+
+ test("conf disables the native delta scan") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark.range(0, 100).write.format("delta").save(path)
+
+ withSQLConf(DeltaScanConf.COMET_DELTA_NATIVE_ENABLED.key -> "false") {
+ val df = spark.read.format("delta").load(path)
+ checkSparkAnswer(df)
+ assert(deltaNativeScans(df).isEmpty)
+ }
+ }
+ }
+
+ test("native delta scan is opt-in: disabled when the conf is not set") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark.range(0, 100).write.format("delta").save(path)
+
+ // The suite base enables the scan globally; drop the key entirely to
+ // observe the out-of-the-box default.
+ spark.conf.unset(DeltaScanConf.COMET_DELTA_NATIVE_ENABLED.key)
+ try {
+ assert(!DeltaScanConf.scanEnabled)
+ val df = spark.read.format("delta").load(path)
+ checkSparkAnswer(df)
+ assert(deltaNativeScans(df).isEmpty)
+ } finally {
+ spark.conf.set(DeltaScanConf.COMET_DELTA_NATIVE_ENABLED.key, "true")
+ }
+ }
+ }
+
+ test("table root under a directory whose name contains a newline falls back to Spark") {
+ // object_store recognizes the `file` scheme but rejects the control character in the
+ // directory name (`%0A` in the URI), so native execution could not open the table where
+ // Spark's Hadoop-backed reader can. The claim gate must decline before native planning.
+ withTempPath { dir =>
+ val path = new File(new File(dir, "dir\n"), "data").getAbsolutePath
+ spark.range(0, 100).write.format("delta").save(path)
+
+ val df = spark.read.format("delta").load(path)
+ assert(
+ deltaNativeScans(df).isEmpty,
+ "Expected no native Delta scan under a newline directory:\n" +
+ s"${df.queryExecution.executedPlan}")
+ checkSparkAnswerAndFallbackReason(
+ df,
+ "Native Delta scan cannot open path 'file:" + dir.getAbsolutePath +
+ "/dir%0A/data': object_store rejects it")
+ }
+ }
+
+ test("shallow clone whose source data files sit under a newline directory falls back") {
+ // The clone's own root is an ordinary path, so only the selected data files (resolved to
+ // the source table's directory) carry the rejected segment: this exercises the
+ // selected-paths probe, not the root gate. The reason names the first such complete path,
+ // a data file under the source directory.
+ withTempPath { dir =>
+ val sourcePath = new File(new File(dir, "dir\n"), "source").getAbsolutePath
+ val clonePath = new File(dir, "clone").getAbsolutePath
+ spark.range(0, 100).write.format("delta").save(sourcePath)
+ spark.sql(s"CREATE TABLE delta.`$clonePath` SHALLOW CLONE delta.`$sourcePath`")
+
+ val df = spark.read.format("delta").load(clonePath)
+ assert(
+ deltaNativeScans(df).isEmpty,
+ "Expected no native Delta scan for a clone of a newline-directory source:\n" +
+ s"${df.queryExecution.executedPlan}")
+ checkSparkAnswerAndFallbackReason(
+ df,
+ "Native Delta scan cannot open path 'file:" + dir.getAbsolutePath +
+ "/dir%0A/source/")
+ }
+ }
+
+ test("converted Parquet table with a newline in a data file basename falls back to Spark") {
+ // CONVERT TO DELTA keeps the existing Parquet file names, so the rejected character sits in
+ // the basename rather than a directory segment: the table root and every parent directory
+ // pass the path probe, and only a check of the complete selected path can decline.
+ withTempPath { dir =>
+ val path = new File(dir, "data").getAbsolutePath
+ spark.range(0, 100).repartition(2).write.parquet(path)
+ val original = new File(path).listFiles().filter(_.getName.endsWith(".parquet")).head
+ val renamed = new File(path, "part-00000\n.snappy.parquet")
+ java.nio.file.Files.move(original.toPath, renamed.toPath)
+ spark.sql(s"CONVERT TO DELTA parquet.`$path`")
+
+ val df = spark.read.format("delta").load(path)
+ assert(
+ deltaNativeScans(df).isEmpty,
+ "Expected no native Delta scan for a converted table with a newline basename:\n" +
+ s"${df.queryExecution.executedPlan}")
+ checkSparkAnswerAndFallbackReason(
+ df,
+ s"Native Delta scan cannot open path 'file:$path/part-00000%0A.snappy.parquet': " +
+ "object_store rejects it")
+ }
+ }
+
+ private def createDvTable(path: String, rows: Long = 1000): Unit = {
+ spark.range(0, rows).selectExpr("id", "id * 2 as v").write.format("delta").save(path)
+ spark.sql(
+ s"ALTER TABLE delta.`$path` SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true')")
+ }
+
+ /**
+ * Same shape as `createDvTable`, plus one extra TINYINT column (value 7) under `columnName`.
+ */
+ private def createDvTableWithExtraColumn(
+ path: String,
+ columnName: String,
+ rows: Long = 1000): Unit = {
+ spark
+ .range(0, rows)
+ .selectExpr("id", s"cast(7 as tinyint) as `$columnName`")
+ .write
+ .format("delta")
+ .save(path)
+ spark.sql(
+ s"ALTER TABLE delta.`$path` SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true')")
+ }
+
+ test("deletion vectors: DELETE-produced DVs read natively with correct results") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ val df = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(df)
+ assert(df.count() == 500)
+ }
+ }
+
+ test(
+ "deletion vectors: user column named like the synthetic internal-column slot keeps its " +
+ "own values") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ val collidingName = "_comet_delta___delta_internal_is_row_deleted"
+ createDvTableWithExtraColumn(path, collidingName)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id = 0")
+
+ val df = spark.read.format("delta").load(path).select("id", collidingName)
+ checkDeltaNativeScanAnswer(df)
+ val survivingValues = df.collect().map(_.getAs[Byte](collidingName)).distinct
+ assert(
+ survivingValues.sameElements(Array(7.toByte)),
+ "expected the user column's own value (7) to survive DV filtering, " +
+ s"got ${survivingValues.toSeq}")
+ }
+ }
+
+ test("deletion vectors: normally named extra column alongside DVs reads natively") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTableWithExtraColumn(path, "tag")
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id = 0")
+
+ val df = spark.read.format("delta").load(path).select("id", "tag")
+ checkDeltaNativeScanAnswer(df)
+ val survivingValues = df.collect().map(_.getAs[Byte]("tag")).distinct
+ assert(
+ survivingValues.sameElements(Array(7.toByte)),
+ "expected the extra column's value (7) to survive DV filtering, got " +
+ survivingValues.toSeq)
+ }
+ }
+
+ test("deletion vectors: UPDATE-produced DVs read natively with correct results") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"UPDATE delta.`$path` SET v = -1 WHERE id < 100")
+
+ val df = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(df)
+ assert(df.filter(col("v") === -1).count() == 100)
+ assert(df.count() == 1000)
+ }
+ }
+
+ test("deletion vectors: multiple DELETEs accumulate correctly") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 3 = 0")
+
+ val df = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(df)
+ // odd ids not divisible by 3
+ assert(df.count() == (0L until 1000L).count(i => i % 2 != 0 && i % 3 != 0))
+ }
+ }
+
+ test(
+ "deletion vectors: maxDeletedRowsPerFile budget declines an oversized DV and " +
+ "claims once raised") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ // repartition(4) guarantees >= 2 physical files so the per-file cardinality gate has
+ // more than one file to inspect, mirroring design F3's multi-file test shape.
+ spark
+ .range(0, 1000)
+ .selectExpr("id", "id * 2 as v")
+ .repartition(4)
+ .write
+ .format("delta")
+ .save(path)
+ spark.sql(
+ s"ALTER TABLE delta.`$path` SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true')")
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ withSQLConf(DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key -> "1") {
+ val df = spark.read.format("delta").load(path)
+ checkSparkAnswer(df)
+ assert(
+ deltaNativeScans(df).isEmpty,
+ "a budget of 1 deleted row per file must decline every DV-bearing file")
+ }
+
+ withSQLConf(DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key -> "1000000") {
+ val df = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(df)
+ }
+ }
+ }
+
+ test("deletion vectors: maxDeletedRowsPerFile decline reason names the conf key") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ withSQLConf(DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key -> "1") {
+ checkSparkAnswerAndFallbackReason(
+ spark.read.format("delta").load(path),
+ DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key)
+ }
+ }
+ }
+
+ test("deletion vectors: fully-deleted region and selective predicate still prune pages") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ val hadoopConf = spark.sparkContext.hadoopConfiguration
+ val oldBlockSize = hadoopConf.get("parquet.block.size")
+ val oldPageSize = hadoopConf.get("parquet.page.size")
+ hadoopConf.setInt("parquet.block.size", 256 * 1024)
+ hadoopConf.setInt("parquet.page.size", 16 * 1024)
+ try {
+ spark
+ .range(0, 500000)
+ .selectExpr("id", "id * 2 as v")
+ .sort("id")
+ .coalesce(1)
+ .write
+ .format("delta")
+ .save(path)
+ } finally {
+ if (oldBlockSize == null) hadoopConf.unset("parquet.block.size")
+ else hadoopConf.set("parquet.block.size", oldBlockSize)
+ if (oldPageSize == null) hadoopConf.unset("parquet.page.size")
+ else hadoopConf.set("parquet.page.size", oldPageSize)
+ }
+ spark.sql(
+ s"ALTER TABLE delta.`$path` SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true')")
+ // Delete a slice inside the predicate range and a large slice outside it.
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id >= 150 AND id < 160")
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id >= 300000")
+
+ def query = spark.read
+ .format("delta")
+ .load(path)
+ .filter(col("id") >= 100 && col("id") < 200)
+ checkDeltaNativeScanAnswer(query)
+
+ val df = query
+ assert(df.collect().length == 90)
+ val scans = deltaNativeScans(df)
+ assert(scans.size == 1)
+ val metrics = scans.head.metrics
+ val pagesPruned = metrics.get("page_index_rows_pruned").map(_.value).getOrElse(0L)
+ assert(
+ pagesPruned > 0,
+ s"expected page-index pruning to compose with DVs; metrics: ${metrics.map { case (k, v) =>
+ s"$k=${v.value}"
+ }}")
+ }
+ }
+
+ test("deletion vectors: aggregation over DV table") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path, rows = 10000)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 7 = 0")
+
+ val df = spark.read.format("delta").load(path).groupBy(col("id") % 13).count()
+ checkDeltaNativeScanAnswer(df)
+ }
+ }
+
+ test("deletion vectors: partitioned table reads natively with correct results") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark
+ .range(0, 1000)
+ .selectExpr("id", "id % 5 as p", "id * 2 as v")
+ .write
+ .format("delta")
+ .partitionBy("p")
+ .save(path)
+ spark.sql(
+ s"ALTER TABLE delta.`$path` SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true')")
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 3 = 0")
+
+ val df = spark.read.format("delta").load(path).filter(col("p") === 2)
+ checkDeltaNativeScanAnswer(df)
+ assert(df.count() == (0L until 1000L).count(i => i % 5 == 2 && i % 3 != 0))
+
+ val all = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(all)
+ assert(all.count() == (0L until 1000L).count(_ % 3 != 0))
+ }
+ }
+
+ test("deletion vectors: combined with constant metadata columns") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id < 250")
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .selectExpr("id", "v", "_metadata.file_name as fn")
+ checkSparkAnswer(df.selectExpr("id", "v", "length(fn) > 0"))
+ // Whether this claims or declines, results must match; if it claimed, verify the
+ // native node is present so the combination is actually exercised when supported.
+ val rows = df.collect()
+ assert(rows.length == 750)
+ assert(rows.forall(_.getString(2).nonEmpty))
+ }
+ }
+
+ test(
+ "deletion vectors: constant-metadata field names are deduplicated against the physical " +
+ "data and partition schemas") {
+ // End-to-end coverage is not possible here: selecting any `_metadata.*` field in the DV
+ // shape always declines today for an unrelated, pre-existing reason -- Spark reuses the
+ // scan's own row-index bookkeeping attribute as `_metadata.row_index`'s source, and
+ // `DeltaScanSupport.rowIndexUnusedAbove` conservatively treats extracting ANY `_metadata`
+ // field as making that attribute live (see "combined with constant metadata columns"
+ // above, which hedges its assertions for the same reason). That decline fires before
+ // `buildDvScanCommon` ever runs, regardless of collision, so it cannot exercise the fix.
+ // Test the builder's dedup logic directly instead, the same way `storeUris` and
+ // `mergedObjectStoreOptions` are unit-tested without a live scan.
+ val physicalDataSchema =
+ StructType(Seq(StructField("_comet_metadata_file_path", ByteType)))
+ val physicalPartitionSchema =
+ StructType(Seq(StructField("_comet_metadata_file_size", LongType)))
+ val fileConstantMetadataColumns = Seq(
+ AttributeReference("file_path", StringType, nullable = false)(),
+ AttributeReference("file_size", LongType, nullable = false)())
+
+ val constantMetadataFields = CometNativeScan.uniqueConstantMetadataFields(
+ fileConstantMetadataColumns,
+ physicalDataSchema.fields.map(_.name).toSet ++ physicalPartitionSchema.fields
+ .map(_.name)
+ .toSet)
+ assert(
+ constantMetadataFields.map(_.name) == Seq(
+ "_comet_metadata_file_path_",
+ "_comet_metadata_file_size_"),
+ "expected both constant-metadata names to be uniquified on collision, got " +
+ s"${constantMetadataFields.map(_.name)}")
+
+ // The DV builder must feed these already-unique names into allocateUniqueInternalFields's
+ // reserved set so the internal-column suffix chain stays consistent with them.
+ val requiredSchema = StructType(
+ Seq(
+ StructField("id", LongType),
+ StructField(CometDeltaNativeScan.IsRowDeletedColumn, ByteType),
+ StructField(CometDeltaNativeScan.RowIndexColumn, LongType)))
+ val internalFields = CometDeltaNativeScan.allocateUniqueInternalFields(
+ requiredSchema,
+ physicalDataSchema,
+ physicalPartitionSchema,
+ constantMetadataFields)
+
+ val allNames = physicalDataSchema.fields.map(_.name) ++
+ physicalPartitionSchema.fields.map(_.name) ++
+ constantMetadataFields.map(_.name) ++
+ internalFields.map(_.name)
+ assert(allNames.distinct.length == allNames.length, s"expected all names distinct: $allNames")
+ }
+
+ test(
+ "non-DV shape: user column named like the synthetic constant-metadata slot keeps its " +
+ "own values") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ val collidingName = "_comet_metadata_file_path"
+ spark
+ .range(0, 100)
+ .selectExpr("id", s"cast(7 as tinyint) as `$collidingName`")
+ .write
+ .format("delta")
+ .save(path)
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .selectExpr("id", s"`$collidingName`", "_metadata.file_path as fp")
+ checkDeltaNativeScanAnswer(df)
+ val rows = df.collect()
+ val survivingValues = rows.map(_.getAs[Byte](collidingName)).distinct
+ assert(
+ survivingValues.sameElements(Array(7.toByte)),
+ "expected the user column's own value (7) to survive the constant-metadata " +
+ s"collision, got ${survivingValues.toSeq}")
+ assert(
+ rows.forall(_.getString(2).nonEmpty),
+ "expected _metadata.file_path to still report a real path")
+ }
+ }
+
+ test("deletion vectors: special characters in table path") {
+ withTempDir { base =>
+ val dir = new java.io.File(base, "s p a r k %dv% test")
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ val df = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(df)
+ assert(df.count() == 500)
+ }
+ }
+
+ test("deletion vectors: decline when row_index is consumed via multi-hop aliases") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .selectExpr("id", "_metadata.row_index as ri")
+ .selectExpr("id", "ri + 1 as ri2")
+ .filter(col("ri2") > 10)
+ checkSparkAnswer(df)
+ assert(deltaNativeScans(df).isEmpty, "derived row_index consumption must decline")
+ }
+ }
+
+ test("deletion vectors: decline when row_index feeds a non-Project operator") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .groupBy(col("_metadata.row_index") % 7)
+ .count()
+ checkSparkAnswer(df)
+ assert(deltaNativeScans(df).isEmpty, "aggregate over row_index must decline")
+ }
+ }
+
+ test("deletion vectors: decline when _metadata.row_index is referenced above the scan") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .selectExpr("id", "_metadata.row_index as ri")
+ checkSparkAnswer(df)
+ assert(
+ deltaNativeScans(df).isEmpty,
+ "plans consuming a real row_index must fall back to Spark")
+ }
+ }
+
+ /**
+ * Every [[SparkPlan]] executed during `body`, captured via a [[QueryExecutionListener]] rather
+ * than a returned `DataFrame`'s own plan: a `DataFrameWriter` action such as `.write.parquet`
+ * has no result `Dataset` to call `.queryExecution` on, so the write's physical plan -- the one
+ * `DeltaScanSupport.declineReason` actually saw -- is only observable this way.
+ */
+ private def capturePlansDuring(body: => Unit): Seq[SparkPlan] = {
Review Comment:
Reproduced with the same 300 ms sleep in `onSuccess`: the write-sink test fails with an empty reason list. Both copies of `capturePlansDuring` now call `CometListenerBusUtils.waitUntilEmpty(spark.sparkContext)` after `body`, inside the `try`, the way `CometIcebergTestBase.capturePlans` does. With the sleep still in place the suites pass, and they pass on repeated runs with it removed.
The comment on `collectTaskInputMetrics` was stale. That helper had the same race and papered over it with `eventually` and a `minRecords` floor. It now drains the bus too, the parameter is gone, and its callers keep their floor assertions. No other listener-based helper in the contrib tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4006962585
##########
dev/ci/compute-changes.py:
##########
@@ -306,6 +306,23 @@
".mvn/**",
"mvnw",
],
+ "delta": [
+ "contrib/delta/**",
Review Comment:
> A few things about this filter and the workflow it triggers
All taken: `!**.md` excluded, `contrib/delta/**` dropped, `CometTestBase.scala` and `CometS3TestBase.scala` added. In the workflow the feature-off build restores the cargo cache, the Python job runs one interpreter, the cache-save step is gone with a note that the restore falls back to the `build_linux` cache, and the `--features delta` flag is gone with a note that it is in the default set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4104403879
##########
native/core/src/parquet/parquet_exec.rs:
##########
@@ -84,6 +89,9 @@ pub(crate) fn init_datasource_exec(
encryption_enabled: bool,
use_field_id: bool,
ignore_missing_field_id: bool,
+ rebase_from_file_metadata: bool,
Review Comment:
[P1] Update every caller when adding these three parameters. Three helpers under `native/core/src/execution/operators/dynamic_filter/join/tests/` still use the 18-argument signature: `timestamp_errors.rs:86`, `schema_errors.rs:65`, and `schema_errors/partition_columns.rs:47`. Building any core unit-test selection now fails because Rust compiles these helpers even when their tests are filtered out. This also blocks CI’s `cargo clippy --all-targets --workspace`. Supply `false, "", ""` at these ordinary-scan test callers, matching the other updated fixtures.
Evidence: At the unmodified reviewed head, running `cargo test --locked --offline -p datafusion-comet --no-default-features --features delta datetime_rebase --no-run` from `native/` exits 101 with three E0061 errors: “this function takes 21 arguments but 18 arguments were supplied.” The base signature accepts 18 arguments. Adding only the three missing arguments at those test call sites in a disposable copy allows the test binary to compile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3920759711
##########
contrib/delta-spark/src/test/scala/org/apache/comet/contrib/delta/JvmLowercaseParitySuite.scala:
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.util.Locale
+
+import org.scalatest.funsuite.AnyFunSuite
+
+import org.apache.comet.serde.operator.JvmCaseTables
+
+/**
+ * Self-validating proof that `JvmCaseTables.mirrorLowercase` -- a line-for-line mirror of the
+ * native `JvmCaseTables::lowercase` in `native/core/src/parquet/schema_adapter.rs`, run over the
+ * tables `JvmCaseTables` generates from the RUNNING JVM -- reproduces this JVM's
+ * `String.toLowerCase(Locale.ROOT)`. Because both the tables and the expectations come from the
+ * same running JVM, this suite passes on ANY supported JDK by construction.
+ *
+ * Needs no `SparkSession`, so it extends `AnyFunSuite` directly rather than `CometDeltaTestBase`.
+ */
+class JvmLowercaseParitySuite extends AnyFunSuite {
Review Comment:
The parity suite is deleted along with the Scala mirror it tested. Main's folding calls the JVM directly through JNI, so there is no second implementation left to drift; #5602 ships its own coverage in CometNativeReaderSuite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3818688990
##########
native/core/src/execution/planner/delta_spark_scan.rs:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! JVM-planned Delta handler for the generic `OpStruct::ContribScan` dispatcher, feature-gated
+//! behind `delta`.
+//!
+//! delta-spark has already done log replay, snapshot resolution, and partition pruning by the
+//! time the scan reaches Comet, so the envelope carries a concrete file list (plus deletion
+//! vector descriptors) and the read path reuses the exact same shared parquet scan builder as
+//! `NativeScan` -- inheriting row-group stats pruning, page-index pruning, and filter pushdown.
+//! Sibling of the kernel-planned handler in `delta_scan.rs`; the two claim different
+//! `type_url`s within the same `ContribScan` envelope.
+
+use std::collections::HashMap;
+
+use datafusion_comet_proto::spark_operator::{
+ ContribScan, DeltaSparkScan, Operator, SparkFilePartition,
+};
+use prost::Message;
+
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::execution::planner::PhysicalPlanner;
+use crate::execution::planner::PlanCreationResult;
+
+/// Type name the JVM-planned Delta contrib claims within the `ContribScan` envelope. The
+/// contrib jar packs a `DeltaSparkScan` with a `type_url` of
+/// `type.googleapis.com/comet.contrib.delta_spark.DeltaSparkScan`; dispatch keys on the
+/// contrib-owned suffix, same convention as the kernel path's `delta_scan.rs`.
+const DELTA_SPARK_SCAN_TYPE_NAME: &str = "comet.contrib.delta_spark.DeltaSparkScan";
+
+/// Contrib entry point for the `OpStruct::ContribScan` dispatcher. Returns `Some(result)` when
+/// the envelope carries a JVM-planned Delta scan, or `None` when the `type_url` belongs to some
+/// other contrib.
+pub(crate) fn try_plan_contrib_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ contrib: &ContribScan,
+) -> Option<PlanCreationResult> {
+ if !contrib.type_url.ends_with(DELTA_SPARK_SCAN_TYPE_NAME) {
+ return None;
+ }
+ Some(
+ DeltaSparkScan::decode(contrib.value.as_slice())
+ .map_err(|e| {
+ GeneralError(format!(
+ "Failed to decode DeltaSparkScan from contrib_scan: {e}"
+ ))
+ })
+ .and_then(|scan| plan_delta_spark_scan(planner, spark_plan, &scan)),
+ )
+}
+
+fn plan_delta_spark_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ scan: &DeltaSparkScan,
+) -> PlanCreationResult {
+ // Delta data files are plain parquet; the read path deliberately reuses
+ // the same shared parquet scan builder as NativeScan so Delta inherits
+ // row-group stats pruning, page-index pruning, and filter pushdown. Only
+ // the file list arrives in Delta-specific form. Note delta_common's
+ // column_mapping_mode is informational in M1: the actual field-id
+ // matching switch is common.use_field_id, same as the Iceberg path.
+ let common = scan
+ .common
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing common data".into()))?;
+
+ let delta_partition = scan
+ .file_partition
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing file_partition".into()))?;
+
+ let spark_partition = SparkFilePartition {
+ partitioned_file: delta_partition
+ .partitioned_file
+ .iter()
+ .map(|f| {
+ f.file
+ .clone()
+ .ok_or_else(|| GeneralError("DeltaSparkPartitionedFile missing inner file".into()))
+ })
+ .collect::<Result<Vec<_>, _>>()?,
+ };
+
+ let (object_store_url, mut files) =
+ planner.prepare_scan_store_and_files(common, &spark_partition)?;
Review Comment:
**[P1] Preserve the object-store authority of every Delta data file**
Could we decline mixed-store scans before claiming them, or preserve each file's actual store identity? `prepare_scan_store_and_files` chooses the first file's store, while `get_partitioned_files` reduces every URL to its path. A shallow clone from bucket A into bucket B followed by an append is a valid Delta table containing absolute source files and new target files. The size-based file packing can put both in one task, which then requests B's key from A. That normally produces `NoSuchKey`, but it can read the wrong data if the same key exists in A. This assumption was already present in the generic helper, but the new Delta reader makes it reachable through ordinary shallow-clone behavior. Please cover a cross-bucket clone plus append and preserve Spark's file order if the scan is split internally.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
Review Comment:
**[P2] Restore logical nested names in the scan output**
Could we separate the physical read schema from the logical output schema and restore nested field names before parent expressions run? `toPhysical` rewrites nested names as well as top-level names. The shared native builder returns that physical schema directly, and the native `ToJson` implementation takes JSON keys from the actual Arrow struct fields. For a name-mapped Delta column `s STRUCT<a: BIGINT>`, enabling `spark.comet.expression.StructsToJson.allowIncompatible=true` makes `SELECT to_json(s)` reach this path. Spark returns `{"a":7}`, but the native expression receives the physical `col-...` name. A stock Delta 4 probe confirms the logical and physical schemas differ this way. Please cover a mapped nested struct with a native expression that observes field names, rather than only positional field access.
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,587 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::ParquetAccessPlan;
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::ObjectStoreExt;
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::PageIndexPolicy;
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::parquet::parquet_support::prepare_object_store_with_configs;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
+ }
+ cursor = row + 1;
+ }
+ if deleted_in_group == num_rows && num_rows > 0 {
+ plan.skip(idx);
+ } else if deleted_in_group > 0 {
+ if group_end > cursor {
+ selectors.push(RowSelector::select((group_end - cursor) as usize));
+ }
+ plan.scan_selection(idx, RowSelection::from(selectors));
+ }
+ group_start = group_end;
+ }
+ // A deleted index beyond the file's total row count means the DV does not
+ // belong to this file (stale or corrupted metadata); silently dropping it
+ // would under-apply deletions.
+ if let Some(&row) = deleted_iter.peek() {
+ return Err(GeneralError(format!(
+ "Deletion vector marks row {row} but the file only has {group_start} rows"
+ )));
+ }
+ Ok(plan)
+}
+
+/// One data file plus everything needed to apply its deletion vector. The
+/// file's size comes from `file.object_meta.size` (built by the planner from
+/// the proto's `file_size`).
+pub struct DvScanFile {
+ pub file: PartitionedFile,
+ /// Full URL of the data file (proto `file_path`).
+ pub file_path: String,
+ pub dv: Option<DeltaSparkDvDescriptor>,
+}
+
+/// Upper bound on concurrent DV-blob and footer fetches per partition. Both
+/// are small ranged reads, so a modest fan-out hides object-store latency
+/// without flooding the store client.
+const DV_FETCH_CONCURRENCY: usize = 8;
+
+/// Called via `block_on` at plan-creation time on the executor task: DV blobs
+/// are small ranged reads and footers are needed to learn row-group
+/// boundaries. Files are fetched concurrently (bounded by
+/// [`DV_FETCH_CONCURRENCY`]) with input order preserved. Footer fetches go
+/// through the scan's shared FileMetadataCache, so the scan's subsequent open
+/// of the same file is served from cache. That reuse relies on each input
+/// [`PartitionedFile`] being returned as-is (only `with_extension` applied),
+/// never rebuilt: the cache entry is keyed by this exact `object_meta` and the
+/// scan later looks it up through the same struct.
+pub async fn attach_access_plans(
+ runtime_env: Arc<RuntimeEnv>,
+ object_store_options: &HashMap<String, String>,
+ files: Vec<DvScanFile>,
+) -> Result<Vec<PartitionedFile>, ExecutionError> {
+ futures::stream::iter(files)
+ .map(|scan_file| {
+ attach_access_plan(Arc::clone(&runtime_env), object_store_options, scan_file)
+ })
+ .buffered(DV_FETCH_CONCURRENCY)
+ .try_collect()
+ .await
+}
+
+/// Resolve one file's deletion vector into an attached [`ParquetAccessPlan`];
+/// files without a DV pass through untouched.
+async fn attach_access_plan(
+ runtime_env: Arc<RuntimeEnv>,
+ object_store_options: &HashMap<String, String>,
+ scan_file: DvScanFile,
+) -> Result<PartitionedFile, ExecutionError> {
+ let DvScanFile {
+ file,
+ file_path,
+ dv,
+ } = scan_file;
+ let dv = match dv {
+ Some(dv) => dv,
+ None => return Ok(file),
+ };
+ if dv.size_in_bytes < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative size {}",
+ dv.size_in_bytes
+ )));
+ }
+
+ let data: Vec<u8> = if let Some(inline) = dv.inline_data {
+ inline
+ } else if let Some(dv_path) = &dv.absolute_path {
+ let offset = dv
+ .offset
+ .ok_or_else(|| GeneralError("On-disk deletion vector missing offset".into()))?;
+ if offset < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative offset {offset}"
+ )));
+ }
+ let offset = offset as u64;
+ // [i32 BE size][data: size_in_bytes][i32 BE crc]
+ let framed_len = 4 + dv.size_in_bytes as u64 + 4;
+ let (dv_url, dv_store_path) = prepare_object_store_with_configs(
+ Arc::clone(&runtime_env),
+ dv_path.clone(),
+ object_store_options,
+ )?;
+ let store = runtime_env.object_store(&dv_url)?;
+ let blob = store
+ .get_range(&dv_store_path, offset..offset + framed_len)
+ .await
+ .map_err(|e| GeneralError(format!("Failed to read deletion vector {dv_path}: {e}")))?;
+ unframe_dv_blob(&blob, dv.size_in_bytes as usize)?.to_vec()
+ } else {
+ return Err(GeneralError(
+ "Deletion vector descriptor has neither inline data nor a path".into(),
+ ));
+ };
+ let deleted = deserialize_dv_bitmap(&data)
+ .map_err(|e| GeneralError(format!("Invalid deletion vector for {file_path}: {e}")))?;
Review Comment:
**[P2] Validate the decoded DV cardinality**
Could we compare `deleted.len()` with `dv.cardinality` before applying the bitmap? The descriptor carries the expected count, but the native reader never checks it. A valid-CRC bitmap containing only row 1, paired with a descriptor declaring two deletions, is accepted by the current decoder and access-plan builder and selects 9 of 10 rows. Delta's JVM reader rejects this mismatch in [`StoredBitmap.validateCardinality`](https://github.com/delta-io/delta/blob/v4.0.0/spark/src/main/scala/org/apache/spark/sql/delta/deletionvectors/StoredBitmap.scala#L70-L96). CRC and row-range validation do not catch a stale but otherwise well-formed bitmap. Please add a mismatched-cardinality case alongside the existing corruption checks.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ val common = commonBuilder.build()
+ val tableRoot = relation.location.rootPaths.head.toString
+ val deltaCommon = OperatorOuterClass.DeltaSparkScanCommon
+ .newBuilder()
+ .setTableRoot(tableRoot)
+ .setColumnMappingMode(columnMappingMode(scanExec))
+ .setSourceKey(DeltaPlanDataInjector.sourceKey(tableRoot, common))
+ .build()
+ val deltaScan = OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaCommon)
+ Operator
+ .newBuilder()
+ .setPlanId(scanExec.id)
+ .setContribScan(DeltaSparkScanEnvelope.pack(deltaScan.build()))
+ .build()
+ }
+ }
+
+ /**
+ * Harvest subquery-bearing predicates for this scan from its covering FilterExec. Spark 3.x
+ * strips them from a scan's `dataFilters` at planning (`FileSourceStrategy` routes them to the
+ * post-scan filter only), while Spark 4.x keeps them in `dataFilters`. Collecting them here at
+ * claim time gives the execution-time resolve-and-push path the same inputs on every Spark
+ * version; the dedup keeps Spark 4.x from carrying duplicates.
+ */
+ def subqueryFiltersFromParent(
+ plan: org.apache.spark.sql.execution.SparkPlan,
+ scanExec: FileSourceScanExec): Seq[org.apache.spark.sql.catalyst.expressions.Expression] = {
+ import org.apache.spark.sql.catalyst.expressions.{PlanExpression, SubqueryExpression}
+ // Nearest FilterExec above the scan (the DV shape interposes nodes between them, so do
+ // not require a direct parent-child edge). Safety comes from the reference guard, not the
+ // plan walk: only conjuncts expressed directly over the scan's own output attributes
+ // survive, so filters from other branches or over aliased projections contribute nothing.
+ val filtersAboveScan = plan.collect {
+ case f: org.apache.spark.sql.execution.FilterExec if f.find(_ eq scanExec).isDefined => f
+ }
+ filtersAboveScan.lastOption
+ .map { f =>
+ splitConjunctivePredicates(f.condition)
+ .filter(_.references.subsetOf(scanExec.outputSet))
Review Comment:
**[P1] Keep scalar-subquery pushdown above non-commuting operators**
Could we restrict this walk to plan shapes where moving the predicate into the scan is safe? The reference check does not protect against `LIMIT` or TopN. With a Delta table `t` containing IDs 0, 1, and 2, `SELECT id FROM (SELECT id FROM t ORDER BY id LIMIT 1) q WHERE id > (SELECT max(id) FROM range(1))` must return no rows. A stock Spark 4.0.3 / Delta 4.0.0 probe produces `Filter -> TakeOrderedAndProject -> FileSourceScanExec`, with empty scan `dataFilters`, and this guard accepts the outer predicate. Moving `id > 0` below TopN returns row 1 instead. The execution-time path then installs that predicate in the native Parquet scan, so enabling row-filter pushdown can change the result. Please keep the covering filter and only harvest across operators for which pushdown is valid. This query would make a useful regression test.
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,587 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::ParquetAccessPlan;
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::ObjectStoreExt;
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::PageIndexPolicy;
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::parquet::parquet_support::prepare_object_store_with_configs;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
+ }
+ cursor = row + 1;
+ }
+ if deleted_in_group == num_rows && num_rows > 0 {
+ plan.skip(idx);
+ } else if deleted_in_group > 0 {
+ if group_end > cursor {
+ selectors.push(RowSelector::select((group_end - cursor) as usize));
+ }
+ plan.scan_selection(idx, RowSelection::from(selectors));
+ }
+ group_start = group_end;
+ }
+ // A deleted index beyond the file's total row count means the DV does not
+ // belong to this file (stale or corrupted metadata); silently dropping it
+ // would under-apply deletions.
+ if let Some(&row) = deleted_iter.peek() {
+ return Err(GeneralError(format!(
+ "Deletion vector marks row {row} but the file only has {group_start} rows"
+ )));
+ }
+ Ok(plan)
+}
+
+/// One data file plus everything needed to apply its deletion vector. The
+/// file's size comes from `file.object_meta.size` (built by the planner from
+/// the proto's `file_size`).
+pub struct DvScanFile {
+ pub file: PartitionedFile,
+ /// Full URL of the data file (proto `file_path`).
+ pub file_path: String,
+ pub dv: Option<DeltaSparkDvDescriptor>,
+}
+
+/// Upper bound on concurrent DV-blob and footer fetches per partition. Both
+/// are small ranged reads, so a modest fan-out hides object-store latency
+/// without flooding the store client.
+const DV_FETCH_CONCURRENCY: usize = 8;
+
+/// Called via `block_on` at plan-creation time on the executor task: DV blobs
+/// are small ranged reads and footers are needed to learn row-group
+/// boundaries. Files are fetched concurrently (bounded by
+/// [`DV_FETCH_CONCURRENCY`]) with input order preserved. Footer fetches go
+/// through the scan's shared FileMetadataCache, so the scan's subsequent open
+/// of the same file is served from cache. That reuse relies on each input
+/// [`PartitionedFile`] being returned as-is (only `with_extension` applied),
+/// never rebuilt: the cache entry is keyed by this exact `object_meta` and the
+/// scan later looks it up through the same struct.
+pub async fn attach_access_plans(
+ runtime_env: Arc<RuntimeEnv>,
+ object_store_options: &HashMap<String, String>,
+ files: Vec<DvScanFile>,
+) -> Result<Vec<PartitionedFile>, ExecutionError> {
+ futures::stream::iter(files)
+ .map(|scan_file| {
+ attach_access_plan(Arc::clone(&runtime_env), object_store_options, scan_file)
+ })
+ .buffered(DV_FETCH_CONCURRENCY)
+ .try_collect()
+ .await
+}
+
+/// Resolve one file's deletion vector into an attached [`ParquetAccessPlan`];
+/// files without a DV pass through untouched.
+async fn attach_access_plan(
+ runtime_env: Arc<RuntimeEnv>,
+ object_store_options: &HashMap<String, String>,
+ scan_file: DvScanFile,
+) -> Result<PartitionedFile, ExecutionError> {
+ let DvScanFile {
+ file,
+ file_path,
+ dv,
+ } = scan_file;
+ let dv = match dv {
+ Some(dv) => dv,
+ None => return Ok(file),
+ };
+ if dv.size_in_bytes < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative size {}",
+ dv.size_in_bytes
+ )));
+ }
+
+ let data: Vec<u8> = if let Some(inline) = dv.inline_data {
+ inline
+ } else if let Some(dv_path) = &dv.absolute_path {
+ let offset = dv
+ .offset
+ .ok_or_else(|| GeneralError("On-disk deletion vector missing offset".into()))?;
+ if offset < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative offset {offset}"
+ )));
+ }
+ let offset = offset as u64;
+ // [i32 BE size][data: size_in_bytes][i32 BE crc]
+ let framed_len = 4 + dv.size_in_bytes as u64 + 4;
+ let (dv_url, dv_store_path) = prepare_object_store_with_configs(
+ Arc::clone(&runtime_env),
+ dv_path.clone(),
+ object_store_options,
+ )?;
Review Comment:
**[P2] Avoid constructing a cold S3 store inside the DV runtime**
Could we resolve the required stores before entering `attach_access_plans`, or make their initialization async-safe? The caller enters `get_runtime().block_on(...)`, but an uncached S3 sidecar reaches this synchronous helper and then `objectstore/s3.rs` calls `get_runtime().block_on(build_credential_provider(...))` again. Tokio rejects that nested `Handle::block_on` with a panic. A fresh executor reading a shallow clone whose data is in bucket A and whose new DV is in bucket B reaches a cold cache entry. Same-bucket tests hide the problem because the data store was created before the outer `block_on`. Explicit endpoint/region or static Hadoop credentials do not avoid the inner credential-provider call. Please add a test with distinct data-file and DV buckets.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
Review Comment:
**[P2] Include configuration for external DV providers**
Could we derive execution-time object-store options from every selected data-file and external-DV authority, or decline combinations that cannot be represented? The common configuration is extracted using only this first data file's scheme, while `extractDvDescriptor` permits an absolute DV URI on another provider. For an S3 data file with an ABFS sidecar, `NativeConfig` forwards `fs.s3a.*` but omits the Hadoop Azure account-key/OAuth settings. The native DV loader then opens the ABFS URI using that same S3-only map, so a table readable by Spark can fail authentication. This is separate from mixed-bucket data-file routing and occurs with a single data file. Same-scheme S3 bucket overrides are already forwarded. A cross-provider sidecar test would cover the missing case.
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,587 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::ParquetAccessPlan;
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::ObjectStoreExt;
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::PageIndexPolicy;
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::parquet::parquet_support::prepare_object_store_with_configs;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
+ }
+ cursor = row + 1;
+ }
+ if deleted_in_group == num_rows && num_rows > 0 {
+ plan.skip(idx);
+ } else if deleted_in_group > 0 {
+ if group_end > cursor {
+ selectors.push(RowSelector::select((group_end - cursor) as usize));
+ }
+ plan.scan_selection(idx, RowSelection::from(selectors));
+ }
+ group_start = group_end;
+ }
+ // A deleted index beyond the file's total row count means the DV does not
+ // belong to this file (stale or corrupted metadata); silently dropping it
+ // would under-apply deletions.
+ if let Some(&row) = deleted_iter.peek() {
+ return Err(GeneralError(format!(
+ "Deletion vector marks row {row} but the file only has {group_start} rows"
+ )));
+ }
+ Ok(plan)
+}
+
+/// One data file plus everything needed to apply its deletion vector. The
+/// file's size comes from `file.object_meta.size` (built by the planner from
+/// the proto's `file_size`).
+pub struct DvScanFile {
+ pub file: PartitionedFile,
+ /// Full URL of the data file (proto `file_path`).
+ pub file_path: String,
+ pub dv: Option<DeltaSparkDvDescriptor>,
+}
+
+/// Upper bound on concurrent DV-blob and footer fetches per partition. Both
+/// are small ranged reads, so a modest fan-out hides object-store latency
+/// without flooding the store client.
+const DV_FETCH_CONCURRENCY: usize = 8;
+
+/// Called via `block_on` at plan-creation time on the executor task: DV blobs
+/// are small ranged reads and footers are needed to learn row-group
+/// boundaries. Files are fetched concurrently (bounded by
+/// [`DV_FETCH_CONCURRENCY`]) with input order preserved. Footer fetches go
+/// through the scan's shared FileMetadataCache, so the scan's subsequent open
+/// of the same file is served from cache. That reuse relies on each input
+/// [`PartitionedFile`] being returned as-is (only `with_extension` applied),
+/// never rebuilt: the cache entry is keyed by this exact `object_meta` and the
+/// scan later looks it up through the same struct.
+pub async fn attach_access_plans(
+ runtime_env: Arc<RuntimeEnv>,
+ object_store_options: &HashMap<String, String>,
+ files: Vec<DvScanFile>,
+) -> Result<Vec<PartitionedFile>, ExecutionError> {
+ futures::stream::iter(files)
+ .map(|scan_file| {
+ attach_access_plan(Arc::clone(&runtime_env), object_store_options, scan_file)
+ })
+ .buffered(DV_FETCH_CONCURRENCY)
+ .try_collect()
+ .await
+}
+
+/// Resolve one file's deletion vector into an attached [`ParquetAccessPlan`];
+/// files without a DV pass through untouched.
+async fn attach_access_plan(
+ runtime_env: Arc<RuntimeEnv>,
+ object_store_options: &HashMap<String, String>,
+ scan_file: DvScanFile,
+) -> Result<PartitionedFile, ExecutionError> {
+ let DvScanFile {
+ file,
+ file_path,
+ dv,
+ } = scan_file;
+ let dv = match dv {
+ Some(dv) => dv,
+ None => return Ok(file),
+ };
Review Comment:
**[P2] Treat an empty DV as a pass-through file**
Could we handle the canonical zero-cardinality descriptor before attempting bitmap decoding? Delta's `DeletionVectorDescriptor.EMPTY` has inline storage, an empty payload, size 0, and cardinality 0. I checked a committed Delta 4 table containing that exact descriptor: the actual Spark `FilePartition` carries it and the normal Delta reader returns all rows. Here it becomes `Some(empty_bytes)` and `deserialize_dv_bitmap` returns `Deletion vector bitmap too short for magic number`. Returning the unchanged file for the empty-DV case would match Spark. Please add this to the existing `attach_access_plans` test.
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,587 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::ParquetAccessPlan;
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::ObjectStoreExt;
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::PageIndexPolicy;
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::parquet::parquet_support::prepare_object_store_with_configs;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
Review Comment:
**[P1] Bound memory used by expanded DV row selections**
Could we account for and bound these allocations, preferably creating the access plan when its file is opened? An alternating deleted/retained bitmap creates one non-coalescing `RowSelector` per row. A bounded probe using these unchanged functions and the exact Roaring 0.11.4 / Parquet 58.4.0 types expanded an 8,224-byte bitmap for 65,536 rows into 65,536 selectors, retaining 1,048,600 bytes and peaking at 2,097,176 bytes. An 8-million-row group therefore needs roughly 128 MiB for the retained selectors alone. `attach_access_plans().buffered(8).try_collect()` bounds fetch concurrency, but retains all completed file plans before the scan starts. None of this allocation is reserved against the execution memory pool. A multi-file test with distinct alternating-row DVs would exercise the executor-OOM case without relying on malformed input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3836625201
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme (finding 8).
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
Review Comment:
Good catch. Added the three long bucket aliases to the comparison set (checked against S3AUtils.lookupPassword in 3.4.1: long first, short overrides it, then global), plus the long form of the per-bucket provider path in the S3A-scoped arm. Both of your cases now decline, long-only in JCEKS and long aliases shadowed by different plain globals, with regressions for each reusing the JCEKS fixture.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3834792911
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,613 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.net.URI
+import java.util.Locale
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.CometScanRule
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands. Anything else on the protocol declines the
+ * table. Note `deletionVectors` and `columnMapping` are declined separately (below) so their
+ * fallback reasons are specific.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format (not a further subclass)? Compared by class name,
+ * not `classOf`, deliberately: this is the first gate on every V1 scan, and it must stay inert
+ * when the contrib jar is deployed without delta-spark on the classpath:
+ * `classOf[DeltaParquetFileFormat]` here raises NoClassDefFoundError inside CometScanRule and
+ * takes down every parquet scan in the session. When the name matches, delta-spark is
+ * necessarily present (the instance exists), so the Delta types past this gate are safe.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Returns the first reason this Delta scan cannot go native, or None when it is claimable. Only
+ * called when [[isDeltaScan]] is true. `scanHelper` is the same [[CometScanExec]] the caller
+ * builds to drive [[CometDeltaNativeScan.convert]] on a claim, reused here (rather than listed
+ * separately) to resolve the scan's selected files for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Descriptor deserialization is the expensive part of both consumers below (the DV
+ // cardinality gate and the store-identity collision gate); hoisted once here so it runs at
+ // most once per claim attempt regardless of how many gates end up needing it. `lazy` because
+ // most scans are not DV-shaped and selectedDvDescriptors short-circuits to Seq.empty for
+ // them, but paying even that check is unnecessary work for gates that return earlier.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles with useMetadataRowIndex=false) injects a
+ // generated row-index column directly into the data schema and disables reader
+ // optimizations; its values must come from Spark's reader. Claiming such a scan would
+ // feed NULL row indexes into deletion-vector construction, silently corrupting DML.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ // Name mode is supported by serializing physical-name schemas (the parquet reader then
+ // matches file columns by name natively). Id mode needs the field-id path and stays
+ // declined until validated.
+ val cmMode = metadata.columnMappingMode.name
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping
+ // EXISTS_DEFAULT; decline any column defaults under column mapping rather than
+ // return nulls where a default belongs.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names (not just top-level column
+ // names) to their physical, column-mapped form, and the shared native builder uses the
+ // required schema verbatim as the scan's output schema: struct fields below the top level
+ // would carry physical names. Ordinal access (GetStructField) is unaffected, but
+ // name-sensitive native expressions (e.g. to_json) read the Arrow struct field names
+ // directly and would leak physical names into query results. Restoring logical names
+ // natively needs a rename adapter/proto field for the logical schema (follow-up); decline
+ // until then.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and not
+ // supported, except Delta's DV bookkeeping columns, which the native path emits as
+ // constants (correct by construction once the DV is applied in the reader).
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is not a DV read: it is Delta DML
+ // bookkeeping (findTouchedFiles building deletion bitmaps from REAL row indexes).
+ // Claiming it with constant row indexes would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // The internal columns must form a suffix of the read schema so data-column
+ // positions agree between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // The row-index column's real values are consumed inside the reader when Spark applies
+ // the DV; native applies the DV itself and emits a dead constant instead, so the value
+ // must be provably unused above the scan (beyond the _metadata reassembly that gets
+ // discarded).
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet; decline rather
+ // than silently return nulls for backfilled columns in old files.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bound the memory the native side will retain for expanded DV row selectors before
+ // committing to native execution: applying a deletion vector expands it into per-row
+ // RowSelectors that are reserved against the execution memory pool at scan time (see
+ // delta_dv.rs), and an alternating deleted/retained bitmap produces one non-coalescing
+ // selector per row. A row group's selector count is bounded above by
+ // 2*cardinality + #row-groups (each deleted row splits at most one run into a
+ // select/skip pair, plus one selector per row-group boundary), so the descriptor's
+ // cardinality -- deserialized at planning time via selectedDvDescriptors, no bitmap
+ // decode needed -- is a sound, pessimistic upper bound on the native reservation.
+ // Pessimistic by design: a large but CONTIGUOUS deletion is declined the same as a
+ // large alternating one, even though it would retain far fewer selectors natively; the
+ // conf below makes that recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from InputFileBlockHolder, a thread-local set by Spark's
+ // FileScanRDD; the native scan does not populate it. Delta's own DELETE/UPDATE/MERGE
+ // find-touched-files scans use input_file_name, so this gate is load-bearing for DML
+ // correctness (mirrors core's check in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core).
+ // The DV shape's trailing row-index column is exempt: the gates above already proved its
+ // values are dead and the native path emits a constant for it.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does
+ // not yet wire the decryption-key broadcast to executors, so claiming even a
+ // supported-encryption scan would fail at execution.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults (schema-evolution backfill of map/struct/array columns)
+ // cannot be serialized; a silently-dropped default would misalign the value/index lists
+ // consumed positionally on the native side. Mirrors core's transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read; otherwise a custom Hadoop FileSystem would fail at execution instead of
+ // falling back gracefully. Mirrors core's unsupportedFsSchemes gate.
+ val libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+ val unsupportedFsSchemes = scanExec.relation.location.rootPaths
Review Comment:
Fixed. The scheme gate now runs over the selected data-file URIs and the DV absolute paths, the same sequences the later authority gates already collect, using the exact predicate the root-paths gate had (lowercased, null tolerant, libhdfs exemption honored at the new call site). It sits ahead of the multi-store gate since an unreadable scheme is the stronger and more actionable reason, and both authority gates presume the URIs are natively resolvable. s3a is recognized by the native scheme parser so the MinIO coverage is untouched. Your probe is now a CI test: the suite mounts viewfs over a local directory, writes through it, shallow-clones to a file: root, and asserts the scan falls back with the scheme reason while answers match, including a mixed-scheme shape that pins the gate ordering end to end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3834790332
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,587 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::ParquetAccessPlan;
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::ObjectStoreExt;
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::PageIndexPolicy;
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::parquet::parquet_support::prepare_object_store_with_configs;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
Review Comment:
Reworked, and your two numbers decompose exactly from source: 32,000,000 attached + 32,000,000 clone (Vec::clone allocates exact capacity on the path your probe took) + 33,554,432 combined (the power-of-two one: into_overall_row_selection collects through a FlatMap whose size_hint is zero, so the vec starts empty and doubles) = 97,554,432, plus 25 bytes we did not attribute. Your retained 65,554,432 falls out the same way since the clone's buffer is moved and dropped during the flat_map while the combined vec keeps its rounded capacity.
One thing your probe could not see: with a pushed-down predicate, page-index pruning intersects the clone's selection before normalization through another zero-size-hint collect, so the clone itself picks up power-of-two capacity too. Measured that case as well (R just above 2^20: true peak 83,886,096 against the old bound's 67,109,024), so even a factor-3 steady state would have been short.
The reservation is now taken up front at (5*selectors + 10*row_groups) * selector_size, which covers both measured cases (224 B margin on the tight one, 1.64x over your measured peak on yours), holds per-run capacity rounding when split_runs is active, and provably never grows at the post-construction resize. Ownership transfer is not achievable from our side at DataFusion 54.1: Extensions has no remove or take, values are Arc-shared and the opener clones the PartitionedFile, and prepare/into_overall_row_selection take self by value. Happy to file the upstream API request if you think it is worth pursuing.
Being upfront about the costs: at the default cardinality cap the per-file steady reservation goes from 64MB to 160MB (2.44x the truly retained bytes, since one number must cover the transient peak with no shrink hook), up to 8 admissions are in flight concurrently, and rejection is an execution-time task failure rather than a fallback, with maxDeletedRowsPerFile as the knob. Still outside the bound: the per-run plan spines when runs split (O(G^2) row-group-scale bytes that DataFusion charges for no parquet scan) and that 25 B residue. A follow-up I would take: restoring a total-rows clamp on the admission bound, which halves the reserve for descriptors whose cardinality exceeds half the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5302683820
**Update:** pushed two follow-up commits extending the scan's pruning and
object-store behavior.
**`perf: fetch Delta deletion vectors and footers concurrently`** DV blob and footer reads were sequential: two serial round-trips per DV'd file before the scan could start, which scales badly on object stores. They now fetch with a bounded fan-out of 8, preserving file order and fail-fast error semantics. Covered by a new end-to-end unit test (inline DVs, on-disk DVs, pass-through files, exact row selections, output ordering).
**`feat: push resolved scalar-subquery filters into the native Delta scan`** predicates like `id >= (SELECT max(ts) FROM checkpoint)` previously contributed nothing to the native scan: subquery results don't exist at planning, so the scan
decoded the full table and Spark's covering FilterExec did all the filtering. They are now resolved at execution time and appended as pushed filters, so row-group and page-index pruning fire the same as for literal bounds. Three version-specific traps handled:
1. Spark 3.x strips subquery predicates from a scan's `dataFilters` (`FileSourceStrategy`); Spark 4.x keeps them. The contrib harvests them from the covering FilterExec at claim time and dedups, so both behaviors converge.
2. The DV plan shape interposes nodes between the filter and the scan, so the harvest matches the nearest filter above the scan, guarded by references scan output.
3. `MergeScalarSubqueries` fuses multiple scalar subqueries into one struct-returning subquery accessed via `GetStructField`; that subtree is folded to a literal before serialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3910169029
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1714 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Zero-I/O in the common case, like the encryption gate above: native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ /**
+ * `CommonConfigurationKeysPublic.HADOOP_SECURITY_CREDENTIAL_CLEAR_TEXT_FALLBACK`, default
+ * `true`, verified via `javap` against `hadoop-common` 3.3.4's
+ * `Configuration#getPasswordFromConfig`: `getPassword` only falls back to reading a plaintext
+ * conf value once `getBoolean(<this key>, true)` holds -- with the flag off, a plaintext value
+ * is invisible to every `getPassword`-based resolver, even when no credential provider is
+ * configured at all.
+ */
+ private val ClearTextFallbackKey = "hadoop.security.credential.clear-text-fallback"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * How Hadoop's OWN consumer reads one of the keys compared by [[s3ConfigDivergenceReason]],
+ * which decides how [[s3KeyDivergenceReason]] computes the Hadoop-effective side of its
+ * equality check. Exactly two consumer families exist among [[AllS3ConfigKeys]] in `hadoop-aws`
+ * 3.3.4, each verified via `javap`/CFR against the real call sites (cited per key on
+ * [[S3ConfigKeyConsumers]]). The tier must mirror the key's ACTUAL consumer: resolving a
+ * [[PropagatedOptionConsumer]] key through the wider `lookupPassword` cascade is NOT fail-safe
+ * for a value-EQUALITY comparator -- a long-form alias value Hadoop itself never reads can
+ * EQUAL native's resolution while Hadoop's true propagate-then-plain-get value differs, turning
+ * a real divergence into a wrongly-claimed scan (the endpoint `${...}`-redirect shape pinned in
+ * `DeltaScanContribSuite`).
+ */
+ private[delta] sealed trait S3ConfigConsumer
+
+ /**
+ * Read via `S3AUtils#lookupPassword(bucket, conf, baseKey)`, verified via `javap` against
+ * `hadoop-aws` 3.3.4: builds `longBucketKey = "fs.s3a.bucket." + bucket + "." + baseKey` (the
+ * FULL, already-`fs.s3a`-prefixed base key appended after the bucket segment) and reads it via
+ * `Configuration#getPassword` BEFORE the short-bucket key, keeping the long value whenever
+ * `getPassword` returns non-empty and only falling through to short-then-global otherwise.
+ * `getPassword` is Hadoop-credential-provider-aware and skips plaintext conf entirely when
+ * [[ClearTextFallbackKey]] is false. Modeled by [[hadoopLookupPasswordEffective]].
+ */
+ private[delta] case object LookupPasswordConsumer extends S3ConfigConsumer
+
+ /**
+ * Read via `S3AUtils#propagateBucketOptions` followed by a plain `Configuration#get`-family
+ * call (`getTrimmed`/`getBoolean`/`getClasses`) against the propagated view: the short bucket
+ * form wins only by having overwritten the global key during propagation, the long bucket form
+ * folds into an unread `fs.s3a.fs.s3a.*` key, and neither a credential provider nor
+ * [[ClearTextFallbackKey]] is ever consulted. Modeled as a plain `Configuration#get` on the
+ * [[propagateBucketOptions]] result, which also expands `${...}` references under that
+ * propagated view exactly like the real consumer.
+ */
+ private[delta] case object PropagatedOptionConsumer extends S3ConfigConsumer
+
+ /**
+ * Every `fs.s3a.*` base key that governs whether a claimed native scan actually behaves like
+ * Hadoop's own reader would, paired with the consumer family Hadoop resolves it through -- ONE
+ * list, with each key's resolution tier declared beside it, so a key can never sit in the
+ * comparator without a deliberate classification (adding one without picking a tier does not
+ * compile). The entries are every per-bucket `fs.s3a.*` base key native's S3 client's
+ * `get_config` (s3.rs) resolves, verified directly against its call sites:
+ * `extract_s3_config_options` (endpoint.region, path.style.access, endpoint,
+ * requester.pays.enabled), `lookup_provider_class` (the Comet-specific
+ * credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name).
+ *
+ * Tier assignments, each verified via `javap`/CFR against `hadoop-aws` 3.3.4:
+ * - access.key/secret.key/session.token: `S3AUtils#getAWSAccessKeys` and
+ * `MarshalledCredentialBinding#fromFileSystem` (reached from
+ * `TemporaryAWSCredentialsProvider`) resolve all three via `S3AUtils#lookupPassword` --
+ * [[LookupPasswordConsumer]].
+ * - aws.credentials.provider and assumed.role.credentials.provider:
+ * `S3AUtils#buildAWSProviderList` -> `loadAWSProviderClasses` -> plain
+ * `Configuration#getClasses` -- [[PropagatedOptionConsumer]].
+ * - assumed.role.arn/session.name: `AssumedRoleCredentialProvider`'s constructor reads both
+ * via plain `Configuration#getTrimmed` -- [[PropagatedOptionConsumer]].
+ * - endpoint (`S3AFileSystem`: `getTrimmed`), endpoint.region (`DefaultS3ClientFactory`:
+ * `getTrimmed`), path.style.access (`S3AFileSystem`: `getBoolean`) --
+ * [[PropagatedOptionConsumer]].
+ * - requester.pays.enabled: not read anywhere in `hadoop-aws` 3.3.4 (the constant does not
+ * even exist in its `Constants` class); later releases read it via plain `getBoolean`
+ * against the propagated conf, so the plain tier is both the faithful forward model and
+ * inert on 3.3.4 -- [[PropagatedOptionConsumer]].
+ * - comet.credential.provider.class: Comet's own activation key, plain conf read on both
+ * sides, never a Hadoop key at all -- [[PropagatedOptionConsumer]].
+ *
+ * SYNC NOTE: the key list must stay a superset of native's `NATIVE_S3A_CONFIG_PROPERTIES`
+ * constant (`native/core/src/parquet/objectstore/s3.rs`, property suffixes without the
+ * `fs.s3a.` prefix) -- `DeltaScanContribSuite`'s discovery-harness test asserts this
+ * mechanically against [[AllS3ConfigKeys]]. Literal strings, not the
+ * [[AwsCredentialsProviderKey]] / [[AssumedRoleCredentialsProviderKey]] vals declared below,
+ * purely to avoid a forward reference inside this `object` body; kept textually identical to
+ * those two constants.
+ */
+ private[delta] val S3ConfigKeyConsumers: Seq[(String, S3ConfigConsumer)] = Seq(
+ "fs.s3a.access.key" -> LookupPasswordConsumer,
+ "fs.s3a.secret.key" -> LookupPasswordConsumer,
+ "fs.s3a.session.token" -> LookupPasswordConsumer,
+ "fs.s3a.aws.credentials.provider" -> PropagatedOptionConsumer,
+ "fs.s3a.assumed.role.arn" -> PropagatedOptionConsumer,
+ "fs.s3a.assumed.role.session.name" -> PropagatedOptionConsumer,
+ "fs.s3a.assumed.role.credentials.provider" -> PropagatedOptionConsumer,
+ "fs.s3a.endpoint" -> PropagatedOptionConsumer,
+ "fs.s3a.endpoint.region" -> PropagatedOptionConsumer,
+ "fs.s3a.path.style.access" -> PropagatedOptionConsumer,
+ "fs.s3a.requester.pays.enabled" -> PropagatedOptionConsumer,
+ "fs.s3a.comet.credential.provider.class" -> PropagatedOptionConsumer)
+
+ /** The compared keys alone, in [[S3ConfigKeyConsumers]] order (discovery-harness surface). */
+ private[delta] val AllS3ConfigKeys: Seq[String] = S3ConfigKeyConsumers.map(_._1)
+
+ /**
+ * The short-bucket-then-global value resolved for `baseKey` under `bucket` from `hadoopConf`,
+ * skipping an empty value at either alias exactly like [[plainValue]]. NOT used by
+ * [[s3ConfigDivergenceReason]]/[[s3KeyDivergenceReason]] any more -- every key checked there
+ * resolves per its declared [[S3ConfigKeyConsumers]] tier (see [[s3KeyDivergenceReason]]),
+ * neither of which matches this function's read. This function's one remaining caller is
+ * [[shortThenGlobalOrReason]], which reads provider-CLASS strings (from the ORIGINAL,
+ * unpropagated conf) for name-support validation in [[providerClassReason]]/
+ * [[assumedRoleProviderClassReason]] -- by the time those run, [[s3ConfigDivergenceReason]] has
+ * already proven Hadoop's and native's effective values agree for the same key, so whichever of
+ * the two (equal) values this narrower read returns does not affect correctness there. NEVER
+ * used to compute native's own effective value -- see [[nativeShortThenGlobal]] for that.
+ */
+ private def shortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket` from the ORIGINAL, unpropagated `hadoopConf` -- `NativeConfig
+ * .extractObjectStoreOptions` forwards `Configuration#get`'s substituted value for every
+ * `fs.s3a.*` entry with no bucket-option propagation step of its own, so the original conf is
+ * the right input here. Unlike [[shortThenGlobal]]/[[plainValue]], this mirrors `get_config`
+ * faithfully: PRESENCE of the short-bucket key alone -- never its emptiness -- decides whether
+ * native falls back to the global key (`get_config` is a plain `HashMap::get`, which returns
+ * `Some` for a key explicitly set to `""`), so an explicitly empty or whitespace-only
+ * short-bucket value resolves to `Some("")` here and never falls through to global -- the
+ * OPPOSITE of Hadoop's own `getPassword`/`lookupPassword` semantics (see
+ * [[hadoopLookupPasswordEffective]]), which treat empty as absent and keep trying the next
+ * alias. The ONLY function used to compute native's effective value in
+ * [[s3KeyDivergenceReason]].
+ *
+ * Deliberately does NOT apply `get_config_trimmed`'s `.trim()` here: [[s3KeyDivergenceReason]]
+ * trims both this value and Hadoop's effective value together, symmetrically, at the point they
+ * are compared, rather than one-sidedly here -- trimming only the native side would flag a
+ * spurious divergence for a value neither side's whitespace actually changes the behavior of
+ * once each side's own downstream parsing normalizes it (e.g. Hadoop's own multi-line
+ * `fs.s3a.aws.credentials.provider` default, which both Hadoop and native additionally trim per
+ * comma-separated entry after splitting), while a one-sided trim would make an
+ * otherwise-identical default value look diverged for every bucket, never claiming natively at
+ * all.
+ */
+ private def nativeShortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ Option(hadoopConf.get(shortKey)).orElse(Option(hadoopConf.get(baseKey)))
+ }
+
+ /**
+ * Faithful in-memory replica of `S3AUtils#propagateBucketOptions` (`hadoop-aws`), which
+ * `S3AFileSystem#initialize` calls FIRST, before any option or credential is read:
+ * `Configuration conf = propagateBucketOptions(originalConf, bucket); ...; setConf(conf);` --
+ * every subsequent `conf.get`/`getPassword` call in that filesystem instance, including
+ * `${...}` variable substitution, resolves against this propagated view, not the original conf.
+ * `hadoop-aws` is not on this module's runtime classpath (see the string-literal-keys note
+ * above), so `S3AUtils#propagateBucketOptions` cannot be called directly; this reproduces its
+ * logic verbatim using only `hadoop-common`'s `Configuration`:
+ * {{{
+ * public static Configuration propagateBucketOptions(Configuration source, String bucket) {
+ * final String bucketPrefix = FS_S3A_BUCKET_PREFIX + bucket + '.';
+ * final Configuration dest = new Configuration(source);
+ * for (Map.Entry<String, String> entry : source) {
+ * final String key = entry.getKey();
+ * final String value = entry.getValue(); // the (unexpanded) value
+ * if (!key.startsWith(bucketPrefix) || bucketPrefix.equals(key)) continue;
+ * final String stripped = key.substring(bucketPrefix.length());
+ * if (stripped.startsWith("bucket.") || "impl".equals(stripped)) {
+ * // ignored
+ * } else {
+ * final String generic = FS_S3A_PREFIX + stripped;
+ * dest.set(generic, value, ...); // overwrites any existing global value
+ * }
+ * }
+ * return dest;
+ * }
+ * }}}
+ * Note the LONG bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) folds to an unread
+ * `fs.s3a.fs.s3a.<key>` key here too, exactly like the real method -- `stripped` already starts
+ * with `fs.s3a.` in that case, so prepending `fs.s3a.` again produces a key nothing ever reads.
+ */
+ private def propagateBucketOptions(hadoopConf: Configuration, bucket: String): Configuration = {
+ val bucketPrefix = s"fs.s3a.bucket.$bucket."
+ val dest = new Configuration(hadoopConf)
+ hadoopConf.iterator().asScala.foreach { entry =>
+ val key = entry.getKey
+ if (key.startsWith(bucketPrefix) && key != bucketPrefix) {
+ val stripped = key.substring(bucketPrefix.length)
+ if (!stripped.startsWith("bucket.") && stripped != "impl") {
+ dest.set(s"fs.s3a.$stripped", entry.getValue)
+ }
+ }
+ }
+ dest
+ }
+
+ /**
+ * Canonical and deprecated Hadoop S3A encryption-algorithm config keys, verified via `javap`
+ * against `hadoop-aws` 3.3.4's `org.apache.hadoop.fs.s3a.Constants`: `S3_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.encryption.algorithm"` (canonical) and `SERVER_SIDE_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.server-side-encryption-algorithm"` (DEPRECATED -- note the hyphen before "algorithm",
+ * unlike the corresponding `*.key` constants below, which both use a `.key` suffix).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, same
+ * rationale as [[HadoopCredentialProviderPathKey]].
+ */
+ private val S3EncryptionAlgorithmKey = "fs.s3a.encryption.algorithm"
+ private val DeprecatedS3EncryptionAlgorithmKey = "fs.s3a.server-side-encryption-algorithm"
+
+ /**
+ * The exact strings `S3AEncryptionMethods#getMethod` accepts, verified via `javap`/CFR against
+ * `hadoop-aws` 3.3.4's `S3AEncryptionMethods` enum: `NONE("")`, `SSE_S3("AES256", serverSide =
+ * true, requiresSecret = false)`, `SSE_KMS("SSE-KMS", serverSide = true, requiresSecret =
+ * false)`, `SSE_C("SSE-C", serverSide = true, requiresSecret = true)`, `CSE_KMS("CSE-KMS",
+ * serverSide = false, requiresSecret = true)`, `CSE_CUSTOM("CSE-CUSTOM", serverSide = false,
+ * requiresSecret = true)`. `getMethod` parses case-insensitively
+ * (`values().find(_.getMethod.equalsIgnoreCase(algorithm))`), matched below the same way.
+ *
+ * ALLOWLIST, not a blocklist (replaces the former SSE-C-only blocklist): only the algorithms S3
+ * decrypts transparently on GET/HEAD given read permission alone, with NO extra request header
+ * and NO client-side step, are safe for a native scan that forwards none of Hadoop's
+ * `fs.s3a.encryption.*`/`fs.s3a.server-side-encryption*` options --
+ * - `AES256` (SSE_S3, `serverSide = true`): plain server-side encryption, transparent on GET.
+ * - `SSE-KMS` (SSE_KMS, `serverSide = true`): server-side, KMS-managed key, transparent on
+ * GET given KMS decrypt permission (no header).
+ * - `DSSE-KMS`: NOT present in this enum on `hadoop-aws` 3.3.4 (confirmed by the six values
+ * listed above) -- `S3AEncryptionMethods.getMethod("DSSE-KMS")` throws
+ * `IOException("Unknown encryption algorithm DSSE-KMS")` on this version, so
+ * `S3AUtils#buildEncryptionSecrets` (and therefore Hadoop's own reader) already fails
+ * before ever reading such a table under 3.3.4, meaning this string can never actually be
+ * the resolved value on the declared target version -- admitting it here is inert there.
+ * Included anyway, forward-compatible, for a newer `hadoop-aws` on the runtime classpath (a
+ * later Hadoop release; this module has no compile-time `hadoop-aws` dependency, see the
+ * string-literal-keys note above) where DSSE-KMS is a real, dual-layer, server-side
+ * algorithm decrypted transparently on GET the same way SSE-KMS is. Every other value
+ * declines: `SSE-C` (SSE_C is `serverSide = true` in Hadoop's own enum, but `requiresSecret
+ * \= true` -- S3 rejects a GET/HEAD for an SSE-C object outright (400 Bad Request) unless
+ * the customer key is resent as a request header on every call, so a native scan that never
+ * learns the key cannot succeed at all, where Hadoop's own reader -- whose request factory
+ * attaches the key -- would), `CSE-KMS`/`CSE-CUSTOM` (`serverSide = false`: client-side
+ * encryption decrypts object bytes locally in the SDK layer, which the native Parquet
+ * reader has no equivalent of -- it would read raw ciphertext), and any future/unknown
+ * value (a value `S3AEncryptionMethods.getMethod` itself would reject is certainly not one
+ * of the three confirmed-transparent algorithms above; declining is the only safe default
+ * for anything this gate cannot positively confirm).
+ */
+ private val AllowedEncryptionAlgorithms: Set[String] = Set("AES256", "SSE-KMS", "DSSE-KMS")
+
+ /**
+ * `bucket`'s effective encryption-algorithm key and value under `hadoopConf`, or `None` when
+ * neither the canonical nor deprecated key is set anywhere consulted. Mirrors
+ * `S3AUtils#buildEncryptionSecrets`'s real resolution order, verified via `javap`/CFR
+ * decompilation of `hadoop-aws` 3.3.4's `S3AUtils.class`:
+ * {{{
+ * String algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.server-side-encryption-algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.server-side-encryption-algorithm");
+ * }}}
+ * i.e. bucket-tier (canonical, then deprecated), THEN global-tier (canonical, then deprecated)
+ * -- the two tiers are never interleaved key-by-key, so this must stay two explicit bucket-tier
+ * lookups followed by two explicit global-tier lookups, not a single
+ * [[hadoopLookupPasswordEffective]] call per key (which would let an unset canonical bucket key
+ * fall through straight to the canonical GLOBAL value ahead of a SET deprecated bucket key, the
+ * wrong answer).
+ *
+ * THE FIX for the SSE-C long-bucket-alias gap is entirely inside the bucket tier:
+ * `lookupBucketSecret` itself is long-then-short, decompiled from `hadoop-aws` 3.3.4's
+ * `S3AUtils.class`:
+ * {{{
+ * // longBucketKey = fs.s3a.bucket.B.fs.s3a.<key>
+ * String longBucketKey = String.format(BUCKET_PATTERN, bucket, baseKey);
+ * String initialVal = getPassword(conf, longBucketKey, null, null);
+ * // shortBucketKey = fs.s3a.bucket.B.<key>
+ * String shortBucketKey = String.format(BUCKET_PATTERN, bucket, subkey);
+ * // keeps initialVal (the LONG value) if non-empty
+ * return getPassword(conf, shortBucketKey, initialVal, null);
+ * }}}
+ * i.e. the SAME long-bucket-key construction and long-wins-if-nonempty semantics as
+ * `S3AUtils#lookupPassword` (see [[LookupPasswordConsumer]]/[[hadoopLookupPasswordEffective]])
+ * -- the encryption algorithm is NOT one of the keys that flows through
+ * `S3AUtils#propagateBucketOptions` (which folds an unrelated per-bucket LONG form into an
+ * unread key). An earlier version of this function modeled the bucket tier as SHORT-only,
+ * documented as "the LONG bucket form is genuinely never consulted for this key" -- that
+ * documentation was wrong (this decompilation supersedes it): a bucket configured only via
+ * `fs.s3a.bucket.B.fs.s3a.encryption.algorithm=SSE-C` bypassed the SSE-C gate entirely, because
+ * Hadoop's own reader DOES read that long form (and picks SSE-C), while this function reported
+ * `None` (nothing set) and the allowlist check below never even ran.
+ *
+ * The canonical-vs-deprecated distinction below is frequently moot in practice: `hadoop-aws`'s
+ * `S3AFileSystem.addDeprecatedKeys()` statically registers `fs.s3a.server-side-encryption-*` as
+ * `Configuration`-level deprecated aliases of `fs.s3a.encryption.*` (verified via `javap`), a
+ * registration that lives in a static field on Hadoop's `Configuration` class -- process-wide
+ * once `S3AFileSystem`'s class has loaded anywhere in the JVM, which a real scan has always
+ * already done by the time this gate runs, since reading the S3 table at all requires loading
+ * that class. Once active, `Configuration#get` resolves either literal key to the identical
+ * value transparently, making the two-key cascade below redundant (but harmless) for that case;
+ * it remains the operative path only when nothing else in the process has loaded
+ * `S3AFileSystem` yet.
+ *
+ * ALSO walks the Hadoop-credential-provider (JCEKS) path via [[resolveViaCredentialAliases]]
+ * for each of the four lookups below, matching `lookupBucketSecret`/`lookupPassword`'s real
+ * per-alias `getPassword` calls (quoted above) exactly: both are `getPassword`, not plain
+ * `Configuration#get`, so a bucket storing the algorithm name ONLY in a JCEKS keystore is
+ * exactly as real a Hadoop deployment shape for this key as it is for the credential keys
+ * [[hadoopLookupPasswordEffective]] already covers -- there is nothing algorithm-specific that
+ * makes JCEKS storage implausible here, so an earlier version of this function skipping it
+ * (documented at the time as "the algorithm NAME is not credential-sensitive data, so storing
+ * it in a keystore is not a realistic Hadoop deployment pattern") was an unjustified, narrower
+ * read than Hadoop's own resolver actually performs, under-declining a bucket whose algorithm
+ * is keystore-only. [[resolveViaCredentialAliases]]'s Arm B/C split still means this is zero
+ * extra I/O for the common case: keystore I/O only happens when a Hadoop credential-provider
+ * path is actually configured for the bucket, contained in that function's own try/catch.
+ * `bucketTier`/`globalTier` return `Left` (propagated straight through by [[orElseTier]]) when
+ * [[resolveViaCredentialAliases]] cannot safely verify a tier at all (an S3A-scoped provider
+ * path, or a corrupt/unreadable global keystore) -- correctly short-circuiting the whole
+ * cascade with a decline rather than silently falling through to a later tier that might look
+ * unset only because the true value was unverifiable.
+ */
+ private def effectiveEncryptionAlgorithm(
+ hadoopConf: Configuration,
+ bucket: String): Either[String, Option[(String, String)]] = {
+ def bucketTier(baseKey: String): Either[String, Option[(String, String)]] = {
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ resolveViaCredentialAliases(
+ hadoopConf,
+ bucket,
+ Seq(longKey, shortKey),
+ honorClearTextFallback = true)
+ .map(_.map(baseKey -> _))
+ }
+ def globalTier(baseKey: String): Either[String, Option[(String, String)]] =
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(baseKey), honorClearTextFallback = true)
+ .map(_.map(baseKey -> _))
+
+ // Short-circuits on Left (unverifiable tier) or Right(Some(_)) (resolved); only Right(None)
+ // (tier definitively unset) falls through to `next`, mirroring buildEncryptionSecrets's
+ // sequential `if (algorithm == null) algorithm = ...` cascade exactly.
+ def orElseTier(
+ current: Either[String, Option[(String, String)]],
+ next: => Either[String, Option[(String, String)]])
+ : Either[String, Option[(String, String)]] =
+ current match {
+ case Left(reason) => Left(reason)
+ case Right(Some(value)) => Right(Some(value))
+ case Right(None) => next
+ }
+
+ orElseTier(
+ bucketTier(S3EncryptionAlgorithmKey),
+ orElseTier(
+ bucketTier(DeprecatedS3EncryptionAlgorithmKey),
+ orElseTier(
+ globalTier(S3EncryptionAlgorithmKey),
+ globalTier(DeprecatedS3EncryptionAlgorithmKey))))
+ }
+
+ private def unsupportedEncryptionAlgorithmDeclineReason(
+ bucket: String,
+ algorithmKey: String,
+ algorithm: String): String =
+ s"Native Delta scan does not support $algorithmKey=$algorithm for $bucket " +
+ "(the native S3 client only supports unencrypted objects and S3's transparent " +
+ "server-side algorithms -- AES256/SSE-S3, SSE-KMS, and DSSE-KMS decrypt on GET/HEAD given " +
+ "read permission alone, with no extra request header; SSE-C additionally requires the " +
+ "customer-provided key resent as a header on every GET/HEAD request, which the native S3 " +
+ "client's extract_s3_config_options never forwards, and CSE-KMS/CSE-CUSTOM decrypt object " +
+ "bytes client-side, a layer the native Parquet reader does not have -- any of these would " +
+ "fail outright or silently read ciphertext where Hadoop's own reader succeeds)"
+
+ /**
+ * First reason any bucket among `uris` is configured for an encryption algorithm the native S3
+ * client cannot safely read, or `None` when claimable. Allowlist-based (see
+ * [[AllowedEncryptionAlgorithms]]): only `AES256`/`SSE-KMS`/`DSSE-KMS` (and unset/empty) pass;
+ * every other resolved value -- `SSE-C`, `CSE-KMS`, `CSE-CUSTOM`, or any unrecognized future
+ * algorithm string -- declines. Deliberately NOT a blocklist keyed on `SSE-C` alone: an
+ * allowlist is safe by construction against a Hadoop release adding a new encryption method
+ * this gate has never heard of, where a blocklist would silently admit it. Never interpolates a
+ * resolved key value, only key names, the bucket, and the (non-secret) algorithm name. Declines
+ * on a `Left` from [[effectiveEncryptionAlgorithm]] too (an unverifiable credential-provider
+ * arm, e.g. an S3A-scoped provider path or a corrupt/unreadable global keystore) -- the
+ * algorithm cannot be ruled safe when it cannot be read at all.
+ */
+ private[delta] def unsupportedEncryptionAlgorithmReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ effectiveEncryptionAlgorithm(hadoopConf, bucket) match {
+ case Left(reason) => Some(reason)
+ case Right(None) => None
+ case Right(Some((key, value))) =>
+ if (!AllowedEncryptionAlgorithms.exists(_.equalsIgnoreCase(value))) {
+ Some(unsupportedEncryptionAlgorithmDeclineReason(bucket, key, value))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Canonical Hadoop S3A HTTP-proxy host config key, verified via CFR decompilation of
+ * `hadoop-aws` 3.3.4's `S3AUtils.class` (`initProxySupport`):
+ * {{{
+ * String proxyHost = conf.getTrimmed("fs.s3a.proxy.host", "");
+ * int proxyPort = conf.getInt("fs.s3a.proxy.port", -1);
+ * if (!proxyHost.isEmpty()) {
+ * ...
+ * String proxyUsername =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.username", null, null);
+ * String proxyPassword =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.password", null, null);
+ * ...
+ * }
+ * }}}
+ * `fs.s3a.proxy.host`/`fs.s3a.proxy.port` resolve via a PLAIN, non-bucket-scoped, non-JCEKS
+ * `Configuration#getTrimmed`/`getInt` call -- NOT `lookupPassword` -- against whatever conf
+ * `S3AFileSystem#initialize` already ran through `propagateBucketOptions` before
+ * `createAwsConf`/`initProxySupport` ever runs; only the SIBLING `fs.s3a.proxy.username`/
+ * `fs.s3a.proxy.password` keys go through `lookupPassword` (bucket long/short/global,
+ * JCEKS-aware). So the host is bucket-aware only via `propagateBucketOptions`'s short-bucket-
+ * form folding, never the long-bucket form, and never a credential-provider read -- the SAME
+ * shape as `endpoint`/`path.style.access` ([[PropagatedOptionConsumer]], see
+ * [[S3ConfigKeyConsumers]]'s doc), not the credential family.
+ *
+ * Resolved here through [[hadoopLookupPasswordEffective]]'s same long-then-short-then-global,
+ * credential-provider-aware cascade anyway (via [[resolveViaCredentialAliases]], with
+ * `honorClearTextFallback = false` since the host's real plain `getTrimmed` never checks
+ * [[ClearTextFallbackKey]] -- a plaintext host must stay visible to this gate even with the
+ * flag off): checking MORE than Hadoop's real, narrower resolution can only ever find a value
+ * Hadoop itself ignores, which for a PRESENCE gate can only turn an already-necessary decline
+ * into one found slightly more eagerly -- never a missed one. A missed decline is exactly the
+ * failure mode this gate exists to close (native has NO HTTP-proxy support of any kind -- no
+ * `fs.s3a.proxy.*` key is read anywhere in `s3.rs`), so over-approximating is the conservative
+ * direction here, unlike a value-EQUALITY comparator (e.g. [[s3KeyDivergenceReason]]) where a
+ * wider read than the key's real consumer can produce a false equality as easily as a false
+ * divergence -- which is exactly why the comparator classifies per key while this gate does not
+ * need to.
+ */
+ private val S3ProxyHostKey = "fs.s3a.proxy.host"
+
+ private def unsupportedProxyReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key configured for $bucket (the native S3 client has " +
+ "no HTTP proxy support at all -- no fs.s3a.proxy.* key is read anywhere in its object " +
+ "store layer -- so a claimed scan would connect to S3 directly instead of routing through " +
+ "the configured proxy, either bypassing an egress/network-segmentation policy or simply " +
+ "failing to reach the endpoint)"
+
+ /**
+ * First reason any bucket among `uris` has an HTTP proxy configured via [[S3ProxyHostKey]], or
+ * `None` when claimable. Ordered alongside [[unsupportedEncryptionAlgorithmReason]] among the
+ * other conf-only gates that are zero-I/O in the common case, ahead of
+ * [[s3ConfigDivergenceReason]]: [[hadoopLookupPasswordEffective]] only performs real keystore
+ * I/O, inside its own try/catch, when a Hadoop credential-provider path is actually configured
+ * for this bucket (see that function's doc for the Arm A/B/C dispatch). Never interpolates a
+ * resolved value: the proxy HOST is not secret, but naming it here would be a strange place to
+ * first surface it, and proxy CREDENTIALS (`fs.s3a.proxy.username`/`fs.s3a.proxy.password`, not
+ * read by this gate at all -- the whole point of gating on the host is that a non-empty host
+ * declines before any proxy credential would ever need to be forwarded) must never appear in a
+ * decline reason regardless.
+ */
+ private[delta] def proxyGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI],
+ propagatedConfCache: MutableMap[String, Configuration] = MutableMap.empty)
+ : Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ try {
+ val propagatedConf =
+ propagatedConfCache.getOrElseUpdate(
+ bucket,
+ propagateBucketOptions(hadoopConf, bucket))
+ resolveViaCredentialAliases(
+ propagatedConf,
+ bucket,
+ longThenShortThenGlobalAliases(bucket, S3ProxyHostKey),
+ honorClearTextFallback = false) match {
Review Comment:
[P2] Read the proxy host through its plain configuration consumer
`honorClearTextFallback = false` only affects the `NoProvider` branch. With a readable global `hadoop.security.credential.provider.path`, `hadoop.security.credential.clear-text-fallback=false`, and a plaintext `fs.s3a.proxy.host`, `GlobalProviderOnly` still calls `getPassword`. If the store has no proxy aliases, this returns `Right(None)`. Hadoop 3.3.4's `S3AUtils.initProxySupport` instead reads the host with `getTrimmed` and uses the configured proxy.
An otherwise supported scan using environment credentials and no configured access/secret/token values can pass the remaining gates, while native never forwards Hadoop's proxy options. Without a separately configured environment proxy, a deployment requiring that proxy loses access when the scan is claimed. Could the gate check the propagated host with its plain consumer independently of credential-store lookup? This counterexample is source-derived. I have not run an end-to-end reproduction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3859501548
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so
+ * `CometPlanAdaptiveDynamicPruningFilters`'s rewrite (via [[CometScanWithPlanData]]) survives
+ * plan copies, the lesson from CometIcebergNativeScanExec (a transient field is dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node, our
+ // own withDynamicPruningFilters, or a generic Catalyst expression rewrite going through
+ // TreeNode.makeCopy, automatically gets a helper consistent with ITS runtimeFilters. A
+ // stored helper field would desync from rewritten filters (the #3510 class of bug). The
+ // cost is that file listing runs once per executed instance (planning listed separately in
+ // the rule extension); correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below (via
+ // `scanHelper.metrics`), and AQE's UI plan-walk calls `.metrics` on every node MID-PLANNING --
+ // including while a DPP subquery is still an adaptive placeholder or a partition filter holds
+ // an unresolved ScalarSubquery (see `hasUnevaluableSubqueryFilter` just below). Constructing
+ // `scanHelper` here, and reading `CometScanExec.metrics` off it, is safe ONLY because that
+ // construction is a cheap case-class build with no file listing, and core's
+ // `CometScanExec.metrics` (spark/.../CometScanExec.scala) touches only `wrapped.driverMetrics`
+ // -- populated by Spark's own planning, not by this scan -- plus a static metric-node
+ // constructor. Neither does file listing (`selectedPartitions`/`getFilePartitions`) or
+ // subquery resolution. If core's `metrics` is ever changed to touch either, forcing
+ // `scanHelper` from this `metrics` override would resurrect the two AQE mid-planning crashes
+ // this invariant was written to prevent.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element (planningHelper passes
+ // partitionFilters = runtimeFilters into CometScanExec's plain constructor field below), so
+ // checking runtimeFilters here avoids constructing/forcing the derived scanHelper just to
+ // read partitioning. The InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep these two
+ // sets in sync; if that rule learns to unwrap a new wrapper form, mirror it here too. The
+ // ScalarSubquery case (e.g. `p = (SELECT max(p) FROM dim ...)`, a partition-column filter,
+ // so it lands in runtimeFilters rather than dataFilters) is presence-based, unlike the
+ // InSubqueryExec check above: Spark exposes no public finished/updated probe on
+ // ExecSubqueryExpression, so any partition filter containing one forces UnknownPartitioning(0)
+ // even once it has actually resolved. That is safe one-directionally only -- worst case an
+ // extra shuffle from an overly conservative partitioning, never a wrong answer, since
+ // execution reads perPartitionData directly and never goes through this getter.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case _: ScalarSubquery => true
Review Comment:
**[P2] Restore the execution partition count after scalar-subquery resolution**
Could we preserve the planning-time guard while exposing the actual partition count during execution? For a partition filter such as `p = (SELECT max(p) FROM thresholds)`, the expression remains a `ScalarSubquery` after its result is available, so this branch keeps `outputPartitioning` at zero.
A fused native parent reads that zero in [buildNativeContext](https://github.com/apache/datafusion-comet/blob/1d3557cf7eef766056d4c082f8be89355118bf53/spark/src/main/scala/org/apache/spark/sql/comet/operators.scala#L784-L795), and [NativeExecContext's validation](https://github.com/apache/datafusion-comet/blob/1d3557cf7eef766056d4c082f8be89355118bf53/spark/src/main/scala/org/apache/spark/sql/comet/operators.scala#L550-L556) rejects nonempty scan partition data with `All per-partition arrays must have length 0`. This also affects contexts without broadcast inputs.
A separate Spark/Delta probe confirmed that the scalar expression remains in the partition filter after `collect()`, with an evaluable result and matching rows. The native-context failure is source-traced, not a full current-head JNI reproduction. Please add a regression that requires the native Delta scan beneath a native parent; the existing scalar-filter test permits fallback.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,539 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = toPhysical(scanExec, relation.partitionSchema),
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme.
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
+
+ val common = commonBuilder.build()
+ val deltaCommon = OperatorOuterClass.DeltaSparkScanCommon
+ .newBuilder()
+ .setTableRoot(tableRoot)
+ .setColumnMappingMode(columnMappingMode(scanExec))
+ .setSourceKey(DeltaPlanDataInjector.sourceKey(tableRoot, common))
+ .build()
+ val deltaScan = OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaCommon)
+ Operator
+ .newBuilder()
+ .setPlanId(scanExec.id)
+ .setContribScan(DeltaSparkScanEnvelope.pack(deltaScan.build()))
+ .build()
+ }
+ }
+
+ /**
+ * One representative store URI per distinct object-store authority this scan's partitions may
+ * need options for: the data-file authority (`firstFileUri`), the table root unconditionally
+ * (UUID-relative DV sidecars resolve against it, and it is cheap to include even when absent),
+ * and every distinct on-disk DV authority from `descriptors`. Inline DVs are filtered out --
+ * they carry no external URI, only embedded bytes. Deduping by authority (rather than by full
+ * URI) keeps this O(distinct authorities) instead of O(files): a table with N deletion-vector
+ * files on the same external store previously produced ~N distinct URIs here, each
+ * independently fed into `mergedObjectStoreOptions`'s `extractObjectStoreOptions` walk over
+ * `hadoopConf`. Candidates are deduped keeping the FIRST URI seen per authority, so callers can
+ * rely on `firstFileUri`/the table root winning over any DV path that happens to share their
+ * authority. Factored out of [[convert]] so the URI-assembly logic (in particular the
+ * `storageType` filter and `absolutePath` resolution) is directly unit-testable with hand-built
+ * [[DeletionVectorDescriptor]] fixtures, without a Spark session or real selected files (a
+ * `file://` scan alone can't exercise a foreign-authority DV).
+ */
+ private[delta] def storeUris(
+ descriptors: Seq[DeletionVectorDescriptor],
+ tableRootPath: Path,
+ firstFileUri: Option[java.net.URI]): Seq[java.net.URI] = {
+ val dvAuthorityUris = descriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(tableRootPath).toUri)
+ val candidates = firstFileUri.toSeq ++ Seq(tableRootPath.toUri) ++ dvAuthorityUris
+ val byAuthority = scala.collection.mutable.LinkedHashMap.empty[String, java.net.URI]
+ candidates.foreach(uri =>
+ byAuthority.getOrElseUpdate(DeltaScanSupport.uriAuthority(uri), uri))
+ byAuthority.values.toSeq
+ }
+
+ /**
+ * Unions `NativeConfig.extractObjectStoreOptions` over every `uris` authority. Safe to simply
+ * union rather than pick one: the extracted keys are scheme-disjoint prefixes (`fs.s3a.*` vs
+ * `fs.azure.*`, ...), so options for different schemes never collide, and re-extracting the
+ * same scheme from two URIs is idempotent. Factored out of [[convert]] so it is directly
+ * unit-testable without a Spark session.
+ */
+ private[delta] def mergedObjectStoreOptions(
+ hadoopConf: org.apache.hadoop.conf.Configuration,
+ uris: Seq[java.net.URI]): Map[String, String] =
+ uris.foldLeft(Map.empty[String, String]) { (merged, uri) =>
+ merged ++ NativeConfig.extractObjectStoreOptions(hadoopConf, uri)
+ }
+
+ /**
+ * Harvest subquery-bearing predicates for this scan from its covering FilterExec. Spark 3.x
+ * strips them from a scan's `dataFilters` at planning (`FileSourceStrategy` routes them to the
+ * post-scan filter only), while Spark 4.x keeps them in `dataFilters`. Collecting them here at
+ * claim time gives the execution-time resolve-and-push path the same inputs on every Spark
+ * version; the dedup keeps Spark 4.x from carrying duplicates.
+ *
+ * Safety comes from the plan walk, not just the reference guard: a filter is only harvested
+ * when every operator between it and the scan commutes with pushing the predicate into the scan
+ * (see `spineToScan`). Reference containment alone proves the predicate is expressible over the
+ * scan's output, not that moving it there is semantics-preserving -- an intervening LIMIT/TopN
+ * (or Sort, Aggregate, Window, join, ...) can change which rows the predicate would have
+ * applied to, so those stop the walk and the filter is left where Spark placed it.
+ */
+ def subqueryFiltersFromParent(
+ plan: org.apache.spark.sql.execution.SparkPlan,
+ scanExec: FileSourceScanExec): Seq[org.apache.spark.sql.catalyst.expressions.Expression] = {
+ import org.apache.spark.sql.catalyst.expressions.{PlanExpression, SubqueryExpression}
+ import org.apache.spark.sql.execution.{FilterExec, ProjectExec, SparkPlan}
+
+ // Whether every node on the path from `node` down to `scanExec` is one that pushdown can
+ // safely cross: a deterministic ProjectExec is 1:1 on rows (an Alias mints a new exprId, so
+ // it cannot alias over the scan's own output attributes, which is what the reference guard
+ // below requires) and a deterministic FilterExec only removes rows, so moving a predicate
+ // expressed over the scan's output through either preserves the query's semantics. A
+ // nondeterministic projection breaks that: a deterministic conjunct does not commute with it,
+ // because pushing the predicate into the scan changes which rows survive to have
+ // nondeterministic expressions (e.g. monotonically_increasing_id()) evaluated over them,
+ // changing the result -- so both guards require `.deterministic`, mirroring Spark's own
+ // PushPredicateThroughNonJoin/CollapseProject rules. Anything else (LIMIT/TopN, Sort,
+ // Aggregate, Window, joins, Union, Sample, ...) can reorder or drop rows in ways that make
+ // "push the predicate down to the scan" change the result, so an unrecognized node stops the
+ // walk: the filter is left uncollected (missed pruning only, never a correctness issue).
+ def spineToScan(node: SparkPlan): Boolean = node match {
+ case n if n eq scanExec => true
+ case p: ProjectExec if p.projectList.forall(_.deterministic) => spineToScan(p.child)
+ case f: FilterExec if f.condition.deterministic => spineToScan(f.child)
+ case _ => false
+ }
+
+ // Nearest FilterExec whose spine down to the scan is Project/Filter-only (the DV shape
+ // interposes such nodes between them, so do not require a direct parent-child edge).
+ val filtersAboveScan = plan.collect {
+ case f: FilterExec if spineToScan(f.child) => f
+ }
+ filtersAboveScan.lastOption
+ .map { f =>
+ splitConjunctivePredicates(f.condition)
+ .filter(_.deterministic)
+ .filter(_.references.subsetOf(scanExec.outputSet))
+ .filter(p =>
+ SubqueryExpression.hasSubquery(p) || p.exists(_.isInstanceOf[PlanExpression[_]]))
+ .filterNot(p => scanExec.dataFilters.exists(_.semanticEquals(p)))
+ }
+ .getOrElse(Seq.empty)
+ }
+
+ /**
+ * Resolve scalar-subquery data filters at execution time and serialize them for native
+ * pushdown, mirroring `CometNativeScanExec.serializedPartitionData`. `supportedDataFilters`
+ * excludes PlanExpressions at planning time (subquery results do not exist yet), so these
+ * bounds reach the native reader only through this path. Filters that fail to serialize are
+ * skipped: Spark keeps a covering FilterExec above the scan, so this is missed pruning only,
+ * never a correctness issue.
+ *
+ * Known core-parity limitation: when the scan is fused under a parent native operator,
+ * `ensureSubqueriesResolved` has already called `updateResult()` on these subqueries and this
+ * path calls it again (Spark's ScalarSubquery.updateResult re-executes unconditionally). Core's
+ * CometNativeScanExec has the identical double-execution; benign for Delta (the subquery's
+ * snapshot is pinned at analysis), wasteful for expensive subqueries. Fix belongs in core.
+ */
+ def resolvedSubqueryFilters(
+ dataFilters: Seq[org.apache.spark.sql.catalyst.expressions.Expression],
+ output: Seq[org.apache.spark.sql.catalyst.expressions.Attribute],
+ requiredSchema: StructType,
+ conf: org.apache.spark.sql.internal.SQLConf)
+ : Seq[org.apache.comet.serde.ExprOuterClass.Expr] = {
+ if (!conf.getConf(org.apache.spark.sql.internal.SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ return Seq.empty
+ }
+ val subqueryFilters = dataFilters.filter(_.exists(_.isInstanceOf[ExecScalarSubquery]))
+ if (subqueryFilters.isEmpty) {
+ return Seq.empty
+ }
+ // Same binding guard as the DV shape's plan-time filters: references limited to the
+ // data-column prefix of the output, where positions agree between the output and the
+ // native read schema. For the plain shape strippedLen is the full required schema.
+ // Guard BEFORE updateResult so discarded filters never execute their subqueries.
+ val strippedLen = requiredSchema.count(f => !internalColumnNames.contains(f.name))
+ val dataColIds = output.take(strippedLen).map(_.exprId).toSet
+ val pushableFilters =
+ subqueryFilters.filter(_.references.forall(r => dataColIds.contains(r.exprId)))
+ pushableFilters.foreach(_.foreach {
+ case s: ExecScalarSubquery => s.updateResult()
+ case _ =>
+ })
+ pushableFilters
+ .flatMap { filter =>
+ // MergeScalarSubqueries can fuse several scalar subqueries into one struct-returning
+ // subquery accessed via GetStructField; fold that whole subtree to a literal (a bare
+ // GetStructField-over-Literal would not serialize).
+ val resolved = filter.transform {
+ case g @ org.apache.spark.sql.catalyst.expressions
+ .GetStructField(_: ExecScalarSubquery, _, _) =>
+ Literal.create(g.eval(null), g.dataType)
+ case s: ExecScalarSubquery =>
+ Literal.create(s.eval(null), s.dataType)
+ }
+ val proto = exprToProto(resolved, output)
+ if (proto.isEmpty) {
+ logWarning(s"Could not serialize resolved scalar subquery filter: $resolved")
+ }
+ proto
+ }
+ }
+
+ /**
+ * DV shape common builder. Layout invariants (declined by DeltaScanSupport when violated): scan
+ * output = requiredSchema attrs (data columns, then the internal columns as a suffix) followed
+ * by partition and constant-metadata columns. The parquet read schema strips the internal
+ * columns; they are appended to the partition schema as per-file constants, so the projection
+ * vector routes them from the constants block.
+ */
+ private def buildDvScanCommon(
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ firstFileUri: Option[java.net.URI],
+ hadoopConf: org.apache.hadoop.conf.Configuration)
+ : Option[OperatorOuterClass.NativeScanCommon.Builder] = {
+ val relation = scanExec.relation
+ val output = scanExec.output
+ val requiredSchema = scanExec.requiredSchema
+
+ val commonBuilder = OperatorOuterClass.NativeScanCommon.newBuilder()
+ commonBuilder.setSource(scanExec.simpleStringWithNodeId())
+
+ val scanTypes = output.flatMap(attr => serializeDataType(attr.dataType))
+ if (scanTypes.length != output.length) {
+ return None
+ }
+ commonBuilder.addAllFields(scanTypes.asJava)
+
+ val strippedRequired =
+ StructType(requiredSchema.filterNot(f => internalColumnNames.contains(f.name)))
+ val strippedLen = strippedRequired.length
+ val requiredLen = requiredSchema.length
+
+ // Keep only data filters that bind identically in the output and the native
+ // (strippedRequired ++ partitionFields) index spaces: references limited to the first
+ // strippedLen output attributes. Internal-column filters (is_row_deleted = 0) are
+ // trivially true after native DV application, and Spark's Filter above the scan
+ // re-evaluates everything anyway.
+ if (scanExec.conf.getConf(
+ org.apache.spark.sql.internal.SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ val dataColIds = output.take(strippedLen).map(_.exprId).toSet
+ val filterProtos = scanHelper.supportedDataFilters
+ .filter(_.references.forall(r => dataColIds.contains(r.exprId)))
+ .flatMap(f => exprToProto(f, output))
+ commonBuilder.addAllDataFilters(filterProtos.asJava)
+ }
+
+ // Constant metadata columns and real partition columns follow the required schema in the
+ // output, exactly like the plain shape.
+ val constantMetadataFields = scanExec.fileConstantMetadataColumns.map(attr =>
+ StructField(
+ s"${CometNativeScan.constantMetadataFieldPrefix}${attr.name}",
+ attr.dataType,
+ attr.nullable))
+ val internalFields = requiredSchema.fields.toSeq
+ .filter(f => internalColumnNames.contains(f.name))
+ .map(f => StructField(s"$deltaConstFieldPrefix${f.name}", f.dataType, f.nullable))
+ // The real partition columns must carry physical names in the proto, same as the data/
+ // required schemas: a retained physical data name can otherwise collide with a partition
+ // column's LOGICAL name after a rename history (e.g. a->b then p->a), and DataFusion's
+ // name-based partition rewrite would then replace the data projection with the partition
+ // constant. constantMetadataFields/internalFields are synthetic slots the
+ // native side allocates for this scan, not table columns, so they are not physicalized.
+ val partitionSchemaFields = toPhysical(scanExec, relation.partitionSchema).fields.toSeq ++
+ constantMetadataFields ++ internalFields
+
+ // Protos carry physical names (column mapping); index math below stays logical.
+ val partitionSchemaProto = schema2Proto(partitionSchemaFields)
+ val requiredSchemaProto = schema2Proto(toPhysical(scanExec, strippedRequired))
+ val dataSchemaProto = schema2Proto(toPhysical(scanExec, relation.dataSchema))
+
+ // Projection: data columns from the (stripped) read schema; internal columns from their
+ // constants slots at the END of the partition fields; the output tail (real partitions +
+ // constant metadata) positionally from the head of the partition fields.
+ val dataSchema = relation.dataSchema
+ val internalBase = dataSchema.length + partitionSchemaFields.length - internalFields.length
+ val internalIndexByName = requiredSchema.fields.toSeq
+ .filter(f => internalColumnNames.contains(f.name))
+ .zipWithIndex
+ .map { case (f, i) => f.name -> (internalBase + i) }
+ .toMap
+ val projectionVector = output.zipWithIndex.map { case (attr, i) =>
+ val idx = if (internalColumnNames.contains(attr.name)) {
+ internalIndexByName(attr.name)
+ } else if (i < requiredLen) {
+ dataSchema.fieldIndex(attr.name)
+ } else {
+ dataSchema.length + (i - requiredLen)
+ }
+ idx.toLong.asInstanceOf[java.lang.Long]
+ }
+ commonBuilder.addAllProjectionVector(projectionVector.asJava)
+
+ commonBuilder.addAllDataSchema(dataSchemaProto.asJava)
+ commonBuilder.addAllRequiredSchema(requiredSchemaProto.asJava)
+ commonBuilder.addAllPartitionSchema(partitionSchemaProto.asJava)
+
+ CometNativeScan.populateScanConfFlags(
+ commonBuilder,
+ strippedRequired,
+ firstFileUri,
+ hadoopConf,
+ scanExec.conf)
+
+ Some(commonBuilder)
+ }
+
+ /** Serialize one file partition into a DeltaSparkScan proto with per-file DV descriptors. */
+ def serializePartition(
+ filePartition: FilePartition,
+ scanExec: FileSourceScanExec,
+ tableRoot: String): Array[Byte] = {
+ val relation = scanExec.relation
+ val sparkPartition = partition2Proto(
+ filePartition,
+ relation.partitionSchema,
+ scanExec.fileConstantMetadataColumns,
+ ShimFileFormat.fileConstantMetadataExtractors(relation.fileFormat))
+
+ val dvShape = isDvShape(scanExec)
+
+ val deltaPartition = OperatorOuterClass.DeltaSparkFilePartition.newBuilder()
+ sparkPartition.getPartitionedFileList.asScala.zip(filePartition.files.toSeq).foreach {
+ case (fileProto, file) =>
+ val fileBuilder = fileProto.toBuilder
+ if (dvShape) {
+ // Append the internal-constant values after the real partition/constant-metadata
+ // values, matching the order of the appended partition-schema fields.
+ scanExec.requiredSchema.fields
+ .filter(f => internalColumnNames.contains(f.name))
+ .foreach { f =>
+ val lit = f.dataType match {
+ case ByteType => Literal(0.toByte, ByteType)
+ case LongType => Literal(0L, LongType)
+ case other =>
+ // Fixed internal invariant (observed Delta 3.3 types); fail loudly on
+ // drift rather than emit a plausible-looking constant.
+ throw new IllegalStateException(
+ s"Unexpected type $other for Delta internal column ${f.name}")
+ }
+ fileBuilder.addPartitionValues(
+ literalToProto(lit, s"delta internal constant ${f.name}"))
+ }
+ }
+ val dfb = OperatorOuterClass.DeltaSparkPartitionedFile
+ .newBuilder()
+ .setFile(fileBuilder.build())
+ extractDvDescriptor(file, tableRoot).foreach(dfb.setDv)
+ deltaPartition.addPartitionedFile(dfb.build())
+ }
+
+ OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setFilePartition(deltaPartition.build())
+ .build()
+ .toByteArray
+ }
+
+ /**
+ * Pull the DV descriptor Delta attached to this file (base64 under
+ * `row_index_filter_id_encoded`), resolving UUID-relative paths to absolute URLs and
+ * Z85-decoding inline bitmaps here on the JVM where delta-spark's codecs live.
+ */
+ private def extractDvDescriptor(
+ file: PartitionedFile,
+ tableRoot: String): Option[OperatorOuterClass.DeltaSparkDvDescriptor] = {
+ val encoded = file.otherConstantMetadataColumnValues
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ val filterType = file.otherConstantMetadataColumnValues
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_TYPE)
+ encoded.map { enc =>
+ filterType match {
+ case Some(RowIndexFilterType.IF_CONTAINED) | None =>
+ case other =>
+ // DeltaScanSupport declines CDF reads, the only source of inverted filters;
+ // reaching here means a gate was bypassed -- fail loudly rather than corrupt.
+ throw new IllegalStateException(
+ s"Native Delta scan cannot apply row index filter type $other")
+ }
+ val desc = DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String])
+ val builder = OperatorOuterClass.DeltaSparkDvDescriptor
+ .newBuilder()
+ .setStorageType(desc.storageType)
+ .setSizeInBytes(desc.sizeInBytes)
+ .setCardinality(desc.cardinality)
+ if (desc.storageType == DeletionVectorDescriptor.INLINE_DV_MARKER) {
+ // The comet-spark jar relocates protobuf, so use the shaded ByteString.
+ builder.setInlineData(
+ org.apache.comet.shaded.protobuf.ByteString.copyFrom(desc.inlineData))
Review Comment:
**[P2] Support the unshaded classpath during reactor compilation**
Could we make the contrib's compile/shading arrangement work before core's package phase as well? A clean root `./mvnw -Pspark-4.0,delta compile` or `test` sees unshaded `spark/target/classes`. The `org.apache.comet.shaded.protobuf` namespace is created only by [core's package-phase shade execution](https://github.com/apache/datafusion-comet/blob/1d3557cf7eef766056d4c082f8be89355118bf53/spark/pom.xml#L567-L578), so this hard-coded reference cannot resolve in that reactor path. The separate core-install/contrib-test CI sequence masks it.
An isolated Scala compilation against Java types generated from the current protocol reproduced `object shaded is not a member of package org.apache.comet`; the unshaded ByteString control compiled. The full reactor attempt was blocked earlier in dependency resolution, so this is compiler/classpath evidence rather than a completed reactor reproduction.
Simply changing the import would invert the problem for packaged-core consumers; the module needs a consistent arrangement for both classpaths.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,539 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = toPhysical(scanExec, relation.partitionSchema),
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme.
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
+
+ val common = commonBuilder.build()
+ val deltaCommon = OperatorOuterClass.DeltaSparkScanCommon
+ .newBuilder()
+ .setTableRoot(tableRoot)
+ .setColumnMappingMode(columnMappingMode(scanExec))
+ .setSourceKey(DeltaPlanDataInjector.sourceKey(tableRoot, common))
+ .build()
+ val deltaScan = OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaCommon)
+ Operator
+ .newBuilder()
+ .setPlanId(scanExec.id)
+ .setContribScan(DeltaSparkScanEnvelope.pack(deltaScan.build()))
+ .build()
+ }
+ }
+
+ /**
+ * One representative store URI per distinct object-store authority this scan's partitions may
+ * need options for: the data-file authority (`firstFileUri`), the table root unconditionally
+ * (UUID-relative DV sidecars resolve against it, and it is cheap to include even when absent),
+ * and every distinct on-disk DV authority from `descriptors`. Inline DVs are filtered out --
+ * they carry no external URI, only embedded bytes. Deduping by authority (rather than by full
+ * URI) keeps this O(distinct authorities) instead of O(files): a table with N deletion-vector
+ * files on the same external store previously produced ~N distinct URIs here, each
+ * independently fed into `mergedObjectStoreOptions`'s `extractObjectStoreOptions` walk over
+ * `hadoopConf`. Candidates are deduped keeping the FIRST URI seen per authority, so callers can
+ * rely on `firstFileUri`/the table root winning over any DV path that happens to share their
+ * authority. Factored out of [[convert]] so the URI-assembly logic (in particular the
+ * `storageType` filter and `absolutePath` resolution) is directly unit-testable with hand-built
+ * [[DeletionVectorDescriptor]] fixtures, without a Spark session or real selected files (a
+ * `file://` scan alone can't exercise a foreign-authority DV).
+ */
+ private[delta] def storeUris(
+ descriptors: Seq[DeletionVectorDescriptor],
+ tableRootPath: Path,
+ firstFileUri: Option[java.net.URI]): Seq[java.net.URI] = {
+ val dvAuthorityUris = descriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(tableRootPath).toUri)
+ val candidates = firstFileUri.toSeq ++ Seq(tableRootPath.toUri) ++ dvAuthorityUris
+ val byAuthority = scala.collection.mutable.LinkedHashMap.empty[String, java.net.URI]
+ candidates.foreach(uri =>
+ byAuthority.getOrElseUpdate(DeltaScanSupport.uriAuthority(uri), uri))
+ byAuthority.values.toSeq
+ }
+
+ /**
+ * Unions `NativeConfig.extractObjectStoreOptions` over every `uris` authority. Safe to simply
+ * union rather than pick one: the extracted keys are scheme-disjoint prefixes (`fs.s3a.*` vs
+ * `fs.azure.*`, ...), so options for different schemes never collide, and re-extracting the
+ * same scheme from two URIs is idempotent. Factored out of [[convert]] so it is directly
+ * unit-testable without a Spark session.
+ */
+ private[delta] def mergedObjectStoreOptions(
+ hadoopConf: org.apache.hadoop.conf.Configuration,
+ uris: Seq[java.net.URI]): Map[String, String] =
+ uris.foldLeft(Map.empty[String, String]) { (merged, uri) =>
+ merged ++ NativeConfig.extractObjectStoreOptions(hadoopConf, uri)
+ }
+
+ /**
+ * Harvest subquery-bearing predicates for this scan from its covering FilterExec. Spark 3.x
+ * strips them from a scan's `dataFilters` at planning (`FileSourceStrategy` routes them to the
+ * post-scan filter only), while Spark 4.x keeps them in `dataFilters`. Collecting them here at
+ * claim time gives the execution-time resolve-and-push path the same inputs on every Spark
+ * version; the dedup keeps Spark 4.x from carrying duplicates.
+ *
+ * Safety comes from the plan walk, not just the reference guard: a filter is only harvested
+ * when every operator between it and the scan commutes with pushing the predicate into the scan
+ * (see `spineToScan`). Reference containment alone proves the predicate is expressible over the
+ * scan's output, not that moving it there is semantics-preserving -- an intervening LIMIT/TopN
+ * (or Sort, Aggregate, Window, join, ...) can change which rows the predicate would have
+ * applied to, so those stop the walk and the filter is left where Spark placed it.
+ */
+ def subqueryFiltersFromParent(
+ plan: org.apache.spark.sql.execution.SparkPlan,
+ scanExec: FileSourceScanExec): Seq[org.apache.spark.sql.catalyst.expressions.Expression] = {
+ import org.apache.spark.sql.catalyst.expressions.{PlanExpression, SubqueryExpression}
+ import org.apache.spark.sql.execution.{FilterExec, ProjectExec, SparkPlan}
+
+ // Whether every node on the path from `node` down to `scanExec` is one that pushdown can
+ // safely cross: a deterministic ProjectExec is 1:1 on rows (an Alias mints a new exprId, so
+ // it cannot alias over the scan's own output attributes, which is what the reference guard
+ // below requires) and a deterministic FilterExec only removes rows, so moving a predicate
+ // expressed over the scan's output through either preserves the query's semantics. A
+ // nondeterministic projection breaks that: a deterministic conjunct does not commute with it,
+ // because pushing the predicate into the scan changes which rows survive to have
+ // nondeterministic expressions (e.g. monotonically_increasing_id()) evaluated over them,
+ // changing the result -- so both guards require `.deterministic`, mirroring Spark's own
+ // PushPredicateThroughNonJoin/CollapseProject rules. Anything else (LIMIT/TopN, Sort,
+ // Aggregate, Window, joins, Union, Sample, ...) can reorder or drop rows in ways that make
+ // "push the predicate down to the scan" change the result, so an unrecognized node stops the
+ // walk: the filter is left uncollected (missed pruning only, never a correctness issue).
+ def spineToScan(node: SparkPlan): Boolean = node match {
+ case n if n eq scanExec => true
+ case p: ProjectExec if p.projectList.forall(_.deterministic) => spineToScan(p.child)
+ case f: FilterExec if f.condition.deterministic => spineToScan(f.child)
+ case _ => false
+ }
+
+ // Nearest FilterExec whose spine down to the scan is Project/Filter-only (the DV shape
+ // interposes such nodes between them, so do not require a direct parent-child edge).
+ val filtersAboveScan = plan.collect {
+ case f: FilterExec if spineToScan(f.child) => f
+ }
+ filtersAboveScan.lastOption
+ .map { f =>
+ splitConjunctivePredicates(f.condition)
+ .filter(_.deterministic)
+ .filter(_.references.subsetOf(scanExec.outputSet))
+ .filter(p =>
+ SubqueryExpression.hasSubquery(p) || p.exists(_.isInstanceOf[PlanExpression[_]]))
+ .filterNot(p => scanExec.dataFilters.exists(_.semanticEquals(p)))
+ }
+ .getOrElse(Seq.empty)
+ }
+
+ /**
+ * Resolve scalar-subquery data filters at execution time and serialize them for native
+ * pushdown, mirroring `CometNativeScanExec.serializedPartitionData`. `supportedDataFilters`
+ * excludes PlanExpressions at planning time (subquery results do not exist yet), so these
+ * bounds reach the native reader only through this path. Filters that fail to serialize are
+ * skipped: Spark keeps a covering FilterExec above the scan, so this is missed pruning only,
+ * never a correctness issue.
+ *
+ * Known core-parity limitation: when the scan is fused under a parent native operator,
+ * `ensureSubqueriesResolved` has already called `updateResult()` on these subqueries and this
+ * path calls it again (Spark's ScalarSubquery.updateResult re-executes unconditionally). Core's
+ * CometNativeScanExec has the identical double-execution; benign for Delta (the subquery's
+ * snapshot is pinned at analysis), wasteful for expensive subqueries. Fix belongs in core.
+ */
+ def resolvedSubqueryFilters(
+ dataFilters: Seq[org.apache.spark.sql.catalyst.expressions.Expression],
+ output: Seq[org.apache.spark.sql.catalyst.expressions.Attribute],
+ requiredSchema: StructType,
+ conf: org.apache.spark.sql.internal.SQLConf)
+ : Seq[org.apache.comet.serde.ExprOuterClass.Expr] = {
+ if (!conf.getConf(org.apache.spark.sql.internal.SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ return Seq.empty
+ }
+ val subqueryFilters = dataFilters.filter(_.exists(_.isInstanceOf[ExecScalarSubquery]))
+ if (subqueryFilters.isEmpty) {
+ return Seq.empty
+ }
+ // Same binding guard as the DV shape's plan-time filters: references limited to the
+ // data-column prefix of the output, where positions agree between the output and the
+ // native read schema. For the plain shape strippedLen is the full required schema.
+ // Guard BEFORE updateResult so discarded filters never execute their subqueries.
+ val strippedLen = requiredSchema.count(f => !internalColumnNames.contains(f.name))
+ val dataColIds = output.take(strippedLen).map(_.exprId).toSet
+ val pushableFilters =
+ subqueryFilters.filter(_.references.forall(r => dataColIds.contains(r.exprId)))
+ pushableFilters.foreach(_.foreach {
+ case s: ExecScalarSubquery => s.updateResult()
+ case _ =>
+ })
+ pushableFilters
+ .flatMap { filter =>
+ // MergeScalarSubqueries can fuse several scalar subqueries into one struct-returning
+ // subquery accessed via GetStructField; fold that whole subtree to a literal (a bare
+ // GetStructField-over-Literal would not serialize).
+ val resolved = filter.transform {
+ case g @ org.apache.spark.sql.catalyst.expressions
+ .GetStructField(_: ExecScalarSubquery, _, _) =>
+ Literal.create(g.eval(null), g.dataType)
+ case s: ExecScalarSubquery =>
+ Literal.create(s.eval(null), s.dataType)
+ }
+ val proto = exprToProto(resolved, output)
+ if (proto.isEmpty) {
+ logWarning(s"Could not serialize resolved scalar subquery filter: $resolved")
+ }
+ proto
+ }
+ }
+
+ /**
+ * DV shape common builder. Layout invariants (declined by DeltaScanSupport when violated): scan
+ * output = requiredSchema attrs (data columns, then the internal columns as a suffix) followed
+ * by partition and constant-metadata columns. The parquet read schema strips the internal
+ * columns; they are appended to the partition schema as per-file constants, so the projection
+ * vector routes them from the constants block.
+ */
+ private def buildDvScanCommon(
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ firstFileUri: Option[java.net.URI],
+ hadoopConf: org.apache.hadoop.conf.Configuration)
+ : Option[OperatorOuterClass.NativeScanCommon.Builder] = {
+ val relation = scanExec.relation
+ val output = scanExec.output
+ val requiredSchema = scanExec.requiredSchema
+
+ val commonBuilder = OperatorOuterClass.NativeScanCommon.newBuilder()
+ commonBuilder.setSource(scanExec.simpleStringWithNodeId())
+
+ val scanTypes = output.flatMap(attr => serializeDataType(attr.dataType))
+ if (scanTypes.length != output.length) {
+ return None
+ }
+ commonBuilder.addAllFields(scanTypes.asJava)
+
+ val strippedRequired =
+ StructType(requiredSchema.filterNot(f => internalColumnNames.contains(f.name)))
+ val strippedLen = strippedRequired.length
+ val requiredLen = requiredSchema.length
+
+ // Keep only data filters that bind identically in the output and the native
+ // (strippedRequired ++ partitionFields) index spaces: references limited to the first
+ // strippedLen output attributes. Internal-column filters (is_row_deleted = 0) are
+ // trivially true after native DV application, and Spark's Filter above the scan
+ // re-evaluates everything anyway.
+ if (scanExec.conf.getConf(
+ org.apache.spark.sql.internal.SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ val dataColIds = output.take(strippedLen).map(_.exprId).toSet
+ val filterProtos = scanHelper.supportedDataFilters
+ .filter(_.references.forall(r => dataColIds.contains(r.exprId)))
+ .flatMap(f => exprToProto(f, output))
+ commonBuilder.addAllDataFilters(filterProtos.asJava)
+ }
+
+ // Constant metadata columns and real partition columns follow the required schema in the
+ // output, exactly like the plain shape.
+ val constantMetadataFields = scanExec.fileConstantMetadataColumns.map(attr =>
+ StructField(
+ s"${CometNativeScan.constantMetadataFieldPrefix}${attr.name}",
+ attr.dataType,
+ attr.nullable))
+ val internalFields = requiredSchema.fields.toSeq
+ .filter(f => internalColumnNames.contains(f.name))
+ .map(f => StructField(s"$deltaConstFieldPrefix${f.name}", f.dataType, f.nullable))
Review Comment:
**[P2] Make DV synthetic field names unique against the physical schemas**
Could we choose collision-free synthetic names, or decline conflicting scans? A legal user column named `_comet_delta___delta_internal_is_row_deleted` collides with the partition constant generated here. DataFusion substitutes partition constants by column name, so the real data projection is replaced by the bookkeeping value `0`.
A separate Spark/Delta probe accepted that TINYINT column with value `7`, created a real DV through DELETE, and showed the expected internal-column suffix in the scan schema. A standalone probe against the locked public DataFusion 54.1.0 reader then read real Parquet values `(1,7),(2,7)` as `(1,0),(2,0)` with this colliding constant; a distinct-name control preserved `7`. These are separate plan-shape and reader probes, not a full current-head Spark/JNI run.
Please cover the collision in the native Delta differential suite, including the physical data and partition namespaces when allocating the synthetic fields.
##########
contrib/delta-spark/dev/run-delta-regression.sh:
##########
@@ -0,0 +1,163 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Run Delta Lake's own Spark test suites against a Comet build with the
+# native Delta scan enabled. Clones delta at $DELTA_VERSION into $WORKDIR,
+# injects Comet into the test SparkSession (DeltaSQLCommandTest) and the
+# test classpath (unmanagedJars), then runs the given testOnly selectors.
+#
+# Usage:
+# COMET_JARS=/path/comet-spark.jar,/path/comet-contrib-delta.jar,/path/flatbuffers.jar \
+# ./run-delta-regression.sh <workdir> 'org.apache.spark.sql.delta.DeletionVectorsSuite' [...]
+#
+# Env:
+# DELTA_VERSION delta tag to test against (default 3.3.2)
+# COMET_JARS comma-separated jars added to the test classpath (required)
+# JAVA_HOME JDK for sbt (17 recommended)
+set -euo pipefail
+
+DELTA_VERSION="${DELTA_VERSION:-3.3.2}"
+WORKDIR="${1:?usage: run-delta-regression.sh <workdir> <suite> [...suites]}"
+shift
+[ $# -ge 1 ] || { echo "no suites given" >&2; exit 2; }
+: "${COMET_JARS:?COMET_JARS must list the comet jars}"
+
+IFS=',' read -ra _jars <<< "$COMET_JARS"
+for j in "${_jars[@]}"; do
+ [ -f "$j" ] || { echo "COMET_JARS entry not found: $j" >&2; exit 2; }
+done
+
+DELTA_DIR="$WORKDIR/delta-$DELTA_VERSION"
+if [ ! -d "$DELTA_DIR" ]; then
+ git clone --depth 1 --branch "v$DELTA_VERSION" https://github.com/delta-io/delta.git "$DELTA_DIR"
+elif [ ! -d "$DELTA_DIR/.git" ]; then
+ echo "stale/partial checkout at $DELTA_DIR; remove it (rm -rf) and rerun" >&2
+ exit 2
+fi
+cd "$DELTA_DIR"
+
+# Add COMET_EXTRA_JARS to every project's test classpath, plus the JDK-17
+# module-access flags Spark needs (both for forked test JVMs and sbt's own JVM).
+if ! grep -q "COMET_EXTRA_JARS" build.sbt; then
+ python3 - <<'EOF'
+s = open('build.sbt').read()
+marker = 'lazy val commonSettings = Seq('
+opens = [
+ "--add-opens=java.base/java.lang=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED",
+ "--add-opens=java.base/java.io=ALL-UNNAMED",
+ "--add-opens=java.base/java.net=ALL-UNNAMED",
+ "--add-opens=java.base/java.nio=ALL-UNNAMED",
+ "--add-opens=java.base/java.util=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED",
+ "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED",
+ "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
+ "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED",
+ "--add-opens=java.base/sun.security.action=ALL-UNNAMED",
+ "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED",
+ "--add-exports=java.base/sun.nio.ch=ALL-UNNAMED",
+]
+opts = ", ".join('"%s"' % o for o in opens)
+inject = (
+ 'lazy val commonSettings = Seq(\n'
+ ' Test / unmanagedJars ++= sys.env.get("COMET_EXTRA_JARS").toSeq\n'
+ ' .flatMap(_.split(",")).map(p => Attributed.blank(file(p))),\n'
+ ' Test / fork := true,\n'
+ ' Test / javaOptions ++= Seq(%s),\n' % opts
+)
+assert marker in s, 'commonSettings marker not found'
+open('build.sbt', 'w').write(s.replace(marker, inject, 1))
+EOF
+fi
+
+# Inject Comet into the shared test SparkSession when COMET_EXTRA_JARS is set.
+TEST_BASE=spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala
+if ! grep -q "CometSparkSessionExtensions" "$TEST_BASE"; then
+ python3 - "$TEST_BASE" <<'EOF'
+import sys
+p = sys.argv[1]
+s = open(p).read()
+old = ''' override protected def sparkConf: SparkConf = {
+ super.sparkConf
+ .set(StaticSQLConf.SPARK_SESSION_EXTENSIONS.key,
+ classOf[DeltaSparkSessionExtension].getName)
+ .set(SQLConf.V2_SESSION_CATALOG_IMPLEMENTATION.key,
+ classOf[DeltaCatalog].getName)
+ }'''
+new = ''' override protected def sparkConf: SparkConf = {
+ val conf = super.sparkConf
+ .set(StaticSQLConf.SPARK_SESSION_EXTENSIONS.key,
+ classOf[DeltaSparkSessionExtension].getName)
+ .set(SQLConf.V2_SESSION_CATALOG_IMPLEMENTATION.key,
+ classOf[DeltaCatalog].getName)
+ if (sys.env.contains("COMET_EXTRA_JARS")) {
+ conf
+ .set(StaticSQLConf.SPARK_SESSION_EXTENSIONS.key,
+ classOf[DeltaSparkSessionExtension].getName +
+ ",org.apache.comet.CometSparkSessionExtensions")
+ .set("spark.comet.enabled", "true")
+ .set("spark.comet.exec.enabled", "true")
+ .set("spark.comet.exec.shuffle.enabled", "true")
+ .set("spark.shuffle.manager",
+ "org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager")
+ .set("spark.memory.offHeap.enabled", "true")
+ .set("spark.memory.offHeap.size", "2g")
+ .set("spark.comet.scan.delta.enabled", "true")
+ } else conf
+ }'''
+assert old in s, 'sparkConf block not found'
+open(p, 'w').write(s.replace(old, new))
+EOF
+fi
+
+# ScanReportHelper is a test-only trait that counts scans by pattern-matching
+# FileSourceScanExec in the executed plan. The Comet Delta scan replaces those
+# nodes, so claimed scans would go uncounted ("0 did not equal 2" in
+# MergeIntoSuiteBase's insert-only data-skipping test). Map the Comet node back
+# to the FileSourceScanExec it was built from: originalPlan carries the same
+# PreparedDeltaFileIndex, so the reported paths and skipping stats are identical.
+SCAN_HELPER=spark/src/test/scala/org/apache/spark/sql/delta/test/ScanReportHelper.scala
+if [ -f "$SCAN_HELPER" ] && ! grep -q "CometDeltaNativeScanExec" "$SCAN_HELPER"; then
+ python3 - "$SCAN_HELPER" <<'EOF'
+import sys
+p = sys.argv[1]
+s = open(p).read()
+old = " case fs: FileSourceScanExec => Seq(fs)\n"
+new = (" case fs: FileSourceScanExec => Seq(fs)\n"
+ " case c: org.apache.spark.sql.comet.CometDeltaNativeScanExec =>\n"
+ " Seq(c.originalPlan)\n")
+assert s.count(old) == 1, s.count(old)
+open(p, 'w').write(s.replace(old, new))
+EOF
+fi
+
+export COMET_EXTRA_JARS="$COMET_JARS"
+export SPARK_LOCAL_IP=127.0.0.1
+export RUST_BACKTRACE=1
+
+cmds=()
+for sel in "$@"; do
+ cmds+=("spark/testOnly $sel")
+done
+
+LOG="$WORKDIR/delta-regression-$(date +%Y%m%d-%H%M%S).log"
+echo "==> logging to $LOG"
+build/sbt "${cmds[@]}" 2>&1 | tee "$LOG" | grep -E "^\[info\] (Tests:|Suites:|All tests|.*\*\*\* FAILED| - )" | tail -80
Review Comment:
**[P3] Resolve the work directory before changing into the Delta checkout**
Could we normalize `WORKDIR` to an absolute path before `cd "$DELTA_DIR"`? A relative argument such as `work` is initially valid, but this log path is later evaluated from inside `work/delta-<version>`, so `tee` tries to create `work/delta-<version>/work/delta-regression-...log`. That directory normally does not exist, and `set -o pipefail` makes the harness fail even when the test command succeeds.
Verified with the unchanged script and a stubbed `sbt` command to isolate path handling: the relative-workdir case exited 1 with `tee: ... No such file or directory`, while the absolute-workdir control exited 0. No actual Delta test suite was run by that path-handling probe.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,981 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.ListBuffer
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands. Anything else on the protocol declines the
+ * table. Note `deletionVectors` and `columnMapping` are declined separately (below) so their
+ * fallback reasons are specific.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: this is
+ * the first gate on every V1 scan and must stay inert when delta-spark is absent from the
+ * classpath, where `classOf[DeltaParquetFileFormat]` would raise NoClassDefFoundError inside
+ * CometScanRule and take down every parquet scan in the session. A name match proves
+ * delta-spark is present, so Delta types used past this gate are safe.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Returns the first reason this Delta scan cannot go native, or None when claimable. Only
+ * called when [[isDeltaScan]] is true. `scanHelper` is the same [[CometScanExec]] the caller
+ * builds to drive [[CometDeltaNativeScan.convert]] on a claim, reused here to resolve the
+ * scan's selected files for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Descriptor deserialization is expensive; hoisted once so it runs at most once per claim
+ // attempt. `lazy` since most scans are not DV-shaped and gates that return earlier should not
+ // pay for it.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported (the same type checker core's own
+ // FileSourceScanExec path runs) so scan-time type gates -- notably the default-on
+ // COMET_PARQUET_UNSIGNED_SMALL_INT_CHECK safety fallback for ShortType, plus the
+ // collation and shredded-variant-struct gates -- apply identically whether the scan is
+ // claimed through core or through this contrib. A pure in-memory schema check with no
+ // file or store I/O, so it runs first, ahead of every other gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated.
+ val cmMode = metadata.columnMappingMode.name
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder uses the
+ // required schema verbatim as output: name-sensitive expressions (e.g. to_json) would leak
+ // physical names into results. Needs a rename adapter for the logical schema; decline until
+ // then.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bound native's memory for expanded DV row selectors (delta_dv.rs), bounded above by
+ // 2*cardinality + #row-groups; the descriptor's cardinality is a sound, pessimistic upper
+ // bound. The conf below makes an over-pessimistic decline recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities; the native builder resolves the
+ // scan's ObjectStoreUrl from the FIRST selected file only, so a later file under a different
+ // store would silently read through the wrong handle. Force file listing (scanHelper is
+ // already built for the claim path) and decline rather than risk it.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // Credentials that resolve ONLY through a Hadoop credential provider (JCEKS et al.) are
+ // invisible to the plain-conf extraction forwarded to the native S3 client; reuses hadoopConf
+ // from the encryption gate above.
+ val credentialReason = credentialAliasReason(hadoopConf, dataFileUris ++ dvUris)
+ if (credentialReason.isDefined) {
+ return credentialReason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, deserialized once and
+ * normalized to absolute on-disk paths via `copyWithAbsolutePath`. Returns `Seq.empty` for the
+ * plain shape ([[CometDeltaNativeScan.isDvShape]] false). Shared by the DV cardinality gate and
+ * [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset. Hoisted so [[declineReason]]'s
+ * root-path and selected-file scheme gates share one parsed set.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged: such a URI cannot come from a Hadoop-backed source. Factored out of
+ * [[declineReason]]'s two scheme gates so both share one predicate, unit-testable without a
+ * Spark session.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt). See [[declineReason]]'s call site for ordering vs. the authority gates
+ * below.
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme plus the raw
+ * authority -- userinfo, host, port -- lowercased so `S3A://Bucket:1234` and
+ * `s3a://bucket:1234` collapse), or `None` when they share one. `file://` paths carry no
+ * authority, so local scans across many directories are unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the individually-parsed host/port/userinfo fields: `getAuthority` already
+ * includes userinfo, so containers on the same storage account don't collapse together; and
+ * `getHost` (and `getUserInfo`/`getPort`) return `null` for the whole authority when it doesn't
+ * conform to RFC 3986's `reg-name` syntax (e.g. an underscore in a GCS bucket name,
+ * `gs://my_bucket`), silently collapsing distinct buckets into one empty-host key. A `null`
+ * authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`: as with [[uriAuthority]], the structured getters return
+ * `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an underscore
+ * in a GCS bucket name), hiding a real userinfo component. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme` + `://` + a literal `***` + `@host[:port]`, for
+ * embedding in a decline reason. NEVER interpolate `uri.getAuthority` or [[uriUserInfo]]
+ * directly into a reason string: doing so would leak credentials embedded as URI userinfo (e.g.
+ * `s3a://AKIA...:secret@bucket`) into the SQL plan's explain output, fallback-reason logging,
+ * or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle -- normally a missing-object
+ * error, but silently wrong data if a same-named object exists under both.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted by [[credentialAliasReason]] below. `hadoop-aws` is
+ * NOT on this module's runtime classpath (`spark-hadoop-cloud` is test-scope only), so
+ * `org.apache.hadoop.fs.s3a.Constants` must never be referenced here -- doing so would raise
+ * `NoClassDefFoundError` and take down every Delta scan in a session with no S3 dependency at
+ * all, not just S3 ones.
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key
+ * (`fs.s3a.bucket.B.<base key minus its "fs.s3a." prefix>`) -- see [[s3aCredentialAliases]] for
+ * why both must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ /**
+ * The aliases the native S3 credentials provider chain tries per bucket (short bucket form,
+ * then global), PLUS the LONG bucket form Hadoop's `S3AUtils#lookupPassword` also consults.
+ * Hadoop resolves long before short before global, keeping a non-empty long value over the
+ * short/global ones (`S3AUtils#getPassword` returns a non-empty `val` unchanged); native reads
+ * only short+global. So a JCEKS entry set ONLY under the long alias is resolved by Hadoop but
+ * invisible to native -- a shadowed value under ANY alias here means the caller must decline.
+ * List order only affects which alias name appears in the reason.
+ */
+ private def s3aCredentialAliases(bucket: String): Seq[String] =
+ Seq(
+ s"fs.s3a.bucket.$bucket.fs.s3a.access.key",
+ s"fs.s3a.bucket.$bucket.fs.s3a.secret.key",
+ s"fs.s3a.bucket.$bucket.fs.s3a.session.token",
+ s"fs.s3a.bucket.$bucket.access.key",
+ s"fs.s3a.bucket.$bucket.secret.key",
+ s"fs.s3a.bucket.$bucket.session.token",
+ "fs.s3a.access.key",
+ "fs.s3a.secret.key",
+ "fs.s3a.session.token")
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority (host, minus any
+ * userinfo or port), or `None` when `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority
+ * manually (last `@`, then last `:`) rather than using `URI#getHost`: the same RFC 3986
+ * `reg-name` pitfall as [[uriAuthority]] applies (an underscore, valid in an S3 bucket name,
+ * makes `getHost` return `null` for the whole authority).
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ /**
+ * The three per-bucket S3A credential base keys the native S3 client's `get_config` (`s3.rs`)
+ * resolves: short bucket key first, then global (native never reads the long bucket key).
+ */
+ private val PlainCredentialBaseKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket`. Used by the credential-provider-class gates below, which read general S3A options
+ * the same way native does.
+ */
+ private def effectiveOptionValue(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ private def plainLongFormCredentialDivergenceReason(bucket: String, longKey: String): String =
+ "Native Delta scan cannot forward long-form bucket credentials for " +
+ s"$bucket ($longKey is set but the native S3 client only reads the short bucket and " +
+ "global keys, so its credentials would differ from Hadoop's)"
+
+ /**
+ * Zero-I/O plain-value divergence check (see [[s3aCredentialAliases]] for the long-before-short
+ * precedence). Native's `get_config` never reads the long form, so it can silently diverge from
+ * Hadoop's effective value. Declines when the long form is set and the two sides disagree
+ * (short set to the SAME value as long passes). Never interpolates a resolved value, only the
+ * key name.
+ *
+ * This applies to credentials ONLY. Hadoop resolves them via `S3AUtils#lookupPassword`, where
+ * the long bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) WINS when set -- a real, silent
+ * divergence from native, which never reads it. Every OTHER `fs.s3a` option instead flows
+ * through `S3AUtils#propagateBucketOptions`, which strips only ONE `fs.s3a.bucket.B.` prefix
+ * layer: a long-form key there folds into `fs.s3a.fs.s3a.<key>`, a key nothing else in Hadoop
+ * ever reads, so the long form is inert for those options -- Hadoop's own effective value
+ * already reduces to short.orElse(global), matching native. No gate is needed (or present) for
+ * that wider set; see DeltaScanContribSuite's pinned control test for the long-form endpoint
+ * case.
+ */
+ private def plainLongFormCredentialReason(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ PlainCredentialBaseKeys.foldLeft(Option.empty[String]) { (declined, baseKey) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val long = plainValue(hadoopConf, longKey)
+ if (long.isEmpty) {
+ None
+ } else {
+ val short = plainValue(hadoopConf, shortKey)
+ val global = plainValue(hadoopConf, baseKey)
+ val hadoopEffective = long.orElse(short).orElse(global)
+ val nativeEffective = short.orElse(global)
+ if (hadoopEffective != nativeEffective) {
+ Some(plainLongFormCredentialDivergenceReason(bucket, longKey))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * String-literal mirror of every credential-provider class name s3.rs's
+ * `build_aws_credential_provider_metadata` and `is_anonymous_credential_provider` recognize
+ * (Hadoop S3A names plus AWS SDK v1/v2 names). `hadoop-aws` is NOT on this module's runtime
+ * classpath (see the note above [[HadoopCredentialProviderPathKey]]), so these are string
+ * literals, never `org.apache.hadoop.fs.s3a.auth.*` or AWS SDK provider `classOf` references.
+ */
+ private val SupportedCredentialProviderClasses: Set[String] = Set(
+ "org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider",
+ "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider",
+ "com.amazonaws.auth.ContainerCredentialsProvider",
+ "com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper",
+ "software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider",
+ "com.amazonaws.auth.InstanceProfileCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider",
+ "com.amazonaws.auth.EnvironmentVariableCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider",
+ "com.amazonaws.auth.WebIdentityTokenCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider",
+ "com.amazonaws.auth.profile.ProfileCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider",
+ "com.amazonaws.auth.AnonymousAWSCredentials")
+
+ private val AnonymousCredentialProviderClasses: Set[String] = Set(
+ "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider",
+ "com.amazonaws.auth.AnonymousAWSCredentials")
+
+ private val HadoopAssumedRoleProviderClass =
+ "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider"
+
+ private val AwsCredentialsProviderKey = "fs.s3a.aws.credentials.provider"
+ private val AssumedRoleCredentialsProviderKey = "fs.s3a.assumed.role.credentials.provider"
+
+ /**
+ * Splits a Hadoop-style comma-separated credential-provider-class list the same way s3.rs's
+ * `parse_credential_provider_names` does: split on comma, trim each entry, drop empties.
+ */
+ private def parseProviderClassNames(value: String): Seq[String] =
+ value.split(",").map(_.trim).filter(_.nonEmpty).toSeq
+
+ private def unsupportedProviderReason(bucket: String, key: String, className: String): String =
+ s"Native Delta scan does not support the credential provider class $className " +
+ s"configured via $key for $bucket (the native S3 client only supports a fixed set of " +
+ "provider classes; an unsupported class would fail at scan execution time, after the " +
+ "scan was already claimed, rather than at planning time)"
+
+ private def mixedAnonymousProviderReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key for $bucket naming an anonymous credential " +
+ "provider together with any other provider (the native S3 client rejects this " +
+ "combination at scan execution time)"
+
+ private def anonymousAssumedRoleProviderReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support an anonymous credential provider in $key for " +
+ s"$bucket (the native S3 client does not allow an anonymous provider as the base " +
+ "credentials for an assumed-role chain)"
+
+ private def unsupportedProviderNameReason(
+ bucket: String,
+ key: String,
+ names: Seq[String]): Option[String] =
+ names
+ .find(name => !SupportedCredentialProviderClasses.contains(name))
+ .map(unsupportedProviderReason(bucket, key, _))
+
+ /**
+ * Decline reason when `bucket`'s effective (short-bucket-then-global; see
+ * [[effectiveOptionValue]]) `assumed.role.credentials.provider` names an unsupported class, or
+ * an anonymous one (native's `build_assume_role_credential_provider_metadata` rejects ANY
+ * anonymous entry here, not just a mix). Unset defaults to native's own hardcoded
+ * `[SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider]` fallback, both
+ * always supported, so `None` is safe.
+ */
+ private def assumedRoleProviderClassReason(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ effectiveOptionValue(hadoopConf, bucket, AssumedRoleCredentialsProviderKey) match {
+ case None => None
+ case Some(value) =>
+ val names = parseProviderClassNames(value)
+ unsupportedProviderNameReason(bucket, AssumedRoleCredentialsProviderKey, names).orElse {
+ if (names.exists(AnonymousCredentialProviderClasses.contains)) {
+ Some(anonymousAssumedRoleProviderReason(bucket, AssumedRoleCredentialsProviderKey))
+ } else {
+ None
+ }
+ }
+ }
+ }
+
+ /**
+ * Decline reason when `bucket`'s effective `aws.credentials.provider` names a class native's
+ * `build_aws_credential_provider_metadata` (s3.rs) does not recognize, mixes an anonymous
+ * provider with any other provider (native's `build_credential_provider` rejects this
+ * combination), or -- when `AssumedRoleCredentialProvider` is among the names -- its
+ * `assumed.role.credentials.provider` sub-chain has the same problem. An unset/empty value is
+ * fine: native falls back to its own default AWS SDK provider chain. Checked so an unsupported
+ * class or invalid combination declines at planning time instead of erroring during scan
+ * execution, after the scan was already claimed.
+ */
+ private def providerClassReason(hadoopConf: Configuration, bucket: String): Option[String] = {
+ effectiveOptionValue(hadoopConf, bucket, AwsCredentialsProviderKey).flatMap { value =>
+ val names = parseProviderClassNames(value)
+ unsupportedProviderNameReason(bucket, AwsCredentialsProviderKey, names)
+ .orElse {
+ if (names.length > 1 && names.exists(AnonymousCredentialProviderClasses.contains)) {
+ Some(mixedAnonymousProviderReason(bucket, AwsCredentialsProviderKey))
+ } else {
+ None
+ }
+ }
+ .orElse {
+ if (names.contains(HadoopAssumedRoleProviderClass)) {
+ assumedRoleProviderClassReason(hadoopConf, bucket)
+ } else {
+ None
+ }
+ }
+ }
+ }
+
+ /**
+ * Returns the first reason any bucket among `uris` names an unsupported (or invalidly combined)
+ * credential-provider class, or `None` when every bucket's provider configuration is one native
+ * can build. See [[providerClassReason]].
+ */
+ private[delta] def providerClassGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) declined else providerClassReason(hadoopConf, bucket)
+ }
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def shadowedCredentialAliasReason(bucket: String, alias: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($alias resolves through $HadoopCredentialProviderPathKey but is not present " +
+ "as a plain configuration value, so the native S3 client would have no credentials)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * Compares each [[s3aCredentialAliases]] alias's plain `Configuration#get` value against
+ * `Configuration#getPassword` (providers first, plain conf as fallback); a resolved value that
+ * differs means the alias is invisible or wrong to native's plain-conf extraction, so decline.
+ * Only called once the GLOBAL provider path is established as the sole path for `bucket`.
+ *
+ * Runs inside try/catch: `getPassword` performs real keystore I/O, and a corrupt or unreadable
+ * store must decline this bucket rather than abort planning for the whole session.
+ */
+ private def verifyGlobalProviderAliases(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ try {
+ s3aCredentialAliases(bucket).foldLeft(Option.empty[String]) { (declined, alias) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ val resolved =
+ Option(hadoopConf.getPassword(alias)).map(new String(_)).filter(_.nonEmpty)
+ resolved match {
+ case Some(value) if !Option(hadoopConf.get(alias)).contains(value) =>
+ Some(shadowedCredentialAliasReason(bucket, alias))
+ case _ => None
+ }
+ }
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableCredentialProviderReason(bucket, e))
+ }
+ }
+
+ /**
+ * The decline reason for `bucket` alone, or `None` when its credentials are safe to forward.
+ * `globalPathSet`/`s3aPathSet` are hoisted by the caller since they don't vary per bucket.
+ *
+ * Checked FIRST, regardless of provider-path config: [[plainLongFormCredentialReason]] is a
+ * zero-I/O, provider-unrelated check that is decisive on its own.
+ *
+ * Zero-I/O precheck: `None` immediately when none of the four provider-path keys are set -- a
+ * config-map lookup only, no keystore access.
+ *
+ * Arm A: the S3A-scoped provider-path keys point S3A's own resolution at a provider
+ * `Configuration#getPassword` does not consult, so any being set declines with no keystore
+ * read.
+ *
+ * Arm B: only the global path is set, which `getPassword` DOES consult; delegates to
+ * [[verifyGlobalProviderAliases]].
+ */
+ private def bucketCredentialAliasReason(
+ hadoopConf: Configuration,
+ bucket: String,
+ globalPathSet: Boolean,
+ s3aPathSet: Boolean): Option[String] = {
+ plainLongFormCredentialReason(hadoopConf, bucket).orElse {
+ val bucketPathKey = s3aBucketProviderPathKey(bucket)
+ val bucketLongPathKey = s3aBucketLongProviderPathKey(bucket)
+ val bucketPathSet = nonEmptyConf(hadoopConf, bucketPathKey)
+ val bucketLongPathSet = nonEmptyConf(hadoopConf, bucketLongPathKey)
+ if (!globalPathSet && !s3aPathSet && !bucketPathSet && !bucketLongPathSet) {
+ None
+ } else if (s3aPathSet || bucketPathSet || bucketLongPathSet) {
+ val offendingKey =
+ if (s3aPathSet) S3aCredentialProviderPathKey
+ else if (bucketPathSet) bucketPathKey
+ else bucketLongPathKey
+ Some(s3aScopedProviderPathReason(bucket, offendingKey))
+ } else {
+ verifyGlobalProviderAliases(hadoopConf, bucket)
+ }
+ }
+ }
+
+ /**
+ * Returns the first reason a native S3 scan cannot faithfully forward this table's Hadoop
+ * credentials, or `None` when claimable. `uris` is the scan's data-file and DV URIs; only
+ * `s3`/`s3a` authorities matter here (ABFS/WASB mooted by the userinfo gate, GCS out of scope).
+ * `Configuration#getPassword` checks every provider FIRST, falling back to plain conf only when
+ * unset, so a keystore-only or shadowing alias is invisible/wrong to native's plain-conf
+ * extraction; when in doubt, decline. Never interpolates a resolved value, only key names.
+ */
+ private[delta] def credentialAliasReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ if (buckets.isEmpty) {
+ return None
Review Comment:
**[P2] Decline Hadoop-only GCS authentication until it can be forwarded**
Could we also gate GCS authentication compatibility? A scan with local/S3 data files and an absolute `gs://private-bucket/...` DV can pass the scheme and authority gates while relying only on Hadoop's `fs.gs.auth.service.account.json.keyfile` for GCS access. This function and the provider-class gate consider only S3 authorities.
Although conversion forwards `fs.gs.*`, the native resolver's [GCS/default branch](https://github.com/apache/datafusion-comet/blob/1d3557cf7eef766056d4c082f8be89355118bf53/native/core/src/parquet/parquet_support.rs#L565-L574) calls `parse_url(&url)` without those options. With no alternative ADC or metadata-service credentials, Spark can read the sidecar but the claimed native scan fails authentication instead of falling back.
The core GCS options omission predates this PR; the finding here is the Delta admission/fallback gap exposing that limitation. This is a source-traced case; no cloud credentials or GCS requests were used. A conservative fallback would be sufficient until native GCS option translation is supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5332578016
Thanks @dwsmith1983 Your proposed split by review surface sounds reasonable. I agree that the reader, its safety gates, and the essential DML/fallback tests should stay together. Thanks also for being open to aligning with #4952 once it lands. We can leave row tracking and CDF for later discussions rather than expand this PR’s scope. The main additional point I’d like us to settle is keeping experimental Delta support explicitly opt-in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "ErikBPF (via GitHub)" <gi...@apache.org>.
ErikBPF commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5518916280
Follow-up to @comphead's request for a non-suite Delta workload: I ran the current head (`020f0399e`) plus the four proposed optimization commits ([full-group DV skip](https://github.com/dwsmith1983/datafusion-comet/pull/1), [ASCII field-name fast path](https://github.com/dwsmith1983/datafusion-comet/pull/2), [compressed native DV decode](https://github.com/dwsmith1983/datafusion-comet/pull/3), and [compressed partial-DV ranges](https://github.com/dwsmith1983/datafusion-comet/pull/4)) on Apollo.
### Public-data workload
- Host: Intel Xeon E5-2690 v4, 28 logical CPUs; Spark `local[14]`
- Spark 3.5.9, Delta 3.3.2, JDK 17.0.20.1; release Comet build
- Source: [NYC TLC Yellow Taxi January 2024 Parquet](https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet), 49,961,641 bytes, SHA-256 `c4d59da7bbc8abaeeeb1727947ee93d9891a71acb42854bd80db1571b2030510`
- Converted to 16 range-partitioned, timestamp-sorted Delta files with Zstd; enabled DVs; deleted trips from 2024-01-12. The DELETE produced one DV with cardinality 103,655.
- Query: 2024-01-10 through 2024-01-17, aggregating `count(*)`, `sum(PULocationID)`, `sum(DOLocationID)`, and integer passenger count.
- Method: one independent Spark session per mode, with one untimed warm-up followed by five measurements.
| Mode | Exact result `(n, pu, do, passengers)` | Five warm samples (s) | Median |
|---|---|---|---|
| Stock Spark | `(569278, 94416440, 94101779, 733118)` | `0.805, 0.775, 0.792, 0.757, 0.722` | `0.775s` |
| Comet, Delta contrib absent (fallback control) | same | `1.116, 0.876, 0.903, 0.790, 0.801` | `0.876s` |
| Comet native Delta scan | same | `1.071, 0.882, 0.780, 0.757, 0.773` | `0.780s` |
The native plan contained `CometDeltaNativeScan`. Spark/fallback reported 2,964,606 scan output rows; native reported 2,860,951, exactly 103,655 fewer, so every DV row was removed inside the scan. Native `bytes_scanned` was 14,375,149 across the 47,060,535-byte table.
### Repository benchmark corroboration
I also ran the checked-in `contrib/delta-spark/dev/bench_delta_comet.py --dv` workload (20M sorted rows, four files). Five-session medians were stock `0.930s`, fallback `0.997s`, native `0.966s`, with identical `(count, checksum) = (800000, 32000000)`. Native emitted 806,254 rows versus Spark's 1,006,254, read 18,754,673 bytes, and reported six row groups pruned.
### Scope and limitations
This is a single-host local-mode check, not a production or cloud-object-store benchmark. Modes ran sequentially without dropping the OS page cache, so the timings show parity/no obvious regression rather than a general speedup claim. The NYC timestamp predicate reported zero page-index and row-group pruning and scanned all 16 files; that is a concrete remaining optimization opportunity. The public source data is real, but the Delta layout, DV delete, and query were constructed for this test.
Build used `make release PROFILES="-Pspark-3.5 -Pdelta"`; stock, fallback, and native modes shared the same Delta tables. Codex assisted with orchestration and result collection; assertions checked identical results, active DV cardinality, and native-plan presence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5529117842
invite sent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "ErikBPF (via GitHub)" <gi...@apache.org>.
ErikBPF commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3920223308
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -76,6 +77,583 @@ fn schema_has_field_ids(schema: &SchemaRef) -> bool {
schema.fields().iter().any(|f| parse_field_id(f).is_some())
}
+// ---------------------------------------------------------------------------------------------
+// JVM-shipped case tables: reproduce the PLANNING JVM's `String.toLowerCase(Locale.ROOT)`,
+// which Spark's Parquet footer field matching is built on.
+//
+// The data arrives on `NativeScanCommon` (populated by `JvmCaseTables.scala` when
+// case_sensitive = false), generated from the very JVM that plans the query, so the native
+// matcher is correct BY CONSTRUCTION for whatever JDK runs Spark. The one contextual mapping
+// (`Locale.ROOT` has exactly one: Greek capital sigma U+03A3) cannot be a per-codepoint table
+// entry, so its condition is ported as an algorithm over a shipped per-codepoint
+// classification -- see `JvmCaseTables::lowercase`.
+// ---------------------------------------------------------------------------------------------
+
+// Sigma-scan classes: the wire contract shared with `JvmCaseTables.scala`. The classes are
+// the UAX#29-style word-break classes (ALetter, Numeric, MidLetter, MidNum, MidNumLet,
+// Extend, Format) as the PLANNING JVM's legacy break iterator actually realizes them --
+// probed per codepoint from `BreakIterator.isBoundary` on the JVM side -- plus classes for
+// its pre-UAX#29 extensions (the danda and the supplementary-plane behaviors of its UTF-16
+// DFA), with cased variants split out via the JDK's `isCased`. Any codepoint outside every
+// shipped range -- and any class value this build does not know -- is a word boundary: the
+// sigma context scan stops there, which is also the safe reading for class values added by a
+// NEWER JVM-side generator.
+const CLASS_ALETTER_CASED: u8 = 1;
+const CLASS_ALETTER: u8 = 2;
+const CLASS_NUMERIC: u8 = 3;
+const CLASS_MID_LETTER: u8 = 4;
+const CLASS_MID_NUM: u8 = 5;
+const CLASS_MID_NUM_LET: u8 = 6;
+/// Cased supplementary char: attaches to the preceding word and closes it (and forms a word
+/// of its own at raw text start).
+const CLASS_SUPP_CASED: u8 = 7;
+/// U+0964/U+0965: word-terminal, chains only into digits.
+const CLASS_DANDA: u8 = 8;
+/// U+0345, the one cased combining mark: cased only when its run is attached to a word.
+const CLASS_EXTEND_CASED: u8 = 9;
+/// Cased digit-base (Nl Roman numerals): joins like CLASS_ALETTER_CASED when reached
+/// directly, but bridges only mid-num punctuation, never mid-letter.
+const CLASS_NUMERIC_CASED: u8 = 10;
+/// Non-cased Mn/Me marks: riders that attach only to genuine letter/digit bases.
+const CLASS_EXTEND: u8 = 11;
+/// Word-forming non-cased supplementary letter: a genuine letter-base that closes the word
+/// immediately after itself.
+const CLASS_SUPP_LETTER: u8 = 12;
+/// Cf format characters: fully transparent (WB4-style) -- deleted from the sequence before
+/// the scans run, so a pure-format rider chain bridges mid punctuation ("AΣ-<ZWJ>b" is one
+/// word exactly like "AΣ-b").
+const CLASS_FORMAT: u8 = 13;
+/// Supplementary chars that attach to the preceding word but never form one themselves
+/// (supplementary combining marks, tag characters): a cased mark riding on one belongs to
+/// the sigma's word only when the run hangs off a real base (`supp_mn_anchor`).
+const CLASS_SUPP_MN: u8 = 14;
+/// Word-forming supplementary digit: like CLASS_SUPP_LETTER except a riding cased mark
+/// carries only across mid-num (digit-context) punctuation, never mid-letter.
+const CLASS_SUPP_NUM: u8 = 15;
+/// Not on the wire: the absence of a class.
+const CLASS_BOUNDARY: u8 = 0;
+
+const CAPITAL_SIGMA: char = '\u{03A3}';
+const SMALL_SIGMA: char = '\u{03C3}';
+const SMALL_FINAL_SIGMA: char = '\u{03C2}';
+
+/// What a supplementary-mark run ultimately hangs off (see `supp_mn_anchor`).
+#[derive(PartialEq, Eq, Clone, Copy)]
+enum SuppMnAnchor {
+ None,
+ Letter,
+ Digit,
+}
+
+/// The planning JVM's case data, parsed once per scan from `NativeScanCommon` and attached to
+/// [`SparkParquetOptions`]. Two tables:
+///
+/// - `lower`: every codepoint the JVM lowercases non-identically, with its full (possibly
+/// multi-char, e.g. U+0130 -> "i" + U+0307) replacement; codepoints absent here lowercase
+/// to themselves;
+/// - `class_ranges`: sorted, disjoint `(start, end, class)` codepoint ranges holding the
+/// word-break classification the JVM probed from its own `BreakIterator`.
+///
+/// `lowercase` applies Java's algorithm over that data: per codepoint, U+03A3 takes its
+/// contextual final/non-final form via the ported `isFinalCased` condition -- word-boundary
+/// based (the JDK's legacy break-iterator word rules), NOT the Unicode-standard Final_Sigma
+/// case-ignorable skip, so e.g. "A1Σ" lowers to "a1ς" -- and every other codepoint takes its
+/// table replacement. `JvmCaseTables.mirrorLowercase` on the Scala side is the line-for-line
+/// mirror of this function over the same generated data; the JVM-side parity suite proves the
+/// pair equal to the running JDK's `String.toLowerCase(Locale.ROOT)` across the full codepoint
+/// space (calibrated to zero mismatches on JDK 17, 21, and 25).
+#[derive(Debug)]
+pub struct JvmCaseTables {
+ /// Non-identity lowercase mappings: codepoint -> full replacement string.
+ lower: HashMap<char, String>,
+ /// Sorted, disjoint (start, end, class) inclusive codepoint ranges for the sigma scan.
+ class_ranges: Vec<(u32, u32, u8)>,
+ /// Precomputed content hash so `SparkParquetOptions`'s derived `Hash` stays cheap.
+ fingerprint: u64,
+}
+
+impl PartialEq for JvmCaseTables {
+ fn eq(&self, other: &Self) -> bool {
+ self.fingerprint == other.fingerprint
+ && self.class_ranges == other.class_ranges
+ && self.lower == other.lower
+ }
+}
+
+impl Eq for JvmCaseTables {}
+
+impl Hash for JvmCaseTables {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ // Equal contents always produce the equal (deterministically computed) fingerprint,
+ // so hashing only the fingerprint is consistent with `PartialEq`.
+ state.write_u64(self.fingerprint);
+ }
+}
+
+fn is_letter_base(cls: u8) -> bool {
+ cls == CLASS_ALETTER_CASED
+ || cls == CLASS_ALETTER
+ || cls == CLASS_SUPP_CASED
+ || cls == CLASS_SUPP_LETTER
+}
+
+fn is_digit_base(cls: u8) -> bool {
+ cls == CLASS_NUMERIC || cls == CLASS_NUMERIC_CASED
+}
+
+impl JvmCaseTables {
+ /// Parse the proto representation: `lower_cp`/`lower_repl` are index-aligned, and
+ /// `class_ranges` holds (start, end, class) triples. Malformed input (length mismatch,
+ /// trailing partial triple, out-of-range codepoints) is dropped entry-by-entry rather
+ /// than rejected: every dropped entry degrades one codepoint to identity/boundary
+ /// behavior instead of failing the scan.
+ pub fn from_proto(lower_cp: &[u32], lower_repl: &[String], class_ranges: &[u32]) -> Self {
+ let mut lower = HashMap::with_capacity(lower_cp.len());
+ for (cp, repl) in lower_cp.iter().zip(lower_repl.iter()) {
+ if let Some(c) = char::from_u32(*cp) {
+ lower.insert(c, repl.clone());
+ }
+ }
+ let ranges: Vec<(u32, u32, u8)> = class_ranges
+ .as_chunks::<3>()
+ .0
+ .iter()
+ .filter(|t| t[0] <= t[1] && t[1] <= 0x10FFFF && u8::try_from(t[2]).is_ok())
+ .map(|t| (t[0], t[1], t[2] as u8))
+ .collect();
+
+ let mut hasher = DefaultHasher::new();
+ for (start, end, class) in &ranges {
+ hasher.write_u32(*start);
+ hasher.write_u32(*end);
+ hasher.write_u8(*class);
+ }
+ let mut lower_sorted: Vec<(&char, &String)> = lower.iter().collect();
+ lower_sorted.sort_by_key(|(c, _)| **c);
+ for (c, repl) in lower_sorted {
+ hasher.write_u32(*c as u32);
+ hasher.write(repl.as_bytes());
+ }
+
+ Self {
+ lower,
+ class_ranges: ranges,
+ fingerprint: hasher.finish(),
+ }
+ }
+
+ /// Sigma-scan class of `c`; `CLASS_BOUNDARY` when no shipped range covers it.
+ fn class_of(&self, c: char) -> u8 {
+ let cp = c as u32;
+ let mut lo = 0usize;
+ let mut hi = self.class_ranges.len();
+ while lo < hi {
+ let mid = (lo + hi) / 2;
+ let (start, end, class) = self.class_ranges[mid];
+ if cp < start {
+ hi = mid;
+ } else if cp > end {
+ lo = mid + 1;
+ } else {
+ return class;
+ }
+ }
+ CLASS_BOUNDARY
+ }
+
+ /// First position at or beyond `start` (stepping by `step`, i.e. -1 backward / +1
+ /// forward) whose class is not CLASS_EXTEND. Returns `None` if the scan runs off the
+ /// array without finding one.
+ fn skip_extends(&self, cps: &[char], start: isize, step: isize) -> Option<usize> {
+ let mut k = start;
+ while k >= 0 && (k as usize) < cps.len() {
+ if self.class_of(cps[k as usize]) != CLASS_EXTEND {
+ return Some(k as usize);
+ }
+ k += step;
+ }
+ None
+ }
+
+ /// As [`Self::skip_extends`], but also skips CLASS_EXTEND_CASED, reporting whether one
+ /// was walked: a cased mark (U+0345) crossed while looking for a base is itself cased
+ /// whenever the landing validates the run.
+ fn skip_extends_tracking_cased(
+ &self,
+ cps: &[char],
+ start: isize,
+ step: isize,
+ ) -> (Option<usize>, bool) {
+ let mut k = start;
+ let mut saw_cased = false;
+ while k >= 0 && (k as usize) < cps.len() {
+ let cls = self.class_of(cps[k as usize]);
+ if cls != CLASS_EXTEND && cls != CLASS_EXTEND_CASED {
+ return (Some(k as usize), saw_cased);
+ }
+ if cls == CLASS_EXTEND_CASED {
+ saw_cased = true;
+ }
+ k += step;
+ }
+ (None, saw_cased)
+ }
+
+ /// What the supplementary-mark run at `k` (CLASS_SUPP_MN) ultimately hangs off, walking
+ /// down through further marks and supplementary chars: a letter-flavored base, a
+ /// digit-flavored base, or nothing word-forming. A cased mark riding the run belongs to
+ /// the sigma's word only per this anchor.
+ fn supp_mn_anchor(&self, cps: &[char], k: usize) -> SuppMnAnchor {
+ let mut m = k as isize - 1;
+ while m >= 0 {
+ let cls = self.class_of(cps[m as usize]);
+ if cls != CLASS_SUPP_MN && cls != CLASS_EXTEND && cls != CLASS_EXTEND_CASED {
+ break;
+ }
+ m -= 1;
+ }
+ if m < 0 {
+ return SuppMnAnchor::None;
+ }
+ match self.class_of(cps[m as usize]) {
+ CLASS_ALETTER_CASED | CLASS_ALETTER | CLASS_SUPP_CASED | CLASS_SUPP_LETTER => {
+ SuppMnAnchor::Letter
+ }
+ CLASS_NUMERIC | CLASS_NUMERIC_CASED | CLASS_SUPP_NUM => SuppMnAnchor::Digit,
+ _ => SuppMnAnchor::None,
+ }
+ }
+
+ /// Backward half of the ported `isFinalCased`: is there a cased letter before position
+ /// `i` within the sigma's word? Runs over the FORMAT-FILTERED sequence; `leading_format`
+ /// says whether format chars were filtered off the raw text start.
+ fn scan_back_finds_cased(&self, cps: &[char], i: usize, leading_format: bool) -> bool {
+ let mut last_letter = true; // the sigma itself is a letter
+ let mut j = i as isize - 1;
+ while j >= 0 {
+ match self.class_of(cps[j as usize]) {
+ CLASS_ALETTER_CASED | CLASS_NUMERIC_CASED => return true,
+ CLASS_ALETTER => {
+ last_letter = true;
+ j -= 1;
+ }
+ CLASS_NUMERIC => {
+ last_letter = false;
+ j -= 1;
+ }
+ CLASS_EXTEND => {
+ // Non-cased marks attach only to a real base below them; anything else
+ // (mid punctuation, danda, boundary, text start) leaves the run
+ // unattached.
+ let Some(k) = self.skip_extends(cps, j, -1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ let is_continuer = b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_EXTEND_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_NUM;
+ if !is_continuer {
+ return false;
+ }
+ j = k as isize;
+ }
+ CLASS_SUPP_CASED => {
+ // Closes the preceding word, so the scan stops -- except at RAW text
+ // start (no filtered-out leading format chars), where the DFA keeps it
+ // joined to what follows.
+ return j == 0 && !leading_format;
+ }
+ CLASS_SUPP_LETTER | CLASS_SUPP_MN | CLASS_SUPP_NUM => {
+ // Attach/close and never themselves cased; nothing beyond is reachable.
+ return false;
+ }
+ CLASS_EXTEND_CASED => {
+ // Cased combining mark (U+0345): cased when its run hangs off a base --
+ // a BMP letter/digit, a word-forming supplementary char (which closes a
+ // word right below the mark, merging the mark into the sigma's
+ // segment), or an ANCHORED supplementary mark.
+ let (Some(k), _) = self.skip_extends_tracking_cased(cps, j - 1, -1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ if b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_NUM
+ {
+ return true;
+ }
+ if b == CLASS_SUPP_MN {
+ return self.supp_mn_anchor(cps, k) != SuppMnAnchor::None;
+ }
+ return false;
+ }
+ CLASS_DANDA => {
+ // Backward across a danda: the word part before it must end in letters
+ // (grammar: letters, optional danda, then number+word chains) -- or
+ // carry a riding cased mark on a word-forming base, or be a cased
+ // supplementary char at text start -- and the danda itself chains only
+ // into digits after it.
+ if last_letter {
+ return false;
+ }
+ let (Some(k), saw_cased_mark) =
+ self.skip_extends_tracking_cased(cps, j - 1, -1)
+ else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ if b == CLASS_ALETTER_CASED {
+ return true;
+ }
+ if b == CLASS_SUPP_CASED {
+ return saw_cased_mark || (k == 0 && !leading_format);
+ }
+ if b == CLASS_SUPP_LETTER {
+ return saw_cased_mark;
+ }
+ if b == CLASS_SUPP_MN {
+ return saw_cased_mark
+ && self.supp_mn_anchor(cps, k) == SuppMnAnchor::Letter;
+ }
+ if b != CLASS_ALETTER {
+ return false;
+ }
+ if saw_cased_mark {
+ return true;
+ }
+ last_letter = true;
+ j = k as isize;
+ }
+ cls @ (CLASS_MID_LETTER | CLASS_MID_NUM | CLASS_MID_NUM_LET) => {
+ // `<mid-letter><let>` / `<mid-num><digit>` require a genuine
+ // letter/digit base before the punctuation; scanning backward
+ // legitimately walks marks-then-base (marks trail their base). A cased
+ // mark walked over rides whatever the punctuation hangs off, including
+ // a context-matching anchored supplementary mark or supplementary
+ // digit.
+ let mw_ok = cls == CLASS_MID_LETTER || cls == CLASS_MID_NUM_LET;
+ let mn_ok = cls == CLASS_MID_NUM || cls == CLASS_MID_NUM_LET;
+ let (Some(real_pos), saw_cased_mark) =
+ self.skip_extends_tracking_cased(cps, j - 1, -1)
+ else {
+ return false;
+ };
+ let b = self.class_of(cps[real_pos]);
+ if last_letter
+ && mw_ok
+ && saw_cased_mark
+ && b == CLASS_SUPP_MN
+ && self.supp_mn_anchor(cps, real_pos) == SuppMnAnchor::Letter
+ {
+ return true;
+ }
+ if !last_letter
+ && mn_ok
+ && saw_cased_mark
+ && (b == CLASS_SUPP_NUM
+ || (b == CLASS_SUPP_MN
+ && self.supp_mn_anchor(cps, real_pos) == SuppMnAnchor::Digit))
+ {
+ return true;
+ }
+ let bridge_valid = (last_letter && mw_ok && is_letter_base(b))
+ || (!last_letter && mn_ok && is_digit_base(b));
+ if !bridge_valid {
+ return false;
+ }
+ if saw_cased_mark {
+ return true;
+ }
+ j = real_pos as isize;
+ }
+ _ => return false,
+ }
+ }
+ false
+ }
+
+ /// Forward half of the ported `isFinalCased`: is there a cased letter after position `i`
+ /// within the sigma's word? Runs over the FORMAT-FILTERED sequence.
+ fn scan_fwd_finds_cased(&self, cps: &[char], i: usize) -> bool {
+ let mut last_letter = true;
+ let mut j = i + 1;
+ while j < cps.len() {
+ match self.class_of(cps[j]) {
+ CLASS_ALETTER_CASED | CLASS_NUMERIC_CASED => return true,
+ CLASS_ALETTER => {
+ last_letter = true;
+ j += 1;
+ }
+ CLASS_NUMERIC => {
+ last_letter = false;
+ j += 1;
+ }
+ CLASS_EXTEND => {
+ // A mark run trailing the anchor is properly attached in text order, so
+ // the run stays open past it, including onto mid punctuation on its far
+ // side.
+ let Some(k) = self.skip_extends(cps, j as isize, 1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ let is_continuer = b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_EXTEND_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_MN
+ || b == CLASS_SUPP_NUM
+ || b == CLASS_DANDA
+ || b == CLASS_MID_LETTER
+ || b == CLASS_MID_NUM
+ || b == CLASS_MID_NUM_LET;
+ if !is_continuer {
+ return false;
+ }
+ j = k;
+ }
+ CLASS_SUPP_CASED | CLASS_EXTEND_CASED => {
+ // Attaches to the current word, so the scan sees it (cased).
+ return true;
+ }
+ CLASS_SUPP_LETTER | CLASS_SUPP_MN | CLASS_SUPP_NUM => {
+ // Attach to the current word and close it; never themselves cased, and
+ // nothing beyond is reachable.
+ return false;
+ }
+ // The danda attaches only to a word part that ends in letters (reached
+ // after digits the word is already closed) and continues only into a digit
+ // -- unless that digit is itself cased (a Roman numeral), which resolves
+ // the scan immediately.
+ CLASS_DANDA if !last_letter => return false,
+ CLASS_DANDA
+ if j + 1 < cps.len() && self.class_of(cps[j + 1]) == CLASS_NUMERIC_CASED =>
+ {
+ return true;
+ }
+ CLASS_DANDA if j + 1 < cps.len() && self.class_of(cps[j + 1]) == CLASS_NUMERIC => {
+ last_letter = false;
+ j += 2;
+ }
+ cls @ (CLASS_MID_LETTER | CLASS_MID_NUM | CLASS_MID_NUM_LET) => {
+ // `<mid-letter><let>` / `<mid-num><digit>` require a genuine
+ // letter/digit base IMMEDIATELY after the punctuation -- unlike the
+ // backward scan, marks here are never skipped past: a mark directly
+ // after the punctuation is attached to the punctuation, not a base, so
+ // it blocks the bridge. (Format chars are already filtered out, which
+ // is what lets "AΣ-<ZWJ>b" bridge exactly like "AΣ-b".)
+ let mw_ok = cls == CLASS_MID_LETTER || cls == CLASS_MID_NUM_LET;
+ let mn_ok = cls == CLASS_MID_NUM || cls == CLASS_MID_NUM_LET;
+ if j + 1 >= cps.len() {
+ return false;
+ }
+ let b = self.class_of(cps[j + 1]);
+ if (last_letter && mw_ok && is_letter_base(b))
+ || (!last_letter && mn_ok && is_digit_base(b))
+ {
+ j += 1;
+ } else {
+ return false;
+ }
+ }
+ _ => return false,
+ }
+ }
+ false
+ }
+
+ /// Lowercase `s` the way the planning JVM's `String.toLowerCase(Locale.ROOT)` does.
+ pub fn lowercase(&self, s: &str) -> String {
+ let raw: Vec<char> = s.chars().collect();
+ // Built lazily on the first sigma: the format-filtered sequence the scans run over
+ // (WB4-style: the legacy break iterator's `<ignore>` class loops on every DFA
+ // state), the raw->filtered index map, and whether format chars led the raw text.
+ let mut filtered: Option<(Vec<char>, Vec<usize>, bool)> = None;
+ let mut out = String::with_capacity(s.len());
+ for (i, &c) in raw.iter().enumerate() {
+ if c == CAPITAL_SIGMA {
+ // The condition consults the ORIGINAL neighbors, exactly as the JDK scans
+ // `src`, not the partially-lowered output. The final/non-final target chars
+ // are Unicode-stable (pinned in `ConditionalSpecialCasing`'s entry table).
+ let (f, idx, leading_format) = filtered.get_or_insert_with(|| {
+ let mut f = Vec::with_capacity(raw.len());
+ let mut idx = vec![0usize; raw.len()];
+ for (k, &rc) in raw.iter().enumerate() {
+ idx[k] = f.len();
+ if self.class_of(rc) != CLASS_FORMAT {
+ f.push(rc);
+ }
+ }
+ let leading_format = self.class_of(raw[0]) == CLASS_FORMAT;
+ (f, idx, leading_format)
+ });
+ let fi = idx[i];
+ let is_final = self.scan_back_finds_cased(f, fi, *leading_format)
+ && !self.scan_fwd_finds_cased(f, fi);
+ out.push(if is_final {
+ SMALL_FINAL_SIGMA
+ } else {
+ SMALL_SIGMA
+ });
+ } else if let Some(repl) = self.lower.get(&c) {
+ out.push_str(repl);
+ } else {
+ out.push(c);
+ }
+ }
+ out
+ }
+}
+
+/// Lowercase `s` for case-insensitive field matching. With tables (populated whenever
+/// `case_sensitive = false`, shared by the core Parquet scan and the Delta contrib scan) this
+/// reproduces the planning JVM's `String.toLowerCase(Locale.ROOT)` exactly.
+///
+/// Without tables, fall back to Rust's `str::to_lowercase`, which agrees with Java on all
+/// simple mappings and differs only where the two Unicode snapshots or the sigma context
+/// diverge. This is a real, live path, not just a defensive default: the Iceberg native scan
+/// (`SparkPhysicalExprAdapterFactory::new(_, None)`) defaults `case_sensitive = false` and
+/// always reaches this fallback for its schema name remap, alongside
+/// `parquet_convert_struct_to_struct`'s general struct-cast matching and Rust-only unit tests
+/// that construct `SparkParquetOptions` directly.
+pub(crate) fn java_lowercase(s: &str, tables: Option<&JvmCaseTables>) -> String {
Review Comment:
Implemented the exact ASCII guard in [dwsmith1983/datafusion-comet#2](https://github.com/dwsmith1983/datafusion-comet/pull/2) (commit `bafc3cdd0`), targeting this PR head branch.
It preserves the JVM path whenever either name is non-ASCII and adds a focused test proving ASCII matching is independent of the JVM case tables. Validation: `cargo test -p datafusion-comet` (316 passed, 19 ignored), clippy/fmt clean. An independent release-mode matcher probe on Apollo measured 40.1–41.3x versus the previous expression over 1,005,000 comparisons/sample.
This intentionally does not attempt the separate schema-wide non-ASCII hoisting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "erikbogado-nstech (via GitHub)" <gi...@apache.org>.
erikbogado-nstech commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5526207732
> @andygrove @sunchao @ErikBPF @comphead do you all have a Slack channel we can use to plan?
No, but I am open to using one if you guys have one. Also open to using discord, since free slack caps message history
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3878126725
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.ListBuffer
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable. Only called when
+ * [[isDeltaScan]] is true. `scanHelper` is the [[CometScanExec]] built to drive
+ * [[CometDeltaNativeScan.convert]] on a claim, reused for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`: it runs at most
+ // once per claim attempt, and not at all for the common non-DV-shape case.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
Review Comment:
**[P2] Decline SSE-C scans until the customer key reaches S3**
With the Delta contrib enabled, an existing S3 Delta table readable through Hadoop using `fs.s3a.encryption.algorithm=SSE-C` plus `fs.s3a.encryption.key` passes this check: it only detects Parquet modular encryption. The later S3 gates also accept it. `NativeConfig` forwards these settings, but native `extract_s3_config_options` drops them, leaving [GET](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) and [HEAD](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) without their required customer encryption headers. The new Delta admission therefore exposes those SSE-C objects to native read failures; the base retained Spark's reader. Could we conservatively decline SSE-C until the native client receives the key?
Fresh isolated checks cover canonical/deprecated global and bucket settings: unchanged storage helpers accept them, Hadoop 3.3.4 request factories attach the key, and the unchanged native translation with locked object_store 0.13.2 produces requests missing all three headers. An explicit SSE-C builder control supplies them. Full admission and the read-failure consequence were source/API-traced; no live S3 or full Spark/Delta/JNI read was run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3826296728
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
Review Comment:
For now this declines column mapping over nested structs rather than restoring the names. Restoring needs a logical-schema field in the proto plus a native rename step, since the existing adapter remap is top-level only and field-id based, and we strip field ids to support pre-upgrade files. The decline does cost the ordinal-access case that worked before, which I converted into an explicit fallback test rather than leaving it implicit. Happy to do the restore-names design as the follow-up that lifts the gate if that ordering works for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5350890419
@sunchao
Merged main to pick up #4952 and reconciled the two Delta efforts as discussed. The JVM-planned scan now rides the generic ContribScan envelope with its own type_url (comet.contrib.delta_spark.DeltaSparkScan), so the dedicated oneof slot is gone (removed and reserved). The native handler is now a sibling of the kernel path's handler, dispatched by type_url, and the module moved to contrib/delta-spark so it no longer overlaps contrib/delta's source root. Our proto messages are renamed DeltaSpark* so both message sets coexist, and nothing from #4952 was reverted or modified; verify-contrib-delta-gate.sh passes unchanged. Both contribs' suites are green side by side (contrib 40/40, CometScanContribSuite and the injector suites 29/29, native 172/172).
A few things I deliberately left for discussion rather than deciding unilaterally: unifying the two claim hooks in CometScanRule (CometScanContrib vs the CometScanRuleExtension SPI), conf naming (spark.comet.scan.delta.* vs spark.comet.scan.deltaNative.*), and Maven packaging (the -Pcontrib-delta add-source vs this module's separate jar, which is what keeps the opt-in story build-free).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5876419327
This is a light fully automated review since there are so many PRs open.
`leaf_policies` in `native/core/src/parquet/datetime_rebase.rs:689` picks the timestamp policy from the physical Arrow type and treats every timezone-free physical timestamp as `Corrected`, on the premise in the doc comment at line 687 that Spark never rebases these. Spark keys on the requested type instead. In `ParquetVectorUpdaterFactory` on 3.5.8, 4.0.1 and 4.1.1, an INT64 column requested as `TimestampType` gets `LongWithRebaseUpdater` (or `LongAsMicrosRebaseUpdater` for millis) whenever the file's mode is not CORRECTED, and `isTimestampTypeMatched` checks only the unit, never `isAdjustedToUTC`. Comet's adapter accepts that pairing by reinterpreting the micros (`parquet_support.rs:304`), so the column is read with no rebase wrapper at all. Take a Delta table whose schema has `ts TIMESTAMP` while its files store `ts` as INT64 `TIMESTAMP(MICROS, isAdjustedToUTC=false)` with no Spark footer metadata, for example pyarrow-written Parquet converted with `CONVERT TO DELTA` under `spa
rk.sql.parquet.inferTimestampNTZ.enabled=false`. On Spark 3.5 with the default `EXCEPTION` read mode, a `1800-01-01 00:00:00` value makes Spark fail with the ancient-datetime `SparkUpgradeException`, while the native Delta scan returns the row. Under `LEGACY`, Spark rebases pre-1582 values and the native scan returns them unchanged. This is the mirror image of the INT96 `TIMESTAMP_NTZ` case from the latest review. Could the physical-to-requested leaf pairing choose the timestamp policy from the requested type in both directions? A metadata-free `TIMESTAMP(MICROS, false)` file read as `TIMESTAMP` under `EXCEPTION` would make a good regression next to that one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-6131709649
@dwsmith1983 could you fix conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3996879797
##########
pom.xml:
##########
@@ -95,6 +95,11 @@ under the License.
<guava.version>33.2.1-jre</guava.version>
<testcontainers.version>1.21.4</testcontainers.version>
<amazon-awssdk-v2.version>2.31.51</amazon-awssdk-v2.version>
+ <!-- Delta Lake pairing for the contrib/delta-spark module (-Pdelta). Each Spark profile
+ overrides these with its matching Delta release; the defaults match the default
+ spark-4.1 profile. Delta 2.x ships as artifact delta-core, 3.x/4.x as delta-spark. -->
+ <delta.artifact>delta-spark</delta.artifact>
+ <delta.version>4.3.1</delta.version>
Review Comment:
`delta.version` is now declared twice in this `<properties>` block: `4.1.0` at line 54 (for the kernel `contrib-delta` profile) and `4.3.1` here. Maven takes the last one so the build is right, but the comment above line 54 now describes a pairing that no longer applies (`spark-4.1 -> 4.1.0`). Could the first declaration and its comment go, or the two contribs use distinct property names so a reader does not have to work out which one wins?
##########
spark/src/main/scala/org/apache/comet/rules/CometScanContrib.scala:
##########
@@ -147,6 +154,21 @@ object CometScanContrib extends Logging {
"declining it and continuing with Comet's built-in handling",
e)
None
+ case e: LinkageError =>
+ // A version-skewed contrib jar (compiled against a Comet internal that has since
Review Comment:
This arm is the right idea, but the containment does not reach discovery. `ContribServices.loadFrom` (`ContribServices.scala:97-99`, not touched by this PR) catches only `NonFatal`, and `ServiceLoader` raises `NoClassDefFoundError` straight from `Class.forName` when a provider's superclass or interface is missing, which is exactly the version-skewed-jar case this comment describes. Because `contribs` is a `lazy val`, the failed initializer is re-run on every access, so every V1 and V2 scan would throw rather than fall back.
Could the discovery loop get the same `LinkageError` arm (log and skip), with a test alongside `FatalScanContrib` that drives discovery against a provider whose interface cannot load?
##########
contrib/delta-spark/README.md:
##########
@@ -0,0 +1,60 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Comet Delta Lake Contrib (experimental)
+
+Native Delta Lake reads for Comet. Delta tables are scanned through Comet's
+existing native Parquet reader, so they get row-group pruning, page-index
+pruning, and filter pushdown, with deletion vectors applied inside the scan.
+
+Support is experimental and explicitly opt-in. Two things are required:
+
+1. This module's jar (`comet-contrib-delta-spark`) on the classpath, alongside
+ `delta-spark`. It is never bundled into `comet-spark`; without it, Comet
+ has no Delta surface at all.
+2. `spark.comet.scan.delta.enabled=true`. The default is `false`, so the jar
+ alone does nothing.
+
+Unsupported tables and features fall back to Spark's reader. See the
+[user guide](https://datafusion.apache.org/comet/user-guide/delta.html)
Review Comment:
Two things in this README:
- This link resolves to `user-guide/delta.html`, but the page lives under `user-guide/latest/`, and `docs/source/conf.py` only has redirects for the pre-existing pages, so it will 404. `latest/delta.html` or a redirect entry would fix it.
- Line 53 builds with `-pl contrib/delta-spark`, which resolves `comet-spark` from the local Maven repo. That is the stale-sibling trap the contributor guide warns about. CI is fine because the workflow installs `common,spark` immediately before. Could the README say the same, or run the full reactor?
##########
spark/pom.xml:
##########
@@ -585,6 +585,19 @@ under the License.
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
</plugin>
+ <plugin>
Review Comment:
This execution now runs on every profile, not only under `-Pdelta`: I measured a 6.7 MB `-tests.jar` in `spark/target`, `install` puts it in the local repo, and `dev/release/publish-to-maven.sh` uploads every jar it finds, so each release would ship six of them. There is partial precedent (the `-test-sources.jar`), but it should be a deliberate choice. Binding the execution inside a `delta` profile in this pom would keep it to the builds that need it. Also the comment says `contrib/delta`; the consumer is `contrib/delta-spark`.
Related maintainer question I am raising here so it gets decided before merge: `dev/release/build-release-comet.sh` never passes `-Pdelta`, so the contrib jar the docs tell users to put on the classpath is never built or published, and `maven.deploy.skip=false` in the contrib pom is moot today. Either the release build adds `-Pdelta` (and then the artifact name `comet-contrib-delta-spark4.1_2.13` deserves a look against the `comet-spark-spark4.1_2.13` convention), or the docs should say build-from-source is the only route for now.
##########
dev/ci/compute-changes.py:
##########
@@ -306,6 +306,23 @@
".mvn/**",
"mvnw",
],
+ "delta": [
+ "contrib/delta/**",
Review Comment:
A few things about this filter and the workflow it triggers, since every queue run touching native or spark sources will now run three container jobs:
- `contrib/delta-spark/README.md` alone triggers the full suite (no `!**.md` exclude, unlike `build_linux`).
- `contrib/delta/**` is the kernel crate, which this workflow never builds (`delta_build_gate.yml` covers it). It can be dropped here.
- `spark/src/test/**` is excluded, but the contrib suites extend `CometTestBase` and `CometS3TestBase` through the new test-jar, so a change to those bases can break the contrib without running it. Adding those two files would close that.
- In the workflow itself: `feature-off-build` restores no cargo cache, so it is a cold debug build to run one test, and `dev-scripts-python` spends four runners byte-compiling two scripts. One version, or folding it into preflight, seems proportionate. The cache-save step gated on `refs/heads/main` never runs under this policy (the restore does fall back warm to the `build_linux` cache, and `--features delta` is a no-op since it is in the default set).
##########
docs/source/contributor-guide/ci.md:
##########
@@ -69,6 +69,7 @@ Each queue-only suite has a label that runs it on a pull request:
| `run-spark-3.4-tests` | Spark SQL tests against Spark 3.4 |
| `run-spark-3.5-tests` | Spark SQL tests against Spark 3.5 |
| `run-spark-4.0-tests` | Spark SQL tests against Spark 4.0 |
+| `run-delta-tests` | Delta contrib tests against Spark 3.5 |
Review Comment:
The matrix in `delta_contrib_test.yml` runs Spark 3.5, 4.0 and 4.1, but this row, the `POLICY` comment in `dev/ci/compute-changes.py:376`, and the reusable-workflow list and label sentence in `.github/workflows/README.md` still describe it as Spark 3.5 only. I confirmed the 4.1 leg passes locally, so the description could claim it as well.
##########
.github/workflows/delta_contrib_test.yml:
##########
@@ -0,0 +1,169 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Delta Contrib Tests
+
+# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
+# live in the umbrella workflow.
+on:
+ workflow_call:
+
+permissions:
+ contents: read
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: rust-lld cannot resolve -ljvm against the Zulu JDK
+ # layout installed by setup-java (same rationale as pr_build_linux.yml).
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+ # The container's default locale is POSIX, which makes the JVM's file-path encoder
+ # reject non-ASCII partition directory names the suites create.
+ LANG: "C.UTF-8"
+ LC_ALL: "C.UTF-8"
+
+jobs:
+
+ contrib-delta:
+ name: Delta contrib (Spark ${{ matrix.profile.spark }})
+ runs-on: ubuntu-24.04
+ container:
Review Comment:
This container has no Docker socket, and `CometDeltaS3Suite` `assume()`s out when `DockerClientFactory.isDockerAvailable` is false, which scalatest reports as canceled and the build treats as green. So the MinIO suite contributes no coverage in CI even though the description lists it as live, and the S3 gate is the logic I would most like exercised end to end.
Could you either mount `/var/run/docker.sock` into this job (or run that one suite outside the container), or state in the workflow that the S3 suite is manual-only and drop it from the description's CI coverage claim?
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1838 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // An opted-in S3-compliant alias scheme (fs.comet.s3Compliant.schemes) is declined before the
+ // generic scheme gate below so the reason says why: core's native scan reads it through the
+ // S3 client, but the S3 divergence gates further down model Hadoop's S3AFileSystem only.
+ val rootUris = scanExec.relation.location.rootPaths.map(_.toUri)
+ val aliasReason = s3CompliantAliasSchemeReason(hadoopConf, rootUris)
+ if (aliasReason.isDefined) {
+ return aliasReason
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes = unsupportedSchemes(rootUris, libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A recognized scheme can still carry a path object_store rejects (a directory name with a
+ // newline surfaces as `%0A`), which native planning hard-fails on while Spark's reader opens
+ // it. Mirrors core's root-path gate; the complete selected paths are probed below.
+ val rejectedRoot = objectStoreRejectedPathReason(rootUris, libhdfs)
+ if (rejectedRoot.isDefined) {
+ return rejectedRoot
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Same path probe for every complete selected path, not just its directory: a shallow
+ // clone's source can sit outside this root, and CONVERT TO DELTA keeps the source Parquet
+ // basenames, so the rejected character can be in the file name itself. The probe is a
+ // native URL parse with no I/O, so once per distinct URI costs less than the scan's own
+ // per-file parse.
+ val rejectedSelected = objectStoreRejectedPathReason(dataFileUris ++ dvUris, libhdfs)
+ if (rejectedSelected.isDefined) {
+ return rejectedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Always zero-I/O (plain propagated-conf read, no keystore): native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Zero-I/O, conf-only, like the proxy gate above: Hadoop's AssumedRoleCredentialProvider
+ // sends fs.s3a.assumed.role.policy as the session policy of its STS AssumeRole request,
+ // while native's assumed-role provider never reads the key -- a claimed scan would assume
+ // the role WITHOUT the configured session restriction, silently widening permissions.
+ val rolePolicyReason =
+ assumedRolePolicyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (rolePolicyReason.isDefined) {
+ return rolePolicyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * parsed exactly like core's scan gate (`NativeConfig.parseSchemeSet`: split on commas,
+ * trimmed, lowercased) and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) => NativeConfig.parseSchemeSet(s)
+ case None => Set("hdfs")
+ }
+
+ /**
+ * Decline reason when any of `uris` uses a scheme opted in as an S3-compliant alias through
+ * `fs.comet.s3Compliant.schemes` (e.g. `blob`), or `None`. Core's native Parquet scan admits
+ * such a scheme and reads it through its S3 client, with `NativeConfig` translating the vendor
+ * `fs.<scheme>.<authority>.*` keys into `fs.s3a.bucket.*` options. Spark, however, reads the
+ * same table through the vendor's own Hadoop FileSystem, not `S3AFileSystem`, and every S3
+ * divergence gate in this object ([[s3ConfigDivergenceReason]] and its siblings) is verified
+ * against `S3AFileSystem`'s consumers only. With no model of how the vendor filesystem resolves
+ * its configuration, whether native and Spark would agree cannot be decided, so the scan is
+ * declined rather than claimed on a guess. Selected data-file and deletion-vector URIs under an
+ * alias scheme are declined by the generic scheme gates, which never admit an alias (see
+ * [[unsupportedSchemes]]).
+ */
+ private[delta] def s3CompliantAliasSchemeReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val aliases = NativeConfig.resolveS3CompliantSchemes(hadoopConf)
+ if (aliases.isEmpty) {
+ return None
+ }
+ val found = uris
+ .flatMap(uri => Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)))
+ .filter(aliases.contains)
+ .distinct
+ if (found.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support S3-compliant alias filesystem scheme(s) " +
+ s"${found.sorted.mkString(", ")} (${CometConf.COMET_S3_COMPLIANT_SCHEMES_KEY}): " +
+ "Spark reads them through a vendor filesystem whose S3 configuration resolution the " +
+ "native scan's S3AFileSystem divergence model cannot verify")
+ }
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source. The alias
+ * set handed to core's gate is deliberately empty: an `fs.comet.s3Compliant.schemes` alias is
+ * never admitted here (see [[s3CompliantAliasSchemeReason]]), even though core admits it.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri, Set.empty)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason naming the first of `uris` whose path object_store rejects even though it
+ * recognizes the scheme ([[CometScanRule.objectStoreAcceptsPath]], e.g. a directory name
+ * containing a newline, `%0A` in the URI), or `None`. Schemes in `libhdfs` never reach
+ * object_store's path parser and are skipped, as is a `null` scheme (see
+ * [[unsupportedSchemes]]); an S3-compliant alias is declined before this gate runs. The probe
+ * is uncached but is a plain native URL parse with no I/O
+ * ([[CometScanRule.objectStoreAcceptsPath]]), so callers pass every complete selected path
+ * (root paths, data files and deletion vectors) once per distinct URI; a converted table can
+ * carry the rejected character in a file basename. The reason masks any userinfo in the named
+ * URI ([[redactedAuthority]]).
+ */
+ private[delta] def objectStoreRejectedPathReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ uris.distinct
+ .find { uri =>
+ val sch = uri.getScheme
+ sch != null && !libhdfs.contains(sch.toLowerCase(Locale.ROOT)) &&
+ !CometScanRule.objectStoreAcceptsPath(uri)
+ }
+ .map { uri =>
+ // Mask userinfo (see redactedAuthority); the raw path keeps its percent encoding so the
+ // reason shows the rejected sequence as written.
+ val shown =
+ if (uriUserInfo(uri).isEmpty) uri.toString
+ else s"${redactedAuthority(uri)}${Option(uri.getRawPath).getOrElse("")}"
+ s"Native Delta scan cannot open path '$shown': object_store rejects it " +
+ "(e.g. an unsupported character in the path)"
+ }
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ /**
+ * `CommonConfigurationKeysPublic.HADOOP_SECURITY_CREDENTIAL_CLEAR_TEXT_FALLBACK`, default
+ * `true`, verified via `javap` against `hadoop-common` 3.3.4's
+ * `Configuration#getPasswordFromConfig`: `getPassword` only falls back to reading a plaintext
+ * conf value once `getBoolean(<this key>, true)` holds -- with the flag off, a plaintext value
+ * is invisible to every `getPassword`-based resolver, even when no credential provider is
+ * configured at all.
+ */
+ private val ClearTextFallbackKey = "hadoop.security.credential.clear-text-fallback"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * How Hadoop's OWN consumer reads one of the keys compared by [[s3ConfigDivergenceReason]],
+ * which decides how [[s3KeyDivergenceReason]] computes the Hadoop-effective side of its
+ * equality check. Exactly two consumer families exist among [[AllS3ConfigKeys]] in `hadoop-aws`
+ * 3.3.4, each verified via `javap`/CFR against the real call sites (cited per key on
+ * [[S3ConfigKeyConsumers]]). The tier must mirror the key's ACTUAL consumer: resolving a
+ * [[PropagatedOptionConsumer]] key through the wider `lookupPassword` cascade is NOT fail-safe
+ * for a value-EQUALITY comparator -- a long-form alias value Hadoop itself never reads can
+ * EQUAL native's resolution while Hadoop's true propagate-then-plain-get value differs, turning
+ * a real divergence into a wrongly-claimed scan (the endpoint `${...}`-redirect shape pinned in
+ * `DeltaScanContribSuite`).
+ */
+ private[delta] sealed trait S3ConfigConsumer
+
+ /**
+ * Read via `S3AUtils#lookupPassword(bucket, conf, baseKey)`, verified via `javap` against
+ * `hadoop-aws` 3.3.4: builds `longBucketKey = "fs.s3a.bucket." + bucket + "." + baseKey` (the
+ * FULL, already-`fs.s3a`-prefixed base key appended after the bucket segment) and reads it via
+ * `Configuration#getPassword` BEFORE the short-bucket key, keeping the long value whenever
+ * `getPassword` returns non-empty and only falling through to short-then-global otherwise.
+ * `getPassword` is Hadoop-credential-provider-aware and skips plaintext conf entirely when
+ * [[ClearTextFallbackKey]] is false. Modeled by [[hadoopLookupPasswordEffective]].
+ */
+ private[delta] case object LookupPasswordConsumer extends S3ConfigConsumer
+
+ /**
+ * Read via `S3AUtils#propagateBucketOptions` followed by a plain `Configuration#get`-family
+ * call (`getTrimmed`/`getBoolean`/`getClasses`) against the propagated view: the short bucket
+ * form wins only by having overwritten the global key during propagation, the long bucket form
+ * folds into an unread `fs.s3a.fs.s3a.*` key, and neither a credential provider nor
+ * [[ClearTextFallbackKey]] is ever consulted. Modeled as a plain `Configuration#get` on the
+ * [[propagateBucketOptions]] result, which also expands `${...}` references under that
+ * propagated view exactly like the real consumer.
+ */
+ private[delta] case object PropagatedOptionConsumer extends S3ConfigConsumer
+
+ /**
+ * Every `fs.s3a.*` base key that governs whether a claimed native scan actually behaves like
+ * Hadoop's own reader would, paired with the consumer family Hadoop resolves it through -- ONE
+ * list, with each key's resolution tier declared beside it, so a key can never sit in the
+ * comparator without a deliberate classification (adding one without picking a tier does not
+ * compile). The entries are every per-bucket `fs.s3a.*` base key native's S3 client's
+ * `get_config` (s3.rs) resolves, verified directly against its call sites:
+ * `extract_s3_config_options` (endpoint.region, path.style.access, endpoint,
+ * requester.pays.enabled), `lookup_provider_class` (the Comet-specific
+ * credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name).
+ *
+ * Tier assignments, each verified via `javap`/CFR against `hadoop-aws` 3.3.4:
+ * - access.key/secret.key/session.token: `S3AUtils#getAWSAccessKeys` and
+ * `MarshalledCredentialBinding#fromFileSystem` (reached from
+ * `TemporaryAWSCredentialsProvider`) resolve all three via `S3AUtils#lookupPassword` --
+ * [[LookupPasswordConsumer]].
+ * - aws.credentials.provider and assumed.role.credentials.provider:
+ * `S3AUtils#buildAWSProviderList` -> `loadAWSProviderClasses` -> plain
+ * `Configuration#getClasses` -- [[PropagatedOptionConsumer]].
+ * - assumed.role.arn/session.name: `AssumedRoleCredentialProvider`'s constructor reads both
+ * via plain `Configuration#getTrimmed` -- [[PropagatedOptionConsumer]].
+ * - endpoint (`S3AFileSystem`: `getTrimmed`), endpoint.region (`DefaultS3ClientFactory`:
+ * `getTrimmed`), path.style.access (`S3AFileSystem`: `getBoolean`) --
+ * [[PropagatedOptionConsumer]].
+ * - requester.pays.enabled: not read anywhere in `hadoop-aws` 3.3.4 (the constant does not
+ * even exist in its `Constants` class); later releases read it via plain `getBoolean`
+ * against the propagated conf, so the plain tier is both the faithful forward model and
+ * inert on 3.3.4 -- [[PropagatedOptionConsumer]].
+ * - comet.credential.provider.class: Comet's own activation key, plain conf read on both
+ * sides, never a Hadoop key at all -- [[PropagatedOptionConsumer]].
+ *
+ * SYNC NOTE: the key list must stay a superset of native's `NATIVE_S3A_CONFIG_PROPERTIES`
+ * constant (`native/core/src/parquet/objectstore/s3.rs`, property suffixes without the
+ * `fs.s3a.` prefix) -- `DeltaScanContribSuite`'s discovery-harness test asserts this
+ * mechanically against [[AllS3ConfigKeys]]. Literal strings, not the
+ * [[AwsCredentialsProviderKey]] / [[AssumedRoleCredentialsProviderKey]] vals declared below,
+ * purely to avoid a forward reference inside this `object` body; kept textually identical to
+ * those two constants.
+ */
+ private[delta] val S3ConfigKeyConsumers: Seq[(String, S3ConfigConsumer)] = Seq(
Review Comment:
Two settings that change which endpoint native talks to do not appear in this model:
- `fs.s3a.connection.ssl.enabled`: Hadoop prefixes a scheme-less `fs.s3a.endpoint` with `http://` when this is false, while native `normalize_endpoint` (`s3.rs:289-293`) always prefixes `https://`. An on-prem MinIO or Ceph table with `fs.s3a.endpoint=minio:9000` and SSL off claims natively and then fails at execution where Spark reads fine. A zero-I/O decline like the proxy gate would cover it (endpoint has no `://` and the effective flag is false), and this one is MinIO-testable.
- `fs.s3a.assumed.role.sts.endpoint` (and `.sts.endpoint.region`): Hadoop sends AssumeRole to the configured STS endpoint, while native builds `AssumeRoleProvider` with SDK defaults (`s3.rs:879-882`). Same shape as the session-policy gate: decline when it is set.
Does the discovery harness in `DeltaScanContribSuite` catch either of these? It looks like it only flags key names containing key/secret/token/password/encryption, so it would miss both.
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,2096 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::mem::size_of;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::{ParquetAccessPlan, RowGroupAccess};
+use datafusion::execution::memory_pool::{MemoryConsumer, MemoryReservation};
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::path::Path;
+use object_store::{ObjectStore, ObjectStoreExt};
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::{PageIndexPolicy, ParquetMetaData};
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ // A corrupt footer can report a negative row count. `num_rows as u64` would otherwise
+ // wrap it into a huge positive value, silently corrupting every row-group boundary
+ // computed from `group_start`/`group_end` below (and therefore which deleted row indexes
+ // land in which row group) instead of failing loudly.
+ if num_rows < 0 {
+ return Err(GeneralError(format!(
+ "Parquet footer reports a negative row count ({num_rows}) for row group {idx}"
+ )));
+ }
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
+ }
+ cursor = row + 1;
+ }
+ if deleted_in_group == num_rows && num_rows > 0 {
+ plan.skip(idx);
+ } else if deleted_in_group > 0 {
+ if group_end > cursor {
+ selectors.push(RowSelector::select((group_end - cursor) as usize));
+ }
+ plan.scan_selection(idx, RowSelection::from(selectors));
+ }
+ group_start = group_end;
+ }
+ // A deleted index beyond the file's total row count means the DV does not
+ // belong to this file (stale or corrupted metadata); silently dropping it
+ // would under-apply deletions.
+ if let Some(&row) = deleted_iter.peek() {
+ return Err(GeneralError(format!(
+ "Deletion vector marks row {row} but the file only has {group_start} rows"
+ )));
+ }
+ Ok(plan)
+}
+
+/// Verify a decoded deletion vector's row count matches the descriptor's
+/// declared `cardinality`, mirroring Delta's JVM reader
+/// (`StoredBitmap.validateCardinality`). The CRC and framing checks catch
+/// corruption but not a stale, otherwise well-formed bitmap whose row count
+/// no longer matches the descriptor -- that would silently under- or
+/// over-delete rows.
+fn validate_cardinality(
+ file_path: &str,
+ expected: i64,
+ deleted: &RoaringTreemap,
+) -> Result<(), ExecutionError> {
+ let actual = deleted.len();
+ if actual != expected as u64 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has cardinality mismatch: descriptor says {expected}, decoded bitmap has {actual} deleted rows"
+ )));
+ }
+ Ok(())
+}
+
+/// One data file plus everything needed to apply its deletion vector. The
+/// file's size comes from `file.object_meta.size` (built by the planner from
+/// the proto's `file_size`).
+///
+/// `data_store` and `dv_store` are resolved by the caller *before* entering
+/// the async `attach_access_plans` runtime (see its doc comment): building an
+/// object store is sync I/O that, for a cold S3 authority, internally issues
+/// its own `Handle::block_on` calls, which panics if nested inside another
+/// `block_on`. Resolving up front means this module never constructs a
+/// store itself.
+pub struct DvScanFile {
+ pub file: PartitionedFile,
+ /// Full URL of the data file (proto `file_path`).
+ pub file_path: String,
+ pub dv: Option<DeltaSparkDvDescriptor>,
+ /// Object store for `file_path`, pre-resolved by the caller. Only read
+ /// when `dv` is `Some` (files without a deletion vector never open their
+ /// footer here), but every file carries one so the struct's shape
+ /// doesn't depend on whether a deletion vector is present.
+ pub data_store: Arc<dyn ObjectStore>,
+ /// Store and within-store path for an on-disk deletion vector's absolute
+ /// path, pre-resolved by the caller. `None` when the file has no
+ /// deletion vector or the deletion vector is stored inline.
+ pub dv_store: Option<(Arc<dyn ObjectStore>, Path)>,
+}
+
+/// Execution-memory-pool reservation covering one file's expanded DV row selectors across
+/// their *entire* lifetime attached to a scan -- from `build_access_plan`'s construction
+/// through DataFusion 54.1's reader normalizing the attached [`ParquetAccessPlan`]
+/// (`create_initial_plan`'s deep clone plus `into_overall_row_selection`'s combined
+/// `RowSelection`; see [`reader_peak_bytes`]) -- attached to the file's [`PartitionedFile`]
+/// extensions alongside its [`ParquetAccessPlan`]. The reservation's lifetime is tied to the
+/// `PartitionedFile` it is attached to, so it is released back to the pool exactly when the
+/// plan is dropped (query completion or an early-terminated scan), never held open longer.
+/// Newtype-wrapped so it occupies its own slot in the multi-slot, type-keyed `extensions` map
+/// (`datafusion_common::extensions::Extensions`) alongside the plan, rather than a bare
+/// `MemoryReservation` colliding with one some other extension might attach.
+pub struct DvAccessPlanReservation(pub MemoryReservation);
+
+/// Total number of [`RowSelector`]s materialized across `plan`'s per-row-group
+/// selections (`RowGroupAccess::Selection`); `Scan`/`Skip` row groups
+/// contribute none. An alternating deleted/retained bitmap produces one
+/// non-coalescing selector per row (see [`reader_peak_bytes`]'s doc comment
+/// for the worst-case accounting), so this count -- not the deletion
+/// vector's cardinality -- is the thing that must be bounded and reserved
+/// against the execution memory pool.
+fn total_selectors(plan: &ParquetAccessPlan) -> usize {
+ plan.inner()
+ .iter()
+ .map(|access| match access {
+ RowGroupAccess::Selection(selection) => selection.iter().count(),
+ _ => 0,
+ })
+ .sum()
+}
+
+/// Multiplier bounding the peak allocation live *during construction* of one
+/// file's [`RowSelection`]s, relative to the conservative selector-count
+/// bound `S = 2 * cardinality + num_row_groups` (one non-coalescing selector
+/// per deleted row in the worst-case alternating pattern, doubled, plus up to
+/// one extra boundary selector per row group). Split `S` into `r`, the
+/// selectors already retained from row groups `build_access_plan` has
+/// finished, and `c`, the selectors accumulated so far in the current row
+/// group's source `Vec`; `r` and `c` partition the selectors counted toward
+/// `S`, so `r + c <= S` always. While the current group is being built, the
+/// `Vec`'s doubling growth strategy can leave its backing allocation at up to
+/// `2 * c` (the next power-of-two capacity above `c`). Once the group
+/// finishes, `RowSelection::from(Vec)` (parquet's `FromIterator` impl,
+/// `with_capacity` + copy) builds a second, separate `Vec` of size `c` from
+/// that source while the source is still alive, so at the moment the copy
+/// begins, the retained selectors, the current group's doubled source `Vec`,
+/// and the copy are all live simultaneously: `r + 2c + c = r + 3c`. Since
+/// `r >= 0`, `r + 3c <= 3r + 3c = 3(r + c) <= 3S`. 3x covers that peak.
+const CONSTRUCTION_PEAK_FACTOR: usize = 3;
+
+/// Upper bound on how much larger a `Vec`'s backing allocation can be than its element count
+/// after being built by repeated pushes: `std`'s doubling growth strategy never leaves a `Vec`
+/// of `n` elements with a backing allocation larger than the next power of two above `n`, which
+/// is at most `2 * n` for any `n >= 1`.
+const VEC_GROWTH_CAPACITY_FACTOR: usize = 2;
+
+/// `RawVec`'s minimum non-zero capacity for element sizes `<= 1024` bytes ([`RowSelector`] is
+/// 16 bytes on 64-bit platforms: a `usize` row count plus a padded `bool`). Applied once per
+/// row group (or per contiguous run of row groups) a fresh `from_fn`/`FlatMap`-driven `Vec`
+/// gets built for (see [`reader_peak_bytes`]), so even a group or run whose true selector count
+/// is tiny still pays this floor.
+const MIN_VEC_CAPACITY_SELECTORS: usize = 4;
+
+/// Conservative upper bound, in bytes, on the peak allocation live while DataFusion 54.1's
+/// reader normalizes one file's attached [`ParquetAccessPlan`] -- the allocation this module's
+/// steady-state reservation must cover, not merely the plan's own retained selector bytes.
+/// THREE allocations can be live simultaneously by the time `into_overall_row_selection`
+/// returns, not two -- the clone is only exact when page-index pruning never touches it:
+///
+/// 1. **Attached original** (`selectors`, exact): `create_initial_plan` deep-clones the
+/// attached plan while the original remains reachable from the file's `extensions` until
+/// the scan consumes it. The ORIGINAL's own selector `Vec`s are exact -- a coalesced
+/// [`RowSelection`] built via `RowSelection::from(Vec<RowSelector>)` (what
+/// `build_access_plan` uses) has no excess capacity, because that conversion is a plain
+/// `with_capacity(len)` copy, not a `size_hint`-blind fold.
+/// 2. **The clone, possibly capacity-inflated** (`<= VEC_GROWTH_CAPACITY_FACTOR * selectors +
+/// MIN_VEC_CAPACITY_SELECTORS * num_row_groups`): if page-index pruning fires
+/// (`PagePruningAccessPlanFilter`; `access_plan.rs`'s `scan_selection` on a row group that
+/// already carries a `RowGroupAccess::Selection` calls `existing.intersection(&page_derived)`
+/// -- `RowSelection::intersection` -> `intersect_row_selections`), it replaces the CLONE's
+/// per-row-group selection with that intersection's output. `intersect_row_selections` is
+/// ANOTHER `from_fn` generator with `size_hint() == (0, None)`, so each intersected row
+/// group's backing `Vec` starts at `with_capacity(0)` and doubles as it grows, independent
+/// of whatever capacity the pre-intersection selection had. This inflated clone is still
+/// live when `into_overall_row_selection` later moves its buffer. Term 1's exactness
+/// guarantee holds for the ORIGINAL always, and for the clone only when page-index pruning
+/// never fires against it -- once it does, the clone must be charged at the SAME
+/// growth-capped bound as a fresh combined-selection `Vec` (term 3), summed once per row
+/// group rather than once per run, since each row group's `Selection` is intersected
+/// independently.
+/// 3. **Per-run combined-selection allocation** (`<= VEC_GROWTH_CAPACITY_FACTOR * (selectors +
+/// num_row_groups) + MIN_VEC_CAPACITY_SELECTORS * num_row_groups`): `into_overall_row_selection`
+/// collects each contiguous run of row groups' selectors into a *new* `RowSelection` via a
+/// `FlatMap` whose `size_hint().0 == 0`, so that run's `Vec` starts at `with_capacity(0)`
+/// and doubles as it grows -- capping its backing allocation at
+/// `max(MIN_VEC_CAPACITY_SELECTORS, next_power_of_two(len))`, which is at most
+/// `MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR * len` for a run of `len`
+/// selectors. `len` is at most that run's share of `selectors` plus one boundary selector
+/// per `RowGroupAccess::Scan` row group in the run (`Scan` always contributes exactly one
+/// `RowSelector::select(num_rows)`; see `access_plan.rs`'s `into_overall_row_selection`).
+/// Summing across at most `num_row_groups` runs (each spans >= 1 row group) bounds the total
+/// at `VEC_GROWTH_CAPACITY_FACTOR * selectors + (MIN_VEC_CAPACITY_SELECTORS +
+/// VEC_GROWTH_CAPACITY_FACTOR) * num_row_groups`.
+///
+/// Summing all three terms and converting to bytes: `((1 + 2 * VEC_GROWTH_CAPACITY_FACTOR) *
+/// selectors + (2 * MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR) * num_row_groups)
+/// * size_of::<RowSelector>()` -- with the constants above, `(5 * selectors + 10 *
+/// num_row_groups) * size_of::<RowSelector>()`. Checked against two measured worst cases:
+///
+/// - No page-index pruning (the original P2 report; term 2 stays exact): one 2,000,000-row
+/// group, 1,000,000 alternating deletions, `selectors = 2,000,000`. Measured allocator peak
+/// 97,554,457 B; the byte-for-byte accounting for the attached original plus the (here,
+/// exact) clone plus the inflated combined selection explains 97,554,432 B of that, a 25 B
+/// residue we did not attribute. This bound gives 160,000,160 B -- much looser here because
+/// it must also cover the next case, where the clone is NOT exact.
+/// - Page-index pruning fires against the clone: one 1,048,577-row group, `selectors =
+/// 1,048,577`. Measured peak 83,886,096 B; this bound gives 83,886,320 B (a 224 B, <1%
+/// margin -- deliberately tight, since this is the case that drives the bound).
+///
+/// Uses checked arithmetic throughout: a selector or row-group count large enough to overflow
+/// `usize` indicates a corrupted or malicious input, reported as a clean error rather than
+/// panicking.
+fn reader_peak_bytes(selectors: usize, num_row_groups: usize) -> Result<usize, ExecutionError> {
+ let overflow = || {
+ GeneralError(format!(
+ "Deletion vector reader-peak bound overflowed for {selectors} selectors and \
+ {num_row_groups} row groups"
+ ))
+ };
+ // Term 1: the attached original -- exact, untouched by page-index pruning (only the clone
+ // is ever intersected; see the doc comment above).
+ let attached_term = selectors;
+ // Term 2: the clone, bounded as if page-index pruning DID fire against every row group
+ // (safe even when it doesn't: term 2's bound is always >= `selectors`, so it never
+ // undershoots the exact case either).
+ let clone_growth = selectors
+ .checked_mul(VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let clone_floor = num_row_groups
+ .checked_mul(MIN_VEC_CAPACITY_SELECTORS)
+ .ok_or_else(overflow)?;
+ let clone_term = clone_growth.checked_add(clone_floor).ok_or_else(overflow)?;
+ // Term 3: into_overall_row_selection's per-run combined-selection allocation.
+ let combined_growth = selectors
+ .checked_mul(VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let combined_floor = num_row_groups
+ .checked_mul(MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let combined_term = combined_growth
+ .checked_add(combined_floor)
+ .ok_or_else(overflow)?;
+
+ let selector_bound = attached_term
+ .checked_add(clone_term)
+ .and_then(|sum| sum.checked_add(combined_term))
+ .ok_or_else(overflow)?;
+ selector_bound
+ .checked_mul(size_of::<RowSelector>())
+ .ok_or_else(overflow)
+}
+
+/// Upper bound, in [`RowSelector`]s, on how many extra selectors the parquet reader's
+/// page-index pruning can add on top of the deletion vector's own selection when normalizing
+/// one file, from that file's already-fetched [`ParquetMetaData`].
+///
+/// `intersect_row_selections` (parquet's `selection.rs`), which combines a page-pruning
+/// selection with the deletion vector's selection, is a `from_fn` generator whose
+/// `size_hint()` is `(0, None)`: for inputs of length `a` and `b`, its output can have up to
+/// `a + b` selectors -- longer than either input. Bounding the page-pruning side of that sum
+/// requires knowing how many selectors a page-index-derived selection could produce: at most
+/// two per data page (one skip, one select, in the worst case of alternating page-level
+/// pruning decisions), summed over every column of every row group.
+///
+/// Returns `0` when `metadata` carries no offset index (`metadata.offset_index()` is `None`).
+/// This is provably safe, not merely a convenient default: page-index pruning cannot produce a
+/// page-level selection without the offset index to locate pages by, so there are no
+/// page-pruning selectors to bound. The offset index is fetched with
+/// `PageIndexPolicy::Optional` from the same `FileMetadataCache` entry the scan's reader later
+/// reopens (see [`attach_access_plan`]'s footer-fetch comment), so this function observes
+/// exactly what the reader will see.
+///
+/// Uses checked arithmetic throughout for the same reason as [`admission_bound_bytes`].
+fn page_selection_bound_selectors(metadata: &ParquetMetaData) -> Result<usize, ExecutionError> {
+ let Some(offset_index) = metadata.offset_index() else {
+ return Ok(0);
+ };
+ let overflow = || {
+ GeneralError(
+ "Deletion vector page-selection bound overflowed while summing offset-index page \
+ locations"
+ .to_string(),
+ )
+ };
+ let mut total_page_locations = 0usize;
+ for row_group in offset_index {
+ for column in row_group {
+ total_page_locations = total_page_locations
+ .checked_add(column.page_locations().len())
+ .ok_or_else(overflow)?;
+ }
+ }
+ total_page_locations.checked_mul(2).ok_or_else(overflow)
+}
+
+/// Execution-memory-pool admission bound, in bytes, for one file's deletion-vector access
+/// plan -- reserved *before* calling `build_access_plan` (see [`attach_access_plan`]'s
+/// pre-reserve call site) to cover the larger of two peaks live at different points in the
+/// plan's lifetime. In practice the reader-normalization peak below dominates the construction
+/// peak unconditionally for any non-trivial input (`reader_peak_bytes(S, G) = (5S + 10G) *
+/// size_of::<RowSelector>()` always exceeds `CONSTRUCTION_PEAK_FACTOR * S *
+/// size_of::<RowSelector>() = 3S * size_of::<RowSelector>()` once `S >= 1`, since the `5S` term
+/// alone already exceeds `3S`); the construction term is retained as a documented floor rather
+/// than dropped, since it is cheap to compute and keeps this bound correct even if the reader's
+/// growth factors ever shrink below construction's.
+///
+/// - **Construction peak** (`CONSTRUCTION_PEAK_FACTOR * S`, see that constant's doc comment):
+/// live while `build_access_plan` builds the plan's `RowSelection`s. Construction's
+/// transient allocations fully unwind before `build_access_plan` returns, so this peak never
+/// overlaps the reader-normalization peak below.
+/// - **Reader-normalization peak** (`reader_peak_bytes(S + page_bound_selectors,
+/// num_row_groups)`, see that function): live later, once DataFusion's reader normalizes the
+/// attached plan. `S = 2 * cardinality + num_row_groups` is the same conservative bound on
+/// the plan's final retained selector count used for the construction peak -- it provably
+/// bounds `R = total_selectors(&plan)` (`R <= S`, from `build_access_plan`'s
+/// one-non-coalescing-selector-per-deleted-row worst case plus one boundary selector per row
+/// group), so `S + page_bound_selectors` bounds `R` after page-index inflation the same way
+/// `S` bounds `R` before it.
+///
+/// These two peaks never overlap in time, so `max` -- not `sum` -- is the correct combinator:
+/// reserving their sum would over-reserve for no safety benefit.
+///
+/// Deliberately not clamped by the file's total row count here, unlike the reader-peak target
+/// `attach_access_plan` resizes down to after construction (see that call site): `S`'s
+/// `+ num_row_groups` boundary term is a worst-case padding margin that can legitimately exceed
+/// the total row count for a small, heavily-deleted file, and admission sizing has no actual
+/// retained-selector count yet to clamp against -- only after construction, once `R` is known,
+/// is clamping to the total row count both meaningful and strictly tighter. Leaving this bound
+/// unclamped only ever makes admission more conservative, never less safe.
+///
+/// Uses checked arithmetic throughout: a cardinality, row-group count, or page bound large
+/// enough to overflow `usize` while computing this bound indicates a corrupted or malicious
+/// descriptor, reported as a clean error rather than panicking.
+fn admission_bound_bytes(
+ cardinality: i64,
+ num_row_groups: usize,
+ page_bound_selectors: usize,
+) -> Result<usize, ExecutionError> {
+ let overflow = || {
+ GeneralError(format!(
+ "Deletion vector admission bound overflowed for cardinality {cardinality}, \
+ {num_row_groups} row groups, and page bound {page_bound_selectors} selectors"
+ ))
+ };
+ let cardinality_usize = usize::try_from(cardinality).map_err(|_| overflow())?;
+ // S: the conservative bound on the plan's final *retained* selector count (what
+ // `total_selectors(&plan)` cannot exceed) -- unchanged from the pre-existing
+ // construction-only bound this function replaces.
+ let s = cardinality_usize
+ .checked_mul(2)
+ .and_then(|doubled| doubled.checked_add(num_row_groups))
+ .ok_or_else(overflow)?;
+
+ let construction_bytes = s
+ .checked_mul(size_of::<RowSelector>())
+ .and_then(|bytes| bytes.checked_mul(CONSTRUCTION_PEAK_FACTOR))
+ .ok_or_else(overflow)?;
+
+ let s_plus_page = s.checked_add(page_bound_selectors).ok_or_else(overflow)?;
+ let reader_bytes = reader_peak_bytes(s_plus_page, num_row_groups)?;
+
+ Ok(construction_bytes.max(reader_bytes))
+}
+
+/// Upper bound on concurrent DV-blob and footer fetches per partition. Both
+/// are small ranged reads, so a modest fan-out hides object-store latency
+/// without flooding the store client.
+const DV_FETCH_CONCURRENCY: usize = 8;
+
+/// Called via `block_on` at plan-creation time on the executor task: DV blobs
+/// are small ranged reads and footers are needed to learn row-group
+/// boundaries. Files are fetched concurrently (bounded by
+/// [`DV_FETCH_CONCURRENCY`]) with input order preserved. Footer fetches go
+/// through the scan's shared FileMetadataCache, so the scan's subsequent open
+/// of the same file is served from cache. That reuse relies on each input
+/// [`PartitionedFile`] being returned as-is (only `with_extension` applied),
+/// never rebuilt: the cache entry is keyed by this exact `object_meta` and the
+/// scan later looks it up through the same struct.
+///
+/// Deliberately takes no object-store options map and imports no
+/// store-construction helper: every [`DvScanFile`] arrives with its stores
+/// already resolved by the caller (see its doc comment), so this async path
+/// structurally cannot build an object store -- only `runtime_env` is still
+/// threaded through, for the shared `FileMetadataCache` and (per file) the
+/// execution `MemoryPool` each expanded access plan's row selectors are
+/// reserved against -- see [`DvAccessPlanReservation`].
+pub async fn attach_access_plans(
+ runtime_env: Arc<RuntimeEnv>,
+ files: Vec<DvScanFile>,
+) -> Result<Vec<PartitionedFile>, ExecutionError> {
+ futures::stream::iter(files)
+ .map(|scan_file| attach_access_plan(Arc::clone(&runtime_env), scan_file))
+ .buffered(DV_FETCH_CONCURRENCY)
+ .try_collect()
+ .await
+}
+
+/// Resolve one file's deletion vector into an attached [`ParquetAccessPlan`];
+/// files without a DV pass through untouched.
+async fn attach_access_plan(
+ runtime_env: Arc<RuntimeEnv>,
+ scan_file: DvScanFile,
+) -> Result<PartitionedFile, ExecutionError> {
+ let DvScanFile {
+ file,
+ file_path,
+ dv,
+ data_store,
+ dv_store,
+ } = scan_file;
+ let dv = match dv {
+ Some(dv) => dv,
+ None => return Ok(file),
+ };
+ // Delta's canonical `DeletionVectorDescriptor.EMPTY`: inline storage, empty
+ // payload, size 0, cardinality 0. Spark's reader returns all rows for it;
+ // decoding would fail (the empty payload is too short for a magic
+ // number), so pass the file through unchanged before attempting to read it.
+ if dv.cardinality == 0 && dv.size_in_bytes == 0 {
+ return Ok(file);
+ }
+ if dv.size_in_bytes < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative size {}",
+ dv.size_in_bytes
+ )));
+ }
+ if dv.cardinality < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative cardinality {}",
+ dv.cardinality
+ )));
+ }
+
+ let data: Vec<u8> = if let Some(inline) = dv.inline_data {
Review Comment:
The on-disk branch verifies the payload against `size_in_bytes` in `unframe_dv_blob`, but the inline branch takes `inline_data` as-is, so an inline payload whose length disagrees with the descriptor decodes silently. Since the JVM does the z85 decode, this is the only native check point for inline DVs. Could it compare the length before `deserialize_dv_bitmap`?
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,2096 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::mem::size_of;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::{ParquetAccessPlan, RowGroupAccess};
+use datafusion::execution::memory_pool::{MemoryConsumer, MemoryReservation};
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::path::Path;
+use object_store::{ObjectStore, ObjectStoreExt};
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::{PageIndexPolicy, ParquetMetaData};
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ // A corrupt footer can report a negative row count. `num_rows as u64` would otherwise
+ // wrap it into a huge positive value, silently corrupting every row-group boundary
+ // computed from `group_start`/`group_end` below (and therefore which deleted row indexes
+ // land in which row group) instead of failing loudly.
+ if num_rows < 0 {
+ return Err(GeneralError(format!(
+ "Parquet footer reports a negative row count ({num_rows}) for row group {idx}"
+ )));
+ }
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
Review Comment:
This is the one unchecked add in an otherwise fully checked path; a corrupt footer with two row groups near `i64::MAX` panics here in debug and wraps in release, which then misfires the "beyond total rows" check below. `checked_add` with the existing error style would match the rest.
On tests: the on-disk fixture writes a single blob at offset 1, and the access-plan test deletes rows in the middle of a group. Two DVs in one on-disk file (exercising the `offset..offset+framed_len` slicing) and a deleted row on a row-group boundary (last row of group k and first row of k+1) would cover paths that are currently untested.
##########
native/core/Cargo.toml:
##########
@@ -103,12 +106,25 @@ datafusion-functions-nested = { version = "55.1.0" }
[features]
backtrace = ["datafusion/backtrace"]
-default = ["hdfs-opendal"]
+default = ["hdfs-opendal", "delta"]
hdfs-opendal = ["opendal", "object_store_opendal", "hdfs-sys"]
jemalloc = ["tikv-jemallocator", "tikv-jemalloc-ctl"]
-# Delta Lake integration. When enabled, links the `comet-contrib-delta` crate
-# into `libcomet` and activates the `OpStruct::DeltaScan` dispatcher arm.
-# Default builds carry zero Delta surface.
+# Native Delta Lake scan support for the JVM-planned path (contrib/delta-spark).
+# In the default set: inert at runtime unless the contrib jar is on the
+# classpath (ServiceLoader) AND spark.comet.scan.delta.enabled is set, so it
+# cannot affect non-Delta scans. Opt out with --no-default-features for slim
Review Comment:
Could this state the rationale directly rather than pointing at the PR review? Something like: inert without both the contrib jar and the config, and no new crates since `roaring` and `crc32fast` are already in the tree. The git history already records the discussion. The footprint figure here (82 KB) also differs from the one in `dev/verify-contrib-delta-gate.sh` (84 KB); one number in one place would be enough.
##########
native/core/src/parquet/datetime_rebase.rs:
##########
@@ -0,0 +1,2662 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Per-file datetime calendar-rebase handling for the parquet scan.
+//!
+//! Spark 2.4 and earlier wrote dates and timestamps in the hybrid Julian + Gregorian calendar;
+//! Spark 3.0+ uses the proleptic Gregorian calendar and records the calendar policy of every
+//! file it writes in the parquet footer's key-value metadata (`org.apache.spark.version`,
+//! `org.apache.spark.legacyDateTime`, `org.apache.spark.legacyINT96`,
+//! `org.apache.spark.timeZone`). Spark's reader resolves the rebase policy from EACH FILE's
+//! writer metadata (`DataSourceUtils.datetimeRebaseSpec` / `int96RebaseSpec`) -- the session's
+//! `spark.sql.parquet.datetimeRebaseModeInRead` conf only applies to files whose metadata does
+//! not decide the policy on its own -- so a reader that ignores the metadata silently returns
+//! values shifted by up to ten days for dates before 1582-10-15 (e.g. `1500-01-01` reads as
+//! `1500-01-10`).
+//!
+//! This module mirrors that per-file resolution: [`resolve_file_rebase_policies`] computes the
+//! date / INT64-timestamp / INT96-timestamp policies from a file's arrow schema metadata (the
+//! parquet key-value pairs survive the parquet -> arrow schema conversion), and
+//! [`wrap_datetime_rebase`] wraps the per-file rewritten expressions' column references in a
+//! [`SparkDatetimeRebaseExpr`] that rebases values exactly where that is possible without the
+//! JVM's historical timezone tables (dates always; timestamps for a fixed UTC writer zone) and
+//! refuses -- rather than silently corrupting -- ancient values it cannot rebase. Nested
+//! columns are rebuilt leaf by leaf (struct / list / map / fixed-size list / dictionary), each
+//! leaf under its own policy, with nulls and offsets preserved. Modern values are always the
+//! identity under every policy: from 1582-10-15 onward for dates, and from
+//! [`LAST_SWITCH_JULIAN_TS_SECONDS`] (1900-01-01T00:00:00Z, Spark's
+//! `RebaseDateTime.lastSwitchJulianTs`) onward for timestamps.
+//!
+//! Spark applies `datetimeRebaseSpec` to INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns
+//! and `int96RebaseSpec` to INT96 columns. The two physical types are indistinguishable in the
+//! arrow schema DataFusion hands the expression adapter (both surface as `Timestamp(us, "UTC")`
+//! after INT96 coercion), so Comet's parquet reader factory stamps the file's INT96 leaf
+//! ordinals -- taken from the parquet footer's own `SchemaDescriptor` -- into the key-value
+//! metadata under [`INT96_LEAVES_METADATA_KEY`] before the arrow schema is derived (see
+//! [`stamp_int96_leaves`] and `eager_page_index_reader_factory.rs`), and the adapter attributes
+//! every timestamp leaf to its spec from that stamp. Without a stamp, the two specs are merged:
+//! agreement decides, disagreement degrades to [`RebasePolicy::CheckAncient`].
+//!
+//! The wrapper sits BENEATH the schema adapter's nested narrowing (the struct -> struct convert
+//! that keeps only the requested children), which is what keeps those ordinals physical -- but
+//! it means the wrapper sees every physical child, requested or not. Spark only ever decodes
+//! the requested nested schema, so [`FileRebasePolicies::restrict_to_requested`] marks the
+//! physical leaves the narrowing drops as the identity: an unrequested ancient `s.ts` never
+//! blocks `select s.d`, exactly as in Spark.
+//!
+//! Currently only enabled by the Delta scan arms via
+//! `SparkParquetOptions::rebase_from_file_metadata`, which also carries the session read modes
+//! ([`SessionRebaseModes`], forwarded from the JVM) that decide the policy for files without
+//! Spark writer metadata; the plain NativeScan keeps its documented no-rebase behavior (see
+//! the compatibility guide and issue #5010).
+
+use std::collections::HashMap;
+use std::fmt::{self, Display};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Date32Array, FixedSizeListArray, GenericListArray, MapArray,
+ OffsetSizeTrait, PrimitiveArray, RecordBatch, StructArray,
+};
+use arrow::datatypes::{
+ ArrowTimestampType, DataType, Date32Type, FieldRef, Schema, SchemaRef, TimeUnit,
+ TimestampMicrosecondType, TimestampMillisecondType, TimestampNanosecondType,
+ TimestampSecondType,
+};
+use arrow::error::ArrowError;
+use datafusion::common::tree_node::{Transformed, TreeNode};
+use datafusion::common::{DataFusionError, Result as DataFusionResult};
+use datafusion::physical_expr::expressions::Column;
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::ColumnarValue;
+use parquet::basic::Type as ParquetPhysicalType;
+use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaData};
+use parquet::schema::types::SchemaDescriptor;
+
+use super::name_fold::fold_names;
+use super::schema_adapter::parse_field_id;
+
+/// Footer key naming the Spark release that wrote the file; absent for non-Spark writers.
+const SPARK_VERSION_METADATA_KEY: &str = "org.apache.spark.version";
+/// Present (empty value) when the file's dates and INT64 timestamps were written with
+/// `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_DATETIME_KEY: &str = "org.apache.spark.legacyDateTime";
+/// Present (empty value) when the file's INT96 timestamps were written with
+/// `spark.sql.parquet.int96RebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_INT96_KEY: &str = "org.apache.spark.legacyINT96";
+/// The writer session's time zone, stamped alongside either legacy flag.
+const SPARK_TIMEZONE_KEY: &str = "org.apache.spark.timeZone";
+
+/// Key-value metadata entry Comet's parquet reader factory adds to a file's footer metadata
+/// (in memory only, never written back) so the expression adapter can tell INT96 timestamp
+/// columns from INT64 ones after both have been coerced to the same arrow type. Value:
+/// `"<leaf count>:<comma-separated INT96 leaf ordinals>"`, where leaves are the file's
+/// primitive columns in `SchemaDescriptor::columns()` order -- the same depth-first order
+/// parquet-rs assigns arrow leaves, so an arrow-side depth-first walk lines up with it. The
+/// leaf count lets the reader detect a stamp that does not describe the schema it is paired
+/// with (see [`Int96Attribution::from_schema`]).
+pub(crate) const INT96_LEAVES_METADATA_KEY: &str = "comet.int96_leaf_columns";
+
+/// Day of the Gregorian cutover (1582-10-15) as days since the epoch; rebasing is the identity
+/// from this day onward. Same value as Spark's `RebaseDateTime.lastSwitchJulianDay`.
+const LAST_SWITCH_JULIAN_DAY: i32 = -141427;
+
+/// Spark's `RebaseDateTime.lastSwitchJulianTs` (and `lastSwitchGregorianTs`) in seconds since
+/// the epoch: 1900-01-01T00:00:00Z. Spark derives it as the latest switch instant across every
+/// zone in its `julian-gregorian-rebase-micros.json` table (`getLastSwitchTs`, which also
+/// asserts the calendars' difference is zero for every zone from then on): most zones ran on
+/// local mean time before 1900, so the last instant at which rebasing changes a value in ANY
+/// zone is 1900-01-01T00:00:00Z, not the 1582 cutover. `createTimestampRebaseFuncInRead`
+/// under `EXCEPTION` throws exactly for `micros < lastSwitchJulianTs` (after converting
+/// `TIMESTAMP_MILLIS` to micros), and `rebaseJulianToGregorianMicros` is the identity from it
+/// onward in every zone. The value is in seconds so it scales exactly to any timestamp unit.
+pub(crate) const LAST_SWITCH_JULIAN_TS_SECONDS: i64 = -2_208_988_800;
+
+/// The per-century differences between the Julian and proleptic Gregorian calendars, and the
+/// Julian-calendar switch days at which each difference starts to apply. Copied verbatim from
+/// Spark's `RebaseDateTime.julianGregDiffs` / `julianGregDiffSwitchDay` (which Spark generated
+/// from `localRebaseJulianToGregorianDays`); `rebase_julian_to_gregorian_days` must stay
+/// value-for-value equal to Spark's `rebaseJulianToGregorianDays`.
+const JULIAN_GREG_DIFFS: [i32; 14] = [2, 1, 0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, 0];
+const JULIAN_GREG_DIFF_SWITCH_DAY: [i32; 14] = [
+ -719164, -682945, -646420, -609895, -536845, -500320, -463795, -390745, -354220, -317695,
+ -244645, -208120, -171595, -141427,
+];
+
+/// Proleptic-Gregorian days since 1970-01-01 for a nominal civil date, via Howard Hinnant's
+/// `days_from_civil`. `d` may exceed the month's length; the excess rolls into the following
+/// month exactly like `LocalDate.of(y, m, 1).plusDays(d - 1)` in Spark's
+/// `localRebaseJulianToGregorianDays` (how the non-existent proleptic date `1000-02-29`,
+/// valid in the Julian calendar, lands on `1000-03-01`).
+fn days_from_civil(y: i64, m: i64, d: i64) -> i64 {
+ let y = if m <= 2 { y - 1 } else { y };
+ let era = y.div_euclid(400);
+ let yoe = y - era * 400; // [0, 399]
+ let mp = (m + 9) % 12; // [0, 11], March = 0
+ let doy = (153 * mp + 2) / 5 + d - 1;
+ let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
+ era * 146097 + doe - 719468
+}
+
+/// Julian-calendar civil date `(year, month, day)` for a day count since 1970-01-01 that labels
+/// days in the Julian calendar (astronomical year numbering: 1 BCE is year 0). Standard
+/// Julian-day-number conversion (E.G. Richards' algorithm), exact for any day.
+fn julian_day_to_civil(days: i64) -> (i64, i64, i64) {
+ // Integer (noon) Julian Day Number of this civil day: 1970-01-01 is JDN 2440588.
+ let jdn = days + 2_440_588;
+ let f = jdn + 1401;
+ let e = 4 * f + 3;
+ let g = e.rem_euclid(1461) / 4;
+ let h = 5 * g + 2;
+ let day = h.rem_euclid(153) / 5 + 1;
+ let month = (h / 153 + 2).rem_euclid(12) + 1;
+ let year = e.div_euclid(1461) - 4716 + (14 - month) / 12;
+ (year, month, day)
+}
+
+/// Exact port of Spark's `RebaseDateTime.rebaseJulianToGregorianDays`: reinterprets a day count
+/// written in the hybrid Julian + Gregorian calendar as the proleptic Gregorian day count of the
+/// same nominal civil date. Identity for days from 1582-10-15 onward. Days before the tables'
+/// range (before Julian `0001-01-01`) take the calendar-arithmetic path, mirroring Spark's
+/// `localRebaseJulianToGregorianDays` fallback.
+pub(crate) fn rebase_julian_to_gregorian_days(days: i32) -> i32 {
+ if days < JULIAN_GREG_DIFF_SWITCH_DAY[0] {
+ let (y, m, d) = julian_day_to_civil(days as i64);
+ (days_from_civil(y, m, 1) + (d - 1)) as i32
+ } else {
+ // Spark's rebaseDays: linear search from the most recent switch day.
+ let mut i = JULIAN_GREG_DIFF_SWITCH_DAY.len();
+ loop {
+ i -= 1;
+ if i == 0 || days >= JULIAN_GREG_DIFF_SWITCH_DAY[i] {
+ break;
+ }
+ }
+ days + JULIAN_GREG_DIFFS[i]
+ }
+}
+
+/// Timezone strings from `org.apache.spark.timeZone` that denote a fixed zero-offset zone in
+/// both `java.util.TimeZone` and `java.time`. Only for these is timestamp rebasing the pure
+/// nominal-date shift [`SparkDatetimeRebaseExpr::rebase_timestamp_utc`] computes; any other (or
+/// absent) zone needs the JVM's historical timezone tables and stays on the
+/// refuse-ancient-values path.
+const UTC_EQUIVALENT_TIMEZONES: [&str; 6] = ["UTC", "Etc/UTC", "GMT", "Etc/GMT", "Z", "+00:00"];
+
+/// How the writer's session time zone (if recorded) affects timestamp rebasing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum WriterTimeZone {
+ /// A fixed zero-offset zone: rebasing reduces to the exact nominal-date shift.
+ Utc,
+ /// Any other zone, or none recorded (pre-3.0 files): ancient values cannot be rebased
+ /// without the JVM's historical timezone data.
+ OtherOrUnknown,
+}
+
+/// One session-level datetime rebase read mode (a `LegacyBehaviorPolicy` value of
+/// `spark.sql.parquet.datetimeRebaseModeInRead` / `int96RebaseModeInRead`), consulted by
+/// [`resolve_file_rebase_policies`] ONLY for files whose footer metadata does not decide the
+/// policy on its own -- exactly the `getOrElse` fallback in Spark's
+/// `DataSourceUtils.getRebaseSpec`. Files that carry `org.apache.spark.version` ignore these
+/// modes entirely, on every Spark version.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) enum RebaseReadMode {
+ /// Refuse ancient values (Spark raises `SparkUpgradeException`); maps to
+ /// [`RebasePolicy::CheckAncient`]. The default mirrors the conservative posture used
+ /// before the conf was plumbed through (and Spark 3.x's own conf default).
+ #[default]
+ Exception,
+ /// Read values as proleptic Gregorian without rebasing.
+ Corrected,
+ /// Rebase from the hybrid Julian + Gregorian calendar.
+ Legacy,
+}
+
+impl RebaseReadMode {
+ /// Parses a `LegacyBehaviorPolicy` conf value. `SQLConf` validates and upper-cases the
+ /// session conf, but a per-relation `datetimeRebaseMode` option arrives verbatim, so the
+ /// match is case-insensitive. Anything unrecognized -- including the empty string a proto
+ /// producer that predates the field sends -- falls back to [`RebaseReadMode::Exception`],
+ /// which refuses ancient values rather than silently corrupting them.
+ pub(crate) fn from_conf_value(value: &str) -> Self {
+ match value.to_ascii_uppercase().as_str() {
+ "CORRECTED" => RebaseReadMode::Corrected,
+ "LEGACY" => RebaseReadMode::Legacy,
+ _ => RebaseReadMode::Exception,
+ }
+ }
+}
+
+/// The session's effective datetime rebase read modes, one per spec class (INT64
+/// dates/timestamps vs INT96 timestamps), forwarded from the JVM at planning time.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) struct SessionRebaseModes {
+ /// `spark.sql.parquet.datetimeRebaseModeInRead` (or the relation's `datetimeRebaseMode`).
+ pub datetime: RebaseReadMode,
+ /// `spark.sql.parquet.int96RebaseModeInRead` (or the relation's `int96RebaseMode`).
+ pub int96: RebaseReadMode,
+}
+
+/// Calendar policy of one file's date or timestamp columns, resolved from writer metadata the
+/// same way Spark's `DataSourceUtils.getRebaseSpec` resolves it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum RebasePolicy {
+ /// Written in the proleptic Gregorian calendar; values pass through untouched.
+ Corrected,
+ /// Written in the hybrid Julian + Gregorian calendar; values must be rebased.
+ Legacy(WriterTimeZone),
+ /// Policy could not be pinned down (contradictory flags, or a non-Spark writer under the
+ /// `EXCEPTION` read mode): modern values -- identical under either calendar -- pass,
+ /// ancient values raise. Mirrors Spark's `EXCEPTION` behavior (`SparkUpgradeException`).
+ CheckAncient,
+}
+
+/// Which of a file's leaf columns are physically INT96, from the stamp the parquet reader
+/// factory adds under [`INT96_LEAVES_METADATA_KEY`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) enum Int96Attribution {
+ /// No stamp, or a stamp whose leaf count does not match the schema it arrived with: the
+ /// INT64 and INT96 timestamp specs cannot be told apart per column and are merged.
+ Unknown,
+ /// Sorted leaf ordinals (depth-first over the file schema's primitive columns) that are
+ /// INT96; every other timestamp leaf is INT64.
+ Known(Vec<usize>),
+}
+
+impl Int96Attribution {
+ /// Parses the stamp out of `schema`'s metadata and validates its leaf count against the
+ /// schema's own depth-first leaf count, so a stamp that does not describe this schema (a
+ /// crafted footer key, or a cached-metadata mismatch) degrades to [`Self::Unknown`].
+ fn from_schema(schema: &Schema) -> Self {
+ let Some(stamp) = schema.metadata().get(INT96_LEAVES_METADATA_KEY) else {
+ return Int96Attribution::Unknown;
+ };
+ let Some((count, ordinals)) = stamp.split_once(':') else {
+ return Int96Attribution::Unknown;
+ };
+ let schema_leaves: usize = schema
+ .fields()
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ if count.parse::<usize>().ok() != Some(schema_leaves) {
+ return Int96Attribution::Unknown;
+ }
+ let parsed: Option<Vec<usize>> = if ordinals.is_empty() {
+ Some(Vec::new())
+ } else {
+ ordinals
+ .split(',')
+ .map(|o| o.parse::<usize>().ok().filter(|o| *o < schema_leaves))
+ .collect()
+ };
+ match parsed {
+ Some(mut leaves) => {
+ leaves.sort_unstable();
+ Int96Attribution::Known(leaves)
+ }
+ None => Int96Attribution::Unknown,
+ }
+ }
+
+ /// `Some(true)` / `Some(false)` when the leaf is known to be INT96 / INT64, `None` when
+ /// the attribution is unknown.
+ fn is_int96(&self, leaf: usize) -> Option<bool> {
+ match self {
+ Int96Attribution::Unknown => None,
+ Int96Attribution::Known(leaves) => Some(leaves.binary_search(&leaf).is_ok()),
+ }
+ }
+}
+
+/// The [`INT96_LEAVES_METADATA_KEY`] value describing `schema`: its leaf count and the
+/// ordinals of its INT96 primitive columns.
+pub(crate) fn int96_leaf_stamp(schema: &SchemaDescriptor) -> String {
+ let ordinals: Vec<String> = schema
+ .columns()
+ .iter()
+ .enumerate()
+ .filter(|(_, column)| column.physical_type() == ParquetPhysicalType::INT96)
+ .map(|(ordinal, _)| ordinal.to_string())
+ .collect();
+ format!("{}:{}", schema.num_columns(), ordinals.join(","))
+}
+
+/// Returns a copy of `metadata` whose key-value metadata carries the [`int96_leaf_stamp`] of
+/// its own schema, or `None` when it already does (the common case after the first open of a
+/// file, since the caller caches the stamped copy). Any pre-existing entry under the key --
+/// a file cannot legitimately carry one -- is replaced, never trusted. Only the file-level
+/// key-value list changes; row groups and page indexes are carried over as-is. The parquet
+/// API cannot carry a file decryptor, nor `FileMetaData`'s crate-private encryption fields
+/// (encryption algorithm, footer signing key metadata), across this rebuild, so callers must
+/// not stamp opens that supply decryption properties -- and the only consumer, the Delta
+/// scan, declines every encrypted-parquet configuration before planning, so a parquet
+/// modular encryption file never reaches this path with or without those properties.
+pub(crate) fn stamp_int96_leaves(metadata: &ParquetMetaData) -> Option<ParquetMetaData> {
+ let file_metadata = metadata.file_metadata();
+ let stamp = int96_leaf_stamp(file_metadata.schema_descr());
+ let existing = file_metadata
+ .key_value_metadata()
+ .and_then(|kvs| kvs.iter().find(|kv| kv.key == INT96_LEAVES_METADATA_KEY))
+ .and_then(|kv| kv.value.as_deref());
+ if existing == Some(stamp.as_str()) {
+ return None;
+ }
+ let mut key_values: Vec<KeyValue> = file_metadata
+ .key_value_metadata()
+ .map(|kvs| {
+ kvs.iter()
+ .filter(|kv| kv.key != INT96_LEAVES_METADATA_KEY)
+ .cloned()
+ .collect()
+ })
+ .unwrap_or_default();
+ key_values.push(KeyValue::new(INT96_LEAVES_METADATA_KEY.to_string(), stamp));
+ let stamped_file_metadata = FileMetaData::new(
+ file_metadata.version(),
+ file_metadata.num_rows(),
+ file_metadata.created_by().map(str::to_string),
+ Some(key_values),
+ file_metadata.schema_descr_ptr(),
+ file_metadata.column_orders().cloned(),
+ );
+ Some(
+ ParquetMetaData::new(stamped_file_metadata, metadata.row_groups().to_vec())
+ .into_builder()
+ .set_column_index(metadata.column_index().cloned())
+ .set_offset_index(metadata.offset_index().cloned())
+ .build(),
+ )
+}
+
+/// Per-file rebase policies for the three affected column classes, plus the INT96
+/// attribution that selects between the two timestamp specs per leaf.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct FileRebasePolicies {
+ /// `DATE` columns, governed by `org.apache.spark.legacyDateTime` alone.
+ pub date: RebasePolicy,
+ /// INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns: the datetime spec (same
+ /// resolution as `date`), as Spark's `ParquetVectorUpdaterFactory` selects for INT64.
+ pub int64_timestamp: RebasePolicy,
+ /// INT96 columns: the INT96 spec (`org.apache.spark.legacyINT96`, min version 3.1.0).
+ pub int96_timestamp: RebasePolicy,
+ /// Which timestamp leaves are INT96. See [`Int96Attribution`].
+ pub int96_leaves: Int96Attribution,
+ /// Sorted depth-first leaf ordinals -- over the physical file schema, the same ordinals
+ /// `int96_leaves` uses -- that the query does not read: nested children the schema
+ /// adapter's struct narrowing drops before any value leaves the scan. Spark never decodes
+ /// them either, so their policy is the identity whatever the file's calendar. Empty until
+ /// [`Self::restrict_to_requested`] runs (every leaf requested).
+ pub unrequested_leaves: Vec<usize>,
+}
+
+impl FileRebasePolicies {
+ /// True when some policy is not the plain proleptic-Gregorian pass-through, i.e. when the
+ /// per-column wrap in [`wrap_datetime_rebase`] can install anything at all.
+ pub(crate) fn any_rebase_needed(&self) -> bool {
+ self.date != RebasePolicy::Corrected
+ || self.int64_timestamp != RebasePolicy::Corrected
+ || self.int96_timestamp != RebasePolicy::Corrected
+ }
+
+ fn is_requested(&self, leaf: usize) -> bool {
+ self.unrequested_leaves.binary_search(&leaf).is_err()
+ }
+
+ /// The policy of the `Date32` leaf at depth-first ordinal `leaf`: the file's date policy,
+ /// or the identity when the query does not read that leaf.
+ fn date_policy(&self, leaf: usize) -> RebasePolicy {
+ if self.is_requested(leaf) {
+ self.date
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+
+ /// The policy of the timezone-carrying timestamp leaf at depth-first ordinal `leaf`: the
+ /// identity when the query does not read it; otherwise its physical type's spec when the
+ /// attribution is known, or else the two specs merged -- agreement decides, disagreement
+ /// degrades to [`RebasePolicy::CheckAncient`], which still passes every modern value and
+ /// refuses only ancient ones.
+ fn timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ Some(false) => self.int64_timestamp,
+ None if self.int64_timestamp == self.int96_timestamp => self.int64_timestamp,
+ None => RebasePolicy::CheckAncient,
+ }
+ }
+
+ /// These policies with every physical leaf the query does not read marked the identity.
+ /// `requested` pairs each top-level field of `physical_schema` (by position) with the type
+ /// of the logical field the schema adapter narrows it to -- `None` for a column without a
+ /// logical counterpart, whose leaves are left as they are (no expression reads it anyway).
+ /// Nested children pair the way the adapter's struct convert selects them (see
+ /// [`push_unrequested_leaves`]); the INT96 attribution is untouched, since the ordinals
+ /// stay physical. `requested` is parallel to the schema's fields; should a caller pass a
+ /// shorter slice, the trailing columns simply keep every leaf (the safe direction).
+ pub(crate) fn restrict_to_requested(
+ mut self,
+ physical_schema: &Schema,
+ requested: &[Option<&DataType>],
+ case_sensitive: bool,
+ use_field_id: bool,
+ ) -> Self {
+ debug_assert_eq!(requested.len(), physical_schema.fields().len());
+ let matching = FieldMatching {
+ case_sensitive,
+ use_field_id,
+ };
+ let mut next_leaf = 0;
+ let mut unrequested = Vec::new();
+ for (field, requested) in physical_schema.fields().iter().zip(requested) {
+ match requested {
+ Some(logical) => push_unrequested_leaves(
+ field.data_type(),
+ logical,
+ &mut next_leaf,
+ matching,
+ &mut unrequested,
+ ),
+ None => next_leaf += leaf_count(field.data_type()),
+ }
+ }
+ // Emitted in depth-first order, so already sorted for `is_requested`'s binary search.
+ self.unrequested_leaves = unrequested;
+ self
+ }
+}
+
+/// The field-matching rules of the schema adapter's nested narrowing
+/// (`parquet_convert_struct_to_struct`): names fold per `case_sensitive`, and Parquet field ids
+/// select fields when `use_field_id` is set.
+#[derive(Debug, Clone, Copy)]
+struct FieldMatching {
+ case_sensitive: bool,
+ use_field_id: bool,
+}
+
+/// Appends to `out` the depth-first leaf ordinals of `physical` (counting from `next_leaf`,
+/// which advances past every leaf of `physical`) that reading it as `requested` drops.
+///
+/// Recurses through exactly the pairings `parquet_convert_array` narrows, and no others: a
+/// struct child is dropped only when NO requested child selects it by either rule the struct
+/// convert uses -- folded name, or Parquet field id when ids are in play -- and an ambiguous
+/// child (several requested children select it) is kept; `List` pairs with `List` by element
+/// type, and `Map` with a `Map` of the same key ordering by its entries, positionally. Any
+/// other pairing -- a leaf, a `LargeList` / `FixedSizeList` / dictionary, a map whose ordering
+/// differs, or a shape mismatch -- is handed to arrow's cast or passed through whole by the
+/// convert, so it keeps every leaf. Keeping a superset of what the narrowing reads is always
+/// safe (a spurious check at worst); dropping a leaf the narrowing reads would skip its
+/// rebase, so every doubt resolves to "requested".
+fn push_unrequested_leaves(
+ physical: &DataType,
+ requested: &DataType,
+ next_leaf: &mut usize,
+ matching: FieldMatching,
+ out: &mut Vec<usize>,
+) {
+ match (physical, requested) {
+ (DataType::Struct(physical_fields), DataType::Struct(requested_fields)) => {
+ let names: Vec<&str> = physical_fields
+ .iter()
+ .chain(requested_fields.iter())
+ .map(|f| f.name().as_str())
+ .collect();
+ let folded = fold_names(&names, matching.case_sensitive);
+ let (physical_folded, requested_folded) = folded.split_at(physical_fields.len());
+ for (i, child) in physical_fields.iter().enumerate() {
+ let child_id = if matching.use_field_id {
+ parse_field_id(child)
+ } else {
+ None
+ };
+ let mut selectors = requested_fields.iter().enumerate().filter(|(j, r)| {
+ requested_folded[*j] == physical_folded[i]
+ || (child_id.is_some() && parse_field_id(r) == child_id)
+ });
+ match (selectors.next(), selectors.next()) {
+ (None, _) => {
+ let n = leaf_count(child.data_type());
+ out.extend(*next_leaf..*next_leaf + n);
+ *next_leaf += n;
+ }
+ (Some((_, requested_child)), None) => push_unrequested_leaves(
+ child.data_type(),
+ requested_child.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (Some(_), Some(_)) => *next_leaf += leaf_count(child.data_type()),
+ }
+ }
+ }
+ (DataType::List(physical_item), DataType::List(requested_item)) => push_unrequested_leaves(
+ physical_item.data_type(),
+ requested_item.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (
+ DataType::Map(physical_entries, physical_sorted),
+ DataType::Map(requested_entries, requested_sorted),
+ ) if physical_sorted == requested_sorted => {
+ match (physical_entries.data_type(), requested_entries.data_type()) {
+ (DataType::Struct(physical_kv), DataType::Struct(requested_kv))
+ if physical_kv.len() == requested_kv.len() =>
+ {
+ for (p, r) in physical_kv.iter().zip(requested_kv.iter()) {
+ push_unrequested_leaves(
+ p.data_type(),
+ r.data_type(),
+ next_leaf,
+ matching,
+ out,
+ );
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+}
+
+/// The writer time zone recorded in `metadata`, classified for timestamp rebasing. Mirrors the
+/// `Option(lookupFileMeta(SPARK_TIMEZONE_METADATA_KEY))` lookup Spark's `getRebaseSpec` performs
+/// for every LEGACY resolution, conf-fallback included; Spark substitutes the JVM default zone
+/// when the key is absent (`RebaseSpec.timeZone`), which is unavailable natively, so an absent or
+/// non-UTC zone classifies as [`WriterTimeZone::OtherOrUnknown`] (dates still rebase fully --
+/// the day rebase is zone-free -- while ancient timestamps refuse rather than guess).
+fn writer_time_zone(metadata: &HashMap<String, String>) -> WriterTimeZone {
+ match metadata.get(SPARK_TIMEZONE_KEY) {
+ Some(tz) if UTC_EQUIVALENT_TIMEZONES.contains(&tz.as_str()) => WriterTimeZone::Utc,
+ _ => WriterTimeZone::OtherOrUnknown,
+ }
+}
+
+/// One spec resolution, mirroring Spark's `DataSourceUtils.getRebaseSpec` exactly: a Spark
+/// version below `min_version` (lexicographic comparison, same as the Scala `String.<`) or a
+/// present legacy flag means LEGACY; a Spark version at/after `min_version` without the flag
+/// means CORRECTED; no Spark version at all falls back to `conf_mode`, the session read conf
+/// forwarded from the JVM (`getRebaseSpec`'s `modeByConfig` fallback, its ONLY use of the
+/// conf): CORRECTED passes values through, LEGACY rebases (with the writer zone from the
+/// file's `org.apache.spark.timeZone` key, same lookup as the metadata-driven LEGACY path),
+/// and EXCEPTION refuses ancient values as [`RebasePolicy::CheckAncient`].
+fn resolve_spec(
+ metadata: &HashMap<String, String>,
+ min_version: &str,
+ legacy_key: &str,
+ conf_mode: RebaseReadMode,
+) -> RebasePolicy {
+ match metadata.get(SPARK_VERSION_METADATA_KEY) {
+ None => match conf_mode {
+ RebaseReadMode::Corrected => RebasePolicy::Corrected,
+ RebaseReadMode::Legacy => RebasePolicy::Legacy(writer_time_zone(metadata)),
+ RebaseReadMode::Exception => RebasePolicy::CheckAncient,
+ },
+ Some(version) => {
+ if version.as_str() < min_version || metadata.contains_key(legacy_key) {
+ RebasePolicy::Legacy(writer_time_zone(metadata))
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+ }
+}
+
+/// Resolves the per-file rebase policies from a file's arrow schema: the parquet footer's
+/// key-value pairs in its metadata decide the specs (the datetime spec uses min version
+/// `3.0.0` and the INT96 spec `3.1.0`, matching `DataSourceUtils.datetimeRebaseSpec` /
+/// `int96RebaseSpec`; `session_modes` supplies the per-spec conf fallback for files without
+/// Spark writer metadata), and the reader factory's INT96 stamp -- validated against the
+/// schema's leaf structure -- attributes each timestamp leaf to its spec.
+pub(crate) fn resolve_file_rebase_policies(
+ physical_file_schema: &Schema,
+ session_modes: SessionRebaseModes,
+) -> FileRebasePolicies {
+ let metadata = physical_file_schema.metadata();
+ let datetime_spec = resolve_spec(
+ metadata,
+ "3.0.0",
+ SPARK_LEGACY_DATETIME_KEY,
+ session_modes.datetime,
+ );
+ let int96_spec = resolve_spec(
+ metadata,
+ "3.1.0",
+ SPARK_LEGACY_INT96_KEY,
+ session_modes.int96,
+ );
+ FileRebasePolicies {
+ date: datetime_spec,
+ int64_timestamp: datetime_spec,
+ int96_timestamp: int96_spec,
+ int96_leaves: Int96Attribution::from_schema(physical_file_schema),
+ unrequested_leaves: Vec::new(),
+ }
+}
+
+/// Number of primitive leaves `dt` contains in a depth-first walk -- the same count and order
+/// parquet-rs uses when it maps the file's `SchemaDescriptor` columns onto the arrow schema, so
+/// arrow-side leaf ordinals line up with [`int96_leaf_stamp`]'s.
+fn leaf_count(dt: &DataType) -> usize {
+ match dt {
+ DataType::Struct(fields) => fields.iter().map(|f| leaf_count(f.data_type())).sum(),
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_count(f.data_type()),
+ DataType::Dictionary(_, value) => leaf_count(value),
+ DataType::RunEndEncoded(_, value) => leaf_count(value.data_type()),
+ DataType::Union(fields, _) => fields.iter().map(|(_, f)| leaf_count(f.data_type())).sum(),
+ _ => 1,
+ }
+}
+
+/// Appends the policy of every leaf of `dt`, in depth-first order, to `out`, consuming leaf
+/// ordinals from `next_leaf` (exactly [`leaf_count`] of them). Only `Date32` and
+/// timezone-carrying timestamps have a policy to apply, and only when the query reads the
+/// leaf; timezone-free timestamps are `TIMESTAMP_NTZ`, which Spark never rebases, and every
+/// other leaf is the identity ([`RebasePolicy::Corrected`]).
+fn leaf_policies(
+ dt: &DataType,
+ next_leaf: &mut usize,
+ policies: &FileRebasePolicies,
+ out: &mut Vec<RebasePolicy>,
+) {
+ match dt {
+ DataType::Date32 => {
+ out.push(policies.date_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Timestamp(_, Some(_)) => {
+ out.push(policies.timestamp_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Struct(fields) => {
+ for f in fields {
+ leaf_policies(f.data_type(), next_leaf, policies, out);
+ }
+ }
+ // Mirrors `leaf_count` variant for variant, so a rebase-affected leaf inside a nested
+ // type `rebase_array` cannot rebuild (views, run-end, union -- never produced from a
+ // parquet schema) still gets its real policy and makes `rebase_array` refuse loudly
+ // instead of being stamped the identity.
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_policies(f.data_type(), next_leaf, policies, out),
+ DataType::Dictionary(_, value) => leaf_policies(value, next_leaf, policies, out),
+ DataType::RunEndEncoded(_, value) => {
+ leaf_policies(value.data_type(), next_leaf, policies, out)
+ }
+ DataType::Union(fields, _) => {
+ for (_, f) in fields.iter() {
+ leaf_policies(f.data_type(), next_leaf, policies, out);
+ }
+ }
+ _ => {
+ *next_leaf += 1;
+ out.push(RebasePolicy::Corrected);
+ }
+ }
+}
+
+/// Wraps every column reference in `expr` whose physical file type contains a rebase-affected
+/// leaf under a policy that needs handling with a [`SparkDatetimeRebaseExpr`] carrying that
+/// column's per-leaf policies, so both the per-file projection and the pushed-down predicate
+/// evaluate rebased values. Columns whose leaves are all the identity -- unaffected types,
+/// affected types under [`RebasePolicy::Corrected`], or leaves the query does not read (see
+/// [`FileRebasePolicies::restrict_to_requested`]) -- pass through unwrapped. (The pruning
+/// predicates derived from the wrapped predicate treat the wrapper as an opaque expression and
+/// skip pruning on those columns -- conservative, since file-level statistics are in the
+/// file's own calendar.)
+pub(crate) fn wrap_datetime_rebase(
+ expr: Arc<dyn PhysicalExpr>,
+ physical_schema: &SchemaRef,
+ policies: &FileRebasePolicies,
+) -> DataFusionResult<Arc<dyn PhysicalExpr>> {
+ expr.transform(|e| {
+ let Some(col) = e.downcast_ref::<Column>() else {
+ return Ok(Transformed::no(e));
+ };
+ // Missing columns were already replaced with literals; any surviving reference is
+ // physical-schema-indexed. Out-of-range means a non-file column (defensive): skip.
+ let Some(field) = physical_schema.fields().get(col.index()) else {
+ return Ok(Transformed::no(e));
+ };
+ // This column's first leaf ordinal: the leaves of every preceding top-level field.
+ let mut next_leaf: usize = physical_schema.fields()[..col.index()]
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ let mut column_leaf_policies = Vec::with_capacity(leaf_count(field.data_type()));
+ leaf_policies(
+ field.data_type(),
+ &mut next_leaf,
+ policies,
+ &mut column_leaf_policies,
+ );
+ if column_leaf_policies
+ .iter()
+ .all(|p| *p == RebasePolicy::Corrected)
+ {
+ return Ok(Transformed::no(e));
+ }
+ Ok(Transformed::yes(Arc::new(SparkDatetimeRebaseExpr {
+ child: e,
+ field: Arc::clone(field),
+ leaf_policies: column_leaf_policies,
+ }) as Arc<dyn PhysicalExpr>))
+ })
+ .map(|t| t.data)
+}
+
+/// Applies a file's calendar-rebase policies to one column: rebases exactly where possible,
+/// raises on ancient values it cannot rebase, and passes modern values (the identity under
+/// every policy) through untouched. Nested columns are rebuilt leaf by leaf with nulls and
+/// offsets preserved. See the module doc for the policy table.
+#[derive(Debug, Eq)]
+struct SparkDatetimeRebaseExpr {
+ child: Arc<dyn PhysicalExpr>,
+ /// The physical file field this expression reads (type preserved by the rebase).
+ field: FieldRef,
+ /// One policy per primitive leaf of `field`'s type, in depth-first order (a single entry
+ /// for a flat column). At least one is not [`RebasePolicy::Corrected`].
+ leaf_policies: Vec<RebasePolicy>,
+}
+
+impl SparkDatetimeRebaseExpr {
+ /// The refusal error, as an [`ArrowError`] so `try_unary` closures can raise it directly;
+ /// it converts into a `DataFusionError` at the `?` in `evaluate`.
+ fn rebase_error(&self, detail: &str) -> ArrowError {
+ ArrowError::ComputeError(format!(
+ "Native scan cannot rebase ancient values in column '{}': the file was written \
+ with the legacy (hybrid Julian/Gregorian) calendar, or does not declare which \
+ calendar it used, and {detail}. Reading it natively would return silently \
+ shifted values; disable the native Delta scan \
+ (spark.comet.scan.delta.enabled=false) to let Spark read this table",
+ self.field.name(),
+ ))
+ }
+
+ fn internal_error(&self, detail: impl Display) -> DataFusionError {
+ DataFusionError::Internal(format!(
+ "SparkDatetimeRebaseExpr on column '{}': {detail}",
+ self.field.name()
+ ))
+ }
+
+ /// Rebases a timestamp column written at a fixed zero-offset zone: shift the nominal day
+ /// with the exact date table, keep the time of day. Matches Spark's
+ /// `rebaseJulianToGregorianMicros` for UTC, where the hybrid calendar's day boundaries sit
+ /// exactly on multiples of a day and no timezone transition can apply (UTC's last switch
+ /// instant in Spark's rebase table is the 1582-10-15 cutover itself).
+ fn rebase_timestamp_utc(&self, v: i64, units_per_day: i64) -> Result<i64, ArrowError> {
+ // Compare in days, not units: the cutover day times a nanosecond day does not fit i64.
+ let day = v.div_euclid(units_per_day);
+ if day >= LAST_SWITCH_JULIAN_DAY as i64 {
+ return Ok(v);
+ }
+ let time_of_day = v - day * units_per_day;
+ let day = i32::try_from(day).map_err(|_| {
+ self.rebase_error("the value is outside the rebaseable timestamp range")
+ })?;
+ let rebased = rebase_julian_to_gregorian_days(day) as i64;
+ rebased
+ .checked_mul(units_per_day)
+ .and_then(|d| d.checked_add(time_of_day))
+ .ok_or_else(|| self.rebase_error("the rebased value overflows the timestamp range"))
+ }
+
+ /// The refuse-ancient-values policy for timestamps: values from
+ /// [`LAST_SWITCH_JULIAN_TS_SECONDS`] onward are identical under both calendars in every
+ /// zone (Spark's `createTimestampRebaseFuncInRead` under `EXCEPTION` accepts exactly
+ /// these, and `rebaseJulianToGregorianMicros` is the identity on them for any zone);
+ /// older values raise.
+ fn check_ancient_timestamp(
+ &self,
+ v: i64,
+ units_per_second: i64,
+ detail: &str,
+ ) -> Result<i64, ArrowError> {
+ if v >= LAST_SWITCH_JULIAN_TS_SECONDS * units_per_second {
+ Ok(v)
+ } else {
+ Err(self.rebase_error(detail))
+ }
+ }
+
+ fn rebase_timestamp_array<T: ArrowTimestampType>(
+ &self,
+ array: &PrimitiveArray<T>,
+ policy: RebasePolicy,
+ units_per_second: i64,
+ ) -> DataFusionResult<ArrayRef> {
+ let tz = array.timezone().map(Arc::<str>::from);
+ let rebased: PrimitiveArray<T> = match policy {
+ RebasePolicy::Corrected => return Ok(Arc::new(array.clone())),
+ RebasePolicy::Legacy(WriterTimeZone::Utc) => arrow::compute::try_unary(array, |v| {
+ self.rebase_timestamp_utc(v, units_per_second * 86_400)
+ })?,
+ RebasePolicy::Legacy(WriterTimeZone::OtherOrUnknown) => {
Review Comment:
This arm and the `CheckAncient` arms below (timestamps here, dates at ~901) go through `try_unary`, which allocates a fresh values buffer and writes every value back unchanged. These are the policies a metadata-free file under EXCEPTION mode hits on every batch. A validity-aware `all(v >= cutoff)` over `values()` followed by `Ok(Arc::clone(array))` would make them allocation-free, and `Legacy(Utc)` could short-circuit the same way when the batch minimum is at or after the cutover.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4081090782
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/DeltaPlanDataInjector.scala:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.{OperatorOuterClass, QueryContextInterner}
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * PlanDataInjector for the Delta contrib scan, discovered by core's ServiceLoader (see the
+ * `META-INF/services` resource). Lives in this package because [[PlanDataInjector]] is
+ * `private[comet]`.
+ */
+class DeltaPlanDataInjector extends PlanDataInjector {
+
+ override val opStructCase: Operator.OpStructCase = Operator.OpStructCase.CONTRIB_SCAN
+
+ override def canInject(op: Operator): Boolean =
Review Comment:
> Could we add focused tests for the serialized Delta Spark envelope and the real `DeltaPlanDataInjector` through the registry?
Tracked in #5947, with the slot, type URL, payload and rejection assertions you list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3831229371
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.net.URI
+import java.util.Locale
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.CometScanRule
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands. Anything else on the protocol declines the
+ * table. Note `deletionVectors` and `columnMapping` are declined separately (below) so their
+ * fallback reasons are specific.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format (not a further subclass)? Compared by class name,
+ * not `classOf`, deliberately: this is the first gate on every V1 scan, and it must stay inert
+ * when the contrib jar is deployed without delta-spark on the classpath:
+ * `classOf[DeltaParquetFileFormat]` here raises NoClassDefFoundError inside CometScanRule and
+ * takes down every parquet scan in the session. When the name matches, delta-spark is
+ * necessarily present (the instance exists), so the Delta types past this gate are safe.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Returns the first reason this Delta scan cannot go native, or None when it is claimable. Only
+ * called when [[isDeltaScan]] is true. `scanHelper` is the same [[CometScanExec]] the caller
+ * builds to drive [[CometDeltaNativeScan.convert]] on a claim, reused here (rather than listed
+ * separately) to resolve the scan's selected files for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles with useMetadataRowIndex=false) injects a
+ // generated row-index column directly into the data schema and disables reader
+ // optimizations; its values must come from Spark's reader. Claiming such a scan would
+ // feed NULL row indexes into deletion-vector construction, silently corrupting DML.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ // Name mode is supported by serializing physical-name schemas (the parquet reader then
+ // matches file columns by name natively). Id mode needs the field-id path and stays
+ // declined until validated.
+ val cmMode = metadata.columnMappingMode.name
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping
+ // EXISTS_DEFAULT; decline any column defaults under column mapping rather than
+ // return nulls where a default belongs.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names (not just top-level column
+ // names) to their physical, column-mapped form, and the shared native builder uses the
+ // required schema verbatim as the scan's output schema: struct fields below the top level
+ // would carry physical names. Ordinal access (GetStructField) is unaffected, but
+ // name-sensitive native expressions (e.g. to_json) read the Arrow struct field names
+ // directly and would leak physical names into query results. Restoring logical names
+ // natively needs a rename adapter/proto field for the logical schema (follow-up); decline
+ // until then.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and not
+ // supported, except Delta's DV bookkeeping columns, which the native path emits as
+ // constants (correct by construction once the DV is applied in the reader).
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is not a DV read: it is Delta DML
+ // bookkeeping (findTouchedFiles building deletion bitmaps from REAL row indexes).
+ // Claiming it with constant row indexes would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // The internal columns must form a suffix of the read schema so data-column
+ // positions agree between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // The row-index column's real values are consumed inside the reader when Spark applies
+ // the DV; native applies the DV itself and emits a dead constant instead, so the value
+ // must be provably unused above the scan (beyond the _metadata reassembly that gets
+ // discarded).
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet; decline rather
+ // than silently return nulls for backfilled columns in old files.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bound the memory the native side will retain for expanded DV row selectors before
+ // committing to native execution: applying a deletion vector expands it into per-row
+ // RowSelectors that are reserved against the execution memory pool at scan time (see
+ // delta_dv.rs), and an alternating deleted/retained bitmap produces one non-coalescing
+ // selector per row. A row group's selector count is bounded above by
+ // 2*cardinality + #row-groups (each deleted row splits at most one run into a
+ // select/skip pair, plus one selector per row-group boundary), so the descriptor's
+ // cardinality -- deserialized at planning time via selectedDvDescriptors, no bitmap
+ // decode needed -- is a sound, pessimistic upper bound on the native reservation.
+ // Pessimistic by design: a large but CONTIGUOUS deletion is declined the same as a
+ // large alternating one, even though it would retain far fewer selectors natively; the
+ // conf below makes that recoverable.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = selectedDvDescriptors(scanHelper, tableRoot)
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from InputFileBlockHolder, a thread-local set by Spark's
+ // FileScanRDD; the native scan does not populate it. Delta's own DELETE/UPDATE/MERGE
+ // find-touched-files scans use input_file_name, so this gate is load-bearing for DML
+ // correctness (mirrors core's check in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core).
+ // The DV shape's trailing row-index column is exempt: the gates above already proved its
+ // values are dead and the native path emits a constant for it.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does
+ // not yet wire the decryption-key broadcast to executors, so claiming even a
+ // supported-encryption scan would fail at execution.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults (schema-evolution backfill of map/struct/array columns)
+ // cannot be serialized; a silently-dropped default would misalign the value/index lists
+ // consumed positionally on the native side. Mirrors core's transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read; otherwise a custom Hadoop FileSystem would fail at execution instead of
+ // falling back gracefully. Mirrors core's unsupportedFsSchemes gate.
+ val libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+ val unsupportedFsSchemes = scanExec.relation.location.rootPaths
+ .map(_.toUri)
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfsSchemes.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ if (unsupportedFsSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedFsSchemes.mkString(", ")}")
+ }
+
+ // A Delta shallow clone across buckets followed by an append is a valid table whose data
+ // files span multiple object-store authorities. The shared native scan builder resolves
+ // the whole scan's ObjectStoreUrl from the FIRST selected file only and then strips every
+ // other file down to its bare object-store path, so a later file under a different store
+ // would silently read through the first file's store handle -- normally a NoSuchKey, but
+ // the wrong data if a same-named key happens to exist in both stores. Force file listing
+ // here (scanHelper is already built for the claim path, so this is not extra work) and
+ // decline rather than risk it.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles,
+ // AQE DPP on Spark 3.4, exec enabled). This tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, deserialized once at
+ * planning time and normalized to absolute on-disk paths via `copyWithAbsolutePath` (a no-op
+ * for inline and already-absolute descriptors), so callers never need `tableRoot` again to
+ * resolve a UUID-relative sidecar. Returns `Seq.empty` for the plain shape
+ * ([[CometDeltaNativeScan.isDvShape]] false on the wrapped scan): only DV reads carry the
+ * row-index-filter metadata this deserializes.
+ *
+ * Shared plumbing: finding 8's cross-authority object-store option merge
+ * ([[CometDeltaNativeScan.convert]]) and finding 3's DV cardinality decline gate both need
+ * every selected file's descriptor; this is the one planning-time deserialization pass for both
+ * consumers.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * Returns a decline reason when `uris` span more than one object-store authority (scheme plus
+ * the URI's raw authority component -- userinfo, host, and port together -- all lowercased so
+ * e.g. `S3A://Bucket:1234` and `s3a://bucket:1234` collapse to the same authority), or `None`
+ * when every URI shares a single authority. `file://` paths never carry an authority, so purely
+ * local scans across any number of distinct directories are unaffected. Factored out of
+ * [[declineReason]] so it is directly unit-testable without a Spark session.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the URI's raw
+ * `getAuthority` rather than the individually-parsed host/port/userinfo fields. Two pitfalls
+ * that motivate this:
+ * - `getAuthority` already includes userinfo (e.g. the container in
+ * `abfss://container@account.dfs.core.windows.net`), so two containers on the same storage
+ * account no longer collapse into one authority the way `getHost` alone would.
+ * - `getHost` (and `getUserInfo`/`getPort`) return `null` for the *entire* authority when it
+ * doesn't conform to RFC 3986's `reg-name` syntax -- e.g. an underscore in a GCS bucket
+ * name (`gs://my_bucket`) -- silently collapsing distinct buckets into the same empty-host
+ * key. `getAuthority` returns the raw authority text regardless of RFC conformance, so it
+ * stays accurate for exactly the URIs where the structured getters fail.
+ *
+ * A `null` authority (schemes with no authority component, e.g. `file:///tmp/x`) normalizes to
+ * the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * True when `dataType` is, or structurally contains (through array elements or map keys/
+ * values), a [[StructType]]. Array and map are structural container types whose own
+ * "element"/"key"/"value" labels are never column-mapped; only the [[StructType]] fields
+ * reachable through them carry Delta's physical, column-mapped names.
+ */
+ private def containsNestedStruct(dataType: DataType): Boolean = dataType match {
+ case _: StructType => true
+ case ArrayType(elementType, _) => containsNestedStruct(elementType)
+ case MapType(keyType, valueType, _) =>
+ containsNestedStruct(keyType) || containsNestedStruct(valueType)
+ case _ => false
+ }
+
+ /**
+ * True when the scan's row-index column value is provably dead above the scan. The standard DV
+ * plan shape routes it only into a `named_struct(... row_index ...) AS _metadata` projection
+ * whose result the final projection discards; anything else (a query actually selecting
+ * `_metadata.row_index`) makes the value live and must decline. Conservative: any unrecognized
+ * consumption pattern returns false.
+ */
+ private def rowIndexUnusedAbove(plan: SparkPlan, scanExec: FileSourceScanExec): Boolean = {
+ val rowIndexAttrs = scanExec.output
+ .filter(_.name == CometDeltaNativeScan.RowIndexColumn)
+ .map(_.exprId)
+ .toSet
+ if (rowIndexAttrs.isEmpty) {
+ return true
+ }
+ // Transitive taint analysis: everything derived (via Project aliases) from the
+ // row-index attribute within the VISIBLE plan. The plan handed to this rule may be
+ // an AQE stage fragment, so anything tainted that reaches the fragment's own output
+ // escapes to invisible consumers and must decline. Non-Project consumption of any
+ // tainted attribute (a Filter, Aggregate, Join key, ...) declines outright.
+ var tainted = rowIndexAttrs
+ var changed = true
+ while (changed) {
+ changed = false
+ plan.foreach {
+ case p: ProjectExec =>
+ p.projectList.foreach {
+ case a: Alias
+ if !tainted.contains(a.exprId) &&
+ a.references.exists(r => tainted.contains(r.exprId)) =>
+ tainted += a.exprId
+ changed = true
+ case _ =>
+ }
+ case _ =>
+ }
+ }
+ val nonProjectConsumer = plan.exists {
+ case _: ProjectExec => false
+ case n if n ne scanExec =>
+ n.expressions.exists(_.references.exists(r => tainted.contains(r.exprId)))
Review Comment:
Took the positional-propagation option, since a blanket decline would reject every DV-backed UNION ALL (Delta appends the row-index column to requiredSchema on all DV reads). The taint fixed point now maps child.output(i) to union.output(i) for every branch, covering UnionExec and CometUnionExec; a CometUnionExec whose frozen output ever diverges in arity from its re-parented children declines outright rather than zipping silently. There's also a generic safety net now: any node with two or more children that isn't a positional union, where a tainted child attribute neither appears in the output by exprId nor in the node's expressions, declines, so this gap class can't silently recur (joins pass since they carry child exprIds; semi/anti joins trip only on eliminated-side taint, where declining is correct). Your exact query plus second-branch selection and aggregate-over-union are regression tests asserting values, SUM(ri) is 15, and an anti-regression proves DV unions without _met
adata still claim both branches natively. One residual documented in the code: ReusedExchangeExec has the same positional shape but is unreachable at claim time because ReuseExchangeAndSubquery runs after the columnar rules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5678443632
> Please activate `delta` for the prerequisite installs and the README's matching command, or use a build sequence that produces the current test JAR.
Done: both prerequisite installs in `delta_contrib_test.yml` and the README's first command now run with `-Pspark-X,delta`, so the install produces this checkout's test-jar before the contrib suites resolve it.
> Please use a setup path that elevates package installation on the plain runner while retaining Docker access.
Done: the MinIO job no longer uses `setup-builder`; it installs `protobuf-compiler` and `clang` with `sudo`, sets up JDK 17 through `setup-java`, and installs the Rust toolchain with `rustup`, mirroring the action's steps. The workflow parses and `check-ci-config.py` passes; the first CI run is the proof, since the workflows here still wait on approval.
> Please include a focused before/after microbenchmark for this new fast path using modern, null-heavy and mixed-age batches.
Measured on a release build, 1M-row microsecond UTC timestamp batches, median of 20 runs, before against after:
| batch | policy | before | after |
| --- | --- | --- | --- |
| modern | CheckAncient | 461 us | 199 us |
| modern | Legacy(Utc) | 698 us | 199 us |
| null-heavy (90% null) | CheckAncient | 215 us | 251 us |
| null-heavy (90% null) | Legacy(Utc) | 194 us | 245 us |
| mixed-age (half ancient) | Legacy(Utc) | 851 us | 1098 us |
The first cut used a validity-aware scalar loop and lost on null-heavy batches, so the landed version takes one vectorised minimum only on null-free batches and returns the input untouched when nothing predates the cutover; batches with nulls or ancient values keep the previous per-value arms, so the cost there is the extra minimum pass. Modern null-free batches, the common case for a metadata-free file under EXCEPTION mode, are 2.3x to 3.5x faster; the mixed-age case pays about 30% more. Pruning is unchanged by this, as you noted.
> Agreed follow-up issues should be linked before treating those requests as resolved.
Filed, one per request: #5943 (stable injection key), #5944 (shared admission checks), #5945 (shared builder for the DV path), #5946 (explicit planning boundary), #5947 (envelope and provider contract tests), #5948 (preparation I/O through the scan's instrumented reader), #5949 (delegate schema preparation to `prepareSchemaForRead`). Each records your criteria and the tests you asked for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4006955146
##########
docs/source/contributor-guide/ci.md:
##########
@@ -69,6 +69,7 @@ Each queue-only suite has a label that runs it on a pull request:
| `run-spark-3.4-tests` | Spark SQL tests against Spark 3.4 |
| `run-spark-3.5-tests` | Spark SQL tests against Spark 3.5 |
| `run-spark-4.0-tests` | Spark SQL tests against Spark 4.0 |
+| `run-delta-tests` | Delta contrib tests against Spark 3.5 |
Review Comment:
> I confirmed the 4.1 leg passes locally, so the description could claim it as well.
The CI guide row, the policy comment in `compute-changes.py` and the workflows README now say 3.5, 4.0 and 4.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4042628009
##########
native/core/src/parquet/datetime_rebase.rs:
##########
@@ -0,0 +1,2794 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Per-file datetime calendar-rebase handling for the parquet scan.
+//!
+//! Spark 2.4 and earlier wrote dates and timestamps in the hybrid Julian + Gregorian calendar;
+//! Spark 3.0+ uses the proleptic Gregorian calendar and records the calendar policy of every
+//! file it writes in the parquet footer's key-value metadata (`org.apache.spark.version`,
+//! `org.apache.spark.legacyDateTime`, `org.apache.spark.legacyINT96`,
+//! `org.apache.spark.timeZone`). Spark's reader resolves the rebase policy from EACH FILE's
+//! writer metadata (`DataSourceUtils.datetimeRebaseSpec` / `int96RebaseSpec`) -- the session's
+//! `spark.sql.parquet.datetimeRebaseModeInRead` conf only applies to files whose metadata does
+//! not decide the policy on its own -- so a reader that ignores the metadata silently returns
+//! values shifted by up to ten days for dates before 1582-10-15 (e.g. `1500-01-01` reads as
+//! `1500-01-10`).
+//!
+//! This module mirrors that per-file resolution: [`resolve_file_rebase_policies`] computes the
+//! date / INT64-timestamp / INT96-timestamp policies from a file's arrow schema metadata (the
+//! parquet key-value pairs survive the parquet -> arrow schema conversion), and
+//! [`wrap_datetime_rebase`] wraps the per-file rewritten expressions' column references in a
+//! [`SparkDatetimeRebaseExpr`] that rebases values exactly where that is possible without the
+//! JVM's historical timezone tables (dates always; timestamps for a fixed UTC writer zone) and
+//! refuses -- rather than silently corrupting -- ancient values it cannot rebase. Nested
+//! columns are rebuilt leaf by leaf (struct / list / map / fixed-size list / dictionary), each
+//! leaf under its own policy, with nulls and offsets preserved. Modern values are always the
+//! identity under every policy: from 1582-10-15 onward for dates, and from
+//! [`LAST_SWITCH_JULIAN_TS_SECONDS`] (1900-01-01T00:00:00Z, Spark's
+//! `RebaseDateTime.lastSwitchJulianTs`) onward for timestamps.
+//!
+//! Spark applies `datetimeRebaseSpec` to INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns
+//! and `int96RebaseSpec` to INT96 columns. The two physical types are indistinguishable in the
+//! arrow schema DataFusion hands the expression adapter (both surface as `Timestamp(us, "UTC")`
+//! after INT96 coercion), so Comet's parquet reader factory stamps the file's INT96 leaf
+//! ordinals -- taken from the parquet footer's own `SchemaDescriptor` -- into the key-value
+//! metadata under [`INT96_LEAVES_METADATA_KEY`] before the arrow schema is derived (see
+//! [`stamp_int96_leaves`] and `eager_page_index_reader_factory.rs`), and the adapter attributes
+//! every timestamp leaf to its spec from that stamp. Without a stamp, the two specs are merged:
+//! agreement decides, disagreement degrades to [`RebasePolicy::CheckAncient`].
+//!
+//! The wrapper sits BENEATH the schema adapter's nested narrowing (the struct -> struct convert
+//! that keeps only the requested children), which is what keeps those ordinals physical -- but
+//! it means the wrapper sees every physical child, requested or not. Spark only ever decodes
+//! the requested nested schema, so [`FileRebasePolicies::restrict_to_requested`] marks the
+//! physical leaves the narrowing drops as the identity: an unrequested ancient `s.ts` never
+//! blocks `select s.d`, exactly as in Spark.
+//!
+//! Currently only enabled by the Delta scan arms via
+//! `SparkParquetOptions::rebase_from_file_metadata`, which also carries the session read modes
+//! ([`SessionRebaseModes`], forwarded from the JVM) that decide the policy for files without
+//! Spark writer metadata; the plain NativeScan keeps its documented no-rebase behavior (see
+//! the compatibility guide and issue #5010).
+
+use std::collections::HashMap;
+use std::fmt::{self, Display};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Date32Array, FixedSizeListArray, GenericListArray, MapArray,
+ OffsetSizeTrait, PrimitiveArray, RecordBatch, StructArray,
+};
+use arrow::datatypes::{
+ ArrowPrimitiveType, ArrowTimestampType, DataType, Date32Type, FieldRef, Schema, SchemaRef,
+ TimeUnit, TimestampMicrosecondType, TimestampMillisecondType, TimestampNanosecondType,
+ TimestampSecondType,
+};
+use arrow::error::ArrowError;
+use datafusion::common::tree_node::{Transformed, TreeNode};
+use datafusion::common::{DataFusionError, Result as DataFusionResult};
+use datafusion::physical_expr::expressions::Column;
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::ColumnarValue;
+use parquet::basic::Type as ParquetPhysicalType;
+use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaData};
+use parquet::schema::types::SchemaDescriptor;
+
+use super::name_fold::fold_names;
+use super::schema_adapter::parse_field_id;
+
+/// Footer key naming the Spark release that wrote the file; absent for non-Spark writers.
+const SPARK_VERSION_METADATA_KEY: &str = "org.apache.spark.version";
+/// Present (empty value) when the file's dates and INT64 timestamps were written with
+/// `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_DATETIME_KEY: &str = "org.apache.spark.legacyDateTime";
+/// Present (empty value) when the file's INT96 timestamps were written with
+/// `spark.sql.parquet.int96RebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_INT96_KEY: &str = "org.apache.spark.legacyINT96";
+/// The writer session's time zone, stamped alongside either legacy flag.
+const SPARK_TIMEZONE_KEY: &str = "org.apache.spark.timeZone";
+
+/// Key-value metadata entry Comet's parquet reader factory adds to a file's footer metadata
+/// (in memory only, never written back) so the expression adapter can tell INT96 timestamp
+/// columns from INT64 ones after both have been coerced to the same arrow type. Value:
+/// `"<leaf count>:<comma-separated INT96 leaf ordinals>"`, where leaves are the file's
+/// primitive columns in `SchemaDescriptor::columns()` order -- the same depth-first order
+/// parquet-rs assigns arrow leaves, so an arrow-side depth-first walk lines up with it. The
+/// leaf count lets the reader detect a stamp that does not describe the schema it is paired
+/// with (see [`Int96Attribution::from_schema`]).
+pub(crate) const INT96_LEAVES_METADATA_KEY: &str = "comet.int96_leaf_columns";
+
+/// Day of the Gregorian cutover (1582-10-15) as days since the epoch; rebasing is the identity
+/// from this day onward. Same value as Spark's `RebaseDateTime.lastSwitchJulianDay`.
+const LAST_SWITCH_JULIAN_DAY: i32 = -141427;
+
+/// Spark's `RebaseDateTime.lastSwitchJulianTs` (and `lastSwitchGregorianTs`) in seconds since
+/// the epoch: 1900-01-01T00:00:00Z. Spark derives it as the latest switch instant across every
+/// zone in its `julian-gregorian-rebase-micros.json` table (`getLastSwitchTs`, which also
+/// asserts the calendars' difference is zero for every zone from then on): most zones ran on
+/// local mean time before 1900, so the last instant at which rebasing changes a value in ANY
+/// zone is 1900-01-01T00:00:00Z, not the 1582 cutover. `createTimestampRebaseFuncInRead`
+/// under `EXCEPTION` throws exactly for `micros < lastSwitchJulianTs` (after converting
+/// `TIMESTAMP_MILLIS` to micros), and `rebaseJulianToGregorianMicros` is the identity from it
+/// onward in every zone. The value is in seconds so it scales exactly to any timestamp unit.
+pub(crate) const LAST_SWITCH_JULIAN_TS_SECONDS: i64 = -2_208_988_800;
+
+/// The per-century differences between the Julian and proleptic Gregorian calendars, and the
+/// Julian-calendar switch days at which each difference starts to apply. Copied verbatim from
+/// Spark's `RebaseDateTime.julianGregDiffs` / `julianGregDiffSwitchDay` (which Spark generated
+/// from `localRebaseJulianToGregorianDays`); `rebase_julian_to_gregorian_days` must stay
+/// value-for-value equal to Spark's `rebaseJulianToGregorianDays`.
+const JULIAN_GREG_DIFFS: [i32; 14] = [2, 1, 0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, 0];
+const JULIAN_GREG_DIFF_SWITCH_DAY: [i32; 14] = [
+ -719164, -682945, -646420, -609895, -536845, -500320, -463795, -390745, -354220, -317695,
+ -244645, -208120, -171595, -141427,
+];
+
+/// Proleptic-Gregorian days since 1970-01-01 for a nominal civil date, via Howard Hinnant's
+/// `days_from_civil`. `d` may exceed the month's length; the excess rolls into the following
+/// month exactly like `LocalDate.of(y, m, 1).plusDays(d - 1)` in Spark's
+/// `localRebaseJulianToGregorianDays` (how the non-existent proleptic date `1000-02-29`,
+/// valid in the Julian calendar, lands on `1000-03-01`).
+fn days_from_civil(y: i64, m: i64, d: i64) -> i64 {
+ let y = if m <= 2 { y - 1 } else { y };
+ let era = y.div_euclid(400);
+ let yoe = y - era * 400; // [0, 399]
+ let mp = (m + 9) % 12; // [0, 11], March = 0
+ let doy = (153 * mp + 2) / 5 + d - 1;
+ let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
+ era * 146097 + doe - 719468
+}
+
+/// Julian-calendar civil date `(year, month, day)` for a day count since 1970-01-01 that labels
+/// days in the Julian calendar (astronomical year numbering: 1 BCE is year 0). Standard
+/// Julian-day-number conversion (E.G. Richards' algorithm), exact for any day.
+fn julian_day_to_civil(days: i64) -> (i64, i64, i64) {
+ // Integer (noon) Julian Day Number of this civil day: 1970-01-01 is JDN 2440588.
+ let jdn = days + 2_440_588;
+ let f = jdn + 1401;
+ let e = 4 * f + 3;
+ let g = e.rem_euclid(1461) / 4;
+ let h = 5 * g + 2;
+ let day = h.rem_euclid(153) / 5 + 1;
+ let month = (h / 153 + 2).rem_euclid(12) + 1;
+ let year = e.div_euclid(1461) - 4716 + (14 - month) / 12;
+ (year, month, day)
+}
+
+/// Exact port of Spark's `RebaseDateTime.rebaseJulianToGregorianDays`: reinterprets a day count
+/// written in the hybrid Julian + Gregorian calendar as the proleptic Gregorian day count of the
+/// same nominal civil date. Identity for days from 1582-10-15 onward. Days before the tables'
+/// range (before Julian `0001-01-01`) take the calendar-arithmetic path, mirroring Spark's
+/// `localRebaseJulianToGregorianDays` fallback.
+pub(crate) fn rebase_julian_to_gregorian_days(days: i32) -> i32 {
+ if days < JULIAN_GREG_DIFF_SWITCH_DAY[0] {
+ let (y, m, d) = julian_day_to_civil(days as i64);
+ (days_from_civil(y, m, 1) + (d - 1)) as i32
+ } else {
+ // Spark's rebaseDays: linear search from the most recent switch day.
+ let mut i = JULIAN_GREG_DIFF_SWITCH_DAY.len();
+ loop {
+ i -= 1;
+ if i == 0 || days >= JULIAN_GREG_DIFF_SWITCH_DAY[i] {
+ break;
+ }
+ }
+ days + JULIAN_GREG_DIFFS[i]
+ }
+}
+
+/// Timezone strings from `org.apache.spark.timeZone` that denote a fixed zero-offset zone in
+/// both `java.util.TimeZone` and `java.time`. Only for these is timestamp rebasing the pure
+/// nominal-date shift [`SparkDatetimeRebaseExpr::rebase_timestamp_utc`] computes; any other (or
+/// absent) zone needs the JVM's historical timezone tables and stays on the
+/// refuse-ancient-values path.
+const UTC_EQUIVALENT_TIMEZONES: [&str; 6] = ["UTC", "Etc/UTC", "GMT", "Etc/GMT", "Z", "+00:00"];
+
+/// How the writer's session time zone (if recorded) affects timestamp rebasing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum WriterTimeZone {
+ /// A fixed zero-offset zone: rebasing reduces to the exact nominal-date shift.
+ Utc,
+ /// Any other zone, or none recorded (pre-3.0 files): ancient values cannot be rebased
+ /// without the JVM's historical timezone data.
+ OtherOrUnknown,
+}
+
+/// One session-level datetime rebase read mode (a `LegacyBehaviorPolicy` value of
+/// `spark.sql.parquet.datetimeRebaseModeInRead` / `int96RebaseModeInRead`), consulted by
+/// [`resolve_file_rebase_policies`] ONLY for files whose footer metadata does not decide the
+/// policy on its own -- exactly the `getOrElse` fallback in Spark's
+/// `DataSourceUtils.getRebaseSpec`. Files that carry `org.apache.spark.version` ignore these
+/// modes entirely, on every Spark version.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) enum RebaseReadMode {
+ /// Refuse ancient values (Spark raises `SparkUpgradeException`); maps to
+ /// [`RebasePolicy::CheckAncient`]. The default mirrors the conservative posture used
+ /// before the conf was plumbed through (and Spark 3.x's own conf default).
+ #[default]
+ Exception,
+ /// Read values as proleptic Gregorian without rebasing.
+ Corrected,
+ /// Rebase from the hybrid Julian + Gregorian calendar.
+ Legacy,
+}
+
+impl RebaseReadMode {
+ /// Parses a `LegacyBehaviorPolicy` conf value. `SQLConf` validates and upper-cases the
+ /// session conf, but a per-relation `datetimeRebaseMode` option arrives verbatim, so the
+ /// match is case-insensitive. Anything unrecognized -- including the empty string a proto
+ /// producer that predates the field sends -- falls back to [`RebaseReadMode::Exception`],
+ /// which refuses ancient values rather than silently corrupting them.
+ pub(crate) fn from_conf_value(value: &str) -> Self {
+ match value.to_ascii_uppercase().as_str() {
+ "CORRECTED" => RebaseReadMode::Corrected,
+ "LEGACY" => RebaseReadMode::Legacy,
+ _ => RebaseReadMode::Exception,
+ }
+ }
+}
+
+/// The session's effective datetime rebase read modes, one per spec class (INT64
+/// dates/timestamps vs INT96 timestamps), forwarded from the JVM at planning time.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) struct SessionRebaseModes {
+ /// `spark.sql.parquet.datetimeRebaseModeInRead` (or the relation's `datetimeRebaseMode`).
+ pub datetime: RebaseReadMode,
+ /// `spark.sql.parquet.int96RebaseModeInRead` (or the relation's `int96RebaseMode`).
+ pub int96: RebaseReadMode,
+}
+
+/// Calendar policy of one file's date or timestamp columns, resolved from writer metadata the
+/// same way Spark's `DataSourceUtils.getRebaseSpec` resolves it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum RebasePolicy {
+ /// Written in the proleptic Gregorian calendar; values pass through untouched.
+ Corrected,
+ /// Written in the hybrid Julian + Gregorian calendar; values must be rebased.
+ Legacy(WriterTimeZone),
+ /// Policy could not be pinned down (contradictory flags, or a non-Spark writer under the
+ /// `EXCEPTION` read mode): modern values -- identical under either calendar -- pass,
+ /// ancient values raise. Mirrors Spark's `EXCEPTION` behavior (`SparkUpgradeException`).
+ CheckAncient,
+}
+
+/// Which of a file's leaf columns are physically INT96, from the stamp the parquet reader
+/// factory adds under [`INT96_LEAVES_METADATA_KEY`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) enum Int96Attribution {
+ /// No stamp, or a stamp whose leaf count does not match the schema it arrived with: the
+ /// INT64 and INT96 timestamp specs cannot be told apart per column and are merged.
+ Unknown,
+ /// Sorted leaf ordinals (depth-first over the file schema's primitive columns) that are
+ /// INT96; every other timestamp leaf is INT64.
+ Known(Vec<usize>),
+}
+
+impl Int96Attribution {
+ /// Parses the stamp out of `schema`'s metadata and validates its leaf count against the
+ /// schema's own depth-first leaf count, so a stamp that does not describe this schema (a
+ /// crafted footer key, or a cached-metadata mismatch) degrades to [`Self::Unknown`].
+ fn from_schema(schema: &Schema) -> Self {
+ let Some(stamp) = schema.metadata().get(INT96_LEAVES_METADATA_KEY) else {
+ return Int96Attribution::Unknown;
+ };
+ let Some((count, ordinals)) = stamp.split_once(':') else {
+ return Int96Attribution::Unknown;
+ };
+ let schema_leaves: usize = schema
+ .fields()
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ if count.parse::<usize>().ok() != Some(schema_leaves) {
+ return Int96Attribution::Unknown;
+ }
+ let parsed: Option<Vec<usize>> = if ordinals.is_empty() {
+ Some(Vec::new())
+ } else {
+ ordinals
+ .split(',')
+ .map(|o| o.parse::<usize>().ok().filter(|o| *o < schema_leaves))
+ .collect()
+ };
+ match parsed {
+ Some(mut leaves) => {
+ leaves.sort_unstable();
+ Int96Attribution::Known(leaves)
+ }
+ None => Int96Attribution::Unknown,
+ }
+ }
+
+ /// `Some(true)` / `Some(false)` when the leaf is known to be INT96 / INT64, `None` when
+ /// the attribution is unknown.
+ fn is_int96(&self, leaf: usize) -> Option<bool> {
+ match self {
+ Int96Attribution::Unknown => None,
+ Int96Attribution::Known(leaves) => Some(leaves.binary_search(&leaf).is_ok()),
+ }
+ }
+}
+
+/// The [`INT96_LEAVES_METADATA_KEY`] value describing `schema`: its leaf count and the
+/// ordinals of its INT96 primitive columns.
+pub(crate) fn int96_leaf_stamp(schema: &SchemaDescriptor) -> String {
+ let ordinals: Vec<String> = schema
+ .columns()
+ .iter()
+ .enumerate()
+ .filter(|(_, column)| column.physical_type() == ParquetPhysicalType::INT96)
+ .map(|(ordinal, _)| ordinal.to_string())
+ .collect();
+ format!("{}:{}", schema.num_columns(), ordinals.join(","))
+}
+
+/// Returns a copy of `metadata` whose key-value metadata carries the [`int96_leaf_stamp`] of
+/// its own schema, or `None` when it already does (the common case after the first open of a
+/// file, since the caller caches the stamped copy). Any pre-existing entry under the key --
+/// a file cannot legitimately carry one -- is replaced, never trusted. Only the file-level
+/// key-value list changes; row groups and page indexes are carried over as-is. The parquet
+/// API cannot carry a file decryptor, nor `FileMetaData`'s crate-private encryption fields
+/// (encryption algorithm, footer signing key metadata), across this rebuild, so callers must
+/// not stamp opens that supply decryption properties -- and the only consumer, the Delta
+/// scan, declines every encrypted-parquet configuration before planning, so a parquet
+/// modular encryption file never reaches this path with or without those properties.
+pub(crate) fn stamp_int96_leaves(metadata: &ParquetMetaData) -> Option<ParquetMetaData> {
+ let file_metadata = metadata.file_metadata();
+ let stamp = int96_leaf_stamp(file_metadata.schema_descr());
+ let existing = file_metadata
+ .key_value_metadata()
+ .and_then(|kvs| kvs.iter().find(|kv| kv.key == INT96_LEAVES_METADATA_KEY))
+ .and_then(|kv| kv.value.as_deref());
+ if existing == Some(stamp.as_str()) {
+ return None;
+ }
+ let mut key_values: Vec<KeyValue> = file_metadata
+ .key_value_metadata()
+ .map(|kvs| {
+ kvs.iter()
+ .filter(|kv| kv.key != INT96_LEAVES_METADATA_KEY)
+ .cloned()
+ .collect()
+ })
+ .unwrap_or_default();
+ key_values.push(KeyValue::new(INT96_LEAVES_METADATA_KEY.to_string(), stamp));
+ let stamped_file_metadata = FileMetaData::new(
+ file_metadata.version(),
+ file_metadata.num_rows(),
+ file_metadata.created_by().map(str::to_string),
+ Some(key_values),
+ file_metadata.schema_descr_ptr(),
+ file_metadata.column_orders().cloned(),
+ );
+ Some(
+ ParquetMetaData::new(stamped_file_metadata, metadata.row_groups().to_vec())
+ .into_builder()
+ .set_column_index(metadata.column_index().cloned())
+ .set_offset_index(metadata.offset_index().cloned())
+ .build(),
+ )
+}
+
+/// Per-file rebase policies for the three affected column classes, plus the INT96
+/// attribution that selects between the two timestamp specs per leaf.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct FileRebasePolicies {
+ /// `DATE` columns, governed by `org.apache.spark.legacyDateTime` alone.
+ pub date: RebasePolicy,
+ /// INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns: the datetime spec (same
+ /// resolution as `date`), as Spark's `ParquetVectorUpdaterFactory` selects for INT64.
+ pub int64_timestamp: RebasePolicy,
+ /// INT96 columns: the INT96 spec (`org.apache.spark.legacyINT96`, min version 3.1.0).
+ pub int96_timestamp: RebasePolicy,
+ /// Which timestamp leaves are INT96. See [`Int96Attribution`].
+ pub int96_leaves: Int96Attribution,
+ /// Sorted depth-first leaf ordinals -- over the physical file schema, the same ordinals
+ /// `int96_leaves` uses -- that the query does not read: nested children the schema
+ /// adapter's struct narrowing drops before any value leaves the scan. Spark never decodes
+ /// them either, so their policy is the identity whatever the file's calendar. Empty until
+ /// [`Self::restrict_to_requested`] runs (every leaf requested).
+ pub unrequested_leaves: Vec<usize>,
+}
+
+impl FileRebasePolicies {
+ /// True when some policy is not the plain proleptic-Gregorian pass-through, i.e. when the
+ /// per-column wrap in [`wrap_datetime_rebase`] can install anything at all.
+ pub(crate) fn any_rebase_needed(&self) -> bool {
+ self.date != RebasePolicy::Corrected
+ || self.int64_timestamp != RebasePolicy::Corrected
+ || self.int96_timestamp != RebasePolicy::Corrected
+ }
+
+ fn is_requested(&self, leaf: usize) -> bool {
+ self.unrequested_leaves.binary_search(&leaf).is_err()
+ }
+
+ /// The policy of the `Date32` leaf at depth-first ordinal `leaf`: the file's date policy,
+ /// or the identity when the query does not read that leaf.
+ fn date_policy(&self, leaf: usize) -> RebasePolicy {
+ if self.is_requested(leaf) {
+ self.date
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+
+ /// The policy of the timezone-carrying timestamp leaf at depth-first ordinal `leaf`: the
+ /// identity when the query does not read it; otherwise its physical type's spec when the
+ /// attribution is known, or else the two specs merged -- agreement decides, disagreement
+ /// degrades to [`RebasePolicy::CheckAncient`], which still passes every modern value and
+ /// refuses only ancient ones.
+ fn timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ Some(false) => self.int64_timestamp,
+ None if self.int64_timestamp == self.int96_timestamp => self.int64_timestamp,
+ None => RebasePolicy::CheckAncient,
+ }
+ }
+
+ /// These policies with every physical leaf the query does not read marked the identity.
+ /// `requested` pairs each top-level field of `physical_schema` (by position) with the type
+ /// of the logical field the schema adapter narrows it to -- `None` for a column without a
+ /// logical counterpart, whose leaves are left as they are (no expression reads it anyway).
+ /// Nested children pair the way the adapter's struct convert selects them (see
+ /// [`push_unrequested_leaves`]); the INT96 attribution is untouched, since the ordinals
+ /// stay physical. `requested` is parallel to the schema's fields; should a caller pass a
+ /// shorter slice, the trailing columns simply keep every leaf (the safe direction).
+ pub(crate) fn restrict_to_requested(
+ mut self,
+ physical_schema: &Schema,
+ requested: &[Option<&DataType>],
+ case_sensitive: bool,
+ use_field_id: bool,
+ ) -> Self {
+ debug_assert_eq!(requested.len(), physical_schema.fields().len());
+ let matching = FieldMatching {
+ case_sensitive,
+ use_field_id,
+ };
+ let mut next_leaf = 0;
+ let mut unrequested = Vec::new();
+ for (field, requested) in physical_schema.fields().iter().zip(requested) {
+ match requested {
+ Some(logical) => push_unrequested_leaves(
+ field.data_type(),
+ logical,
+ &mut next_leaf,
+ matching,
+ &mut unrequested,
+ ),
+ None => next_leaf += leaf_count(field.data_type()),
+ }
+ }
+ // Emitted in depth-first order, so already sorted for `is_requested`'s binary search.
+ self.unrequested_leaves = unrequested;
+ self
+ }
+}
+
+/// The field-matching rules of the schema adapter's nested narrowing
+/// (`parquet_convert_struct_to_struct`): names fold per `case_sensitive`, and Parquet field ids
+/// select fields when `use_field_id` is set.
+#[derive(Debug, Clone, Copy)]
+struct FieldMatching {
+ case_sensitive: bool,
+ use_field_id: bool,
+}
+
+/// Appends to `out` the depth-first leaf ordinals of `physical` (counting from `next_leaf`,
+/// which advances past every leaf of `physical`) that reading it as `requested` drops.
+///
+/// Recurses through exactly the pairings `parquet_convert_array` narrows, and no others: a
+/// struct child is dropped only when NO requested child selects it by either rule the struct
+/// convert uses -- folded name, or Parquet field id when ids are in play -- and an ambiguous
+/// child (several requested children select it) is kept; `List` pairs with `List` by element
+/// type, and `Map` with a `Map` of the same key ordering by its entries, positionally. Any
+/// other pairing -- a leaf, a `LargeList` / `FixedSizeList` / dictionary, a map whose ordering
+/// differs, or a shape mismatch -- is handed to arrow's cast or passed through whole by the
+/// convert, so it keeps every leaf. Keeping a superset of what the narrowing reads is always
+/// safe (a spurious check at worst); dropping a leaf the narrowing reads would skip its
+/// rebase, so every doubt resolves to "requested".
+fn push_unrequested_leaves(
+ physical: &DataType,
+ requested: &DataType,
+ next_leaf: &mut usize,
+ matching: FieldMatching,
+ out: &mut Vec<usize>,
+) {
+ match (physical, requested) {
+ (DataType::Struct(physical_fields), DataType::Struct(requested_fields)) => {
+ let names: Vec<&str> = physical_fields
+ .iter()
+ .chain(requested_fields.iter())
+ .map(|f| f.name().as_str())
+ .collect();
+ // A fold failure means the names could not be compared at all; keeping every leaf
+ // requested is the safe superset, the same as the pass-through pairings below.
+ let Ok(folded) = fold_names(&names, matching.case_sensitive) else {
+ *next_leaf += leaf_count(physical);
+ return;
+ };
+ let (physical_folded, requested_folded) = folded.split_at(physical_fields.len());
+ for (i, child) in physical_fields.iter().enumerate() {
+ let child_id = if matching.use_field_id {
+ parse_field_id(child)
+ } else {
+ None
+ };
+ let mut selectors = requested_fields.iter().enumerate().filter(|(j, r)| {
+ requested_folded[*j] == physical_folded[i]
+ || (child_id.is_some() && parse_field_id(r) == child_id)
+ });
+ match (selectors.next(), selectors.next()) {
+ (None, _) => {
+ let n = leaf_count(child.data_type());
+ out.extend(*next_leaf..*next_leaf + n);
+ *next_leaf += n;
+ }
+ (Some((_, requested_child)), None) => push_unrequested_leaves(
+ child.data_type(),
+ requested_child.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (Some(_), Some(_)) => *next_leaf += leaf_count(child.data_type()),
+ }
+ }
+ }
+ (DataType::List(physical_item), DataType::List(requested_item)) => push_unrequested_leaves(
+ physical_item.data_type(),
+ requested_item.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (
+ DataType::Map(physical_entries, physical_sorted),
+ DataType::Map(requested_entries, requested_sorted),
+ ) if physical_sorted == requested_sorted => {
+ match (physical_entries.data_type(), requested_entries.data_type()) {
+ (DataType::Struct(physical_kv), DataType::Struct(requested_kv))
+ if physical_kv.len() == requested_kv.len() =>
+ {
+ for (p, r) in physical_kv.iter().zip(requested_kv.iter()) {
+ push_unrequested_leaves(
+ p.data_type(),
+ r.data_type(),
+ next_leaf,
+ matching,
+ out,
+ );
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+}
+
+/// The writer time zone recorded in `metadata`, classified for timestamp rebasing. Mirrors the
+/// `Option(lookupFileMeta(SPARK_TIMEZONE_METADATA_KEY))` lookup Spark's `getRebaseSpec` performs
+/// for every LEGACY resolution, conf-fallback included; Spark substitutes the JVM default zone
+/// when the key is absent (`RebaseSpec.timeZone`), which is unavailable natively, so an absent or
+/// non-UTC zone classifies as [`WriterTimeZone::OtherOrUnknown`] (dates still rebase fully --
+/// the day rebase is zone-free -- while ancient timestamps refuse rather than guess).
+fn writer_time_zone(metadata: &HashMap<String, String>) -> WriterTimeZone {
+ match metadata.get(SPARK_TIMEZONE_KEY) {
+ Some(tz) if UTC_EQUIVALENT_TIMEZONES.contains(&tz.as_str()) => WriterTimeZone::Utc,
+ _ => WriterTimeZone::OtherOrUnknown,
+ }
+}
+
+/// One spec resolution, mirroring Spark's `DataSourceUtils.getRebaseSpec` exactly: a Spark
+/// version below `min_version` (lexicographic comparison, same as the Scala `String.<`) or a
+/// present legacy flag means LEGACY; a Spark version at/after `min_version` without the flag
+/// means CORRECTED; no Spark version at all falls back to `conf_mode`, the session read conf
+/// forwarded from the JVM (`getRebaseSpec`'s `modeByConfig` fallback, its ONLY use of the
+/// conf): CORRECTED passes values through, LEGACY rebases (with the writer zone from the
+/// file's `org.apache.spark.timeZone` key, same lookup as the metadata-driven LEGACY path),
+/// and EXCEPTION refuses ancient values as [`RebasePolicy::CheckAncient`].
+fn resolve_spec(
+ metadata: &HashMap<String, String>,
+ min_version: &str,
+ legacy_key: &str,
+ conf_mode: RebaseReadMode,
+) -> RebasePolicy {
+ match metadata.get(SPARK_VERSION_METADATA_KEY) {
+ None => match conf_mode {
+ RebaseReadMode::Corrected => RebasePolicy::Corrected,
+ RebaseReadMode::Legacy => RebasePolicy::Legacy(writer_time_zone(metadata)),
+ RebaseReadMode::Exception => RebasePolicy::CheckAncient,
+ },
+ Some(version) => {
+ if version.as_str() < min_version || metadata.contains_key(legacy_key) {
+ RebasePolicy::Legacy(writer_time_zone(metadata))
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+ }
+}
+
+/// Resolves the per-file rebase policies from a file's arrow schema: the parquet footer's
+/// key-value pairs in its metadata decide the specs (the datetime spec uses min version
+/// `3.0.0` and the INT96 spec `3.1.0`, matching `DataSourceUtils.datetimeRebaseSpec` /
+/// `int96RebaseSpec`; `session_modes` supplies the per-spec conf fallback for files without
+/// Spark writer metadata), and the reader factory's INT96 stamp -- validated against the
+/// schema's leaf structure -- attributes each timestamp leaf to its spec.
+pub(crate) fn resolve_file_rebase_policies(
+ physical_file_schema: &Schema,
+ session_modes: SessionRebaseModes,
+) -> FileRebasePolicies {
+ let metadata = physical_file_schema.metadata();
+ let datetime_spec = resolve_spec(
+ metadata,
+ "3.0.0",
+ SPARK_LEGACY_DATETIME_KEY,
+ session_modes.datetime,
+ );
+ let int96_spec = resolve_spec(
+ metadata,
+ "3.1.0",
+ SPARK_LEGACY_INT96_KEY,
+ session_modes.int96,
+ );
+ FileRebasePolicies {
+ date: datetime_spec,
+ int64_timestamp: datetime_spec,
+ int96_timestamp: int96_spec,
+ int96_leaves: Int96Attribution::from_schema(physical_file_schema),
+ unrequested_leaves: Vec::new(),
+ }
+}
+
+/// Number of primitive leaves `dt` contains in a depth-first walk -- the same count and order
+/// parquet-rs uses when it maps the file's `SchemaDescriptor` columns onto the arrow schema, so
+/// arrow-side leaf ordinals line up with [`int96_leaf_stamp`]'s.
+fn leaf_count(dt: &DataType) -> usize {
+ match dt {
+ DataType::Struct(fields) => fields.iter().map(|f| leaf_count(f.data_type())).sum(),
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_count(f.data_type()),
+ DataType::Dictionary(_, value) => leaf_count(value),
+ DataType::RunEndEncoded(_, value) => leaf_count(value.data_type()),
+ DataType::Union(fields, _) => fields.iter().map(|(_, f)| leaf_count(f.data_type())).sum(),
+ _ => 1,
+ }
+}
+
+/// Appends the policy of every leaf of `dt`, in depth-first order, to `out`, consuming leaf
+/// ordinals from `next_leaf` (exactly [`leaf_count`] of them). Only `Date32` and
+/// timezone-carrying timestamps have a policy to apply, and only when the query reads the
+/// leaf; timezone-free timestamps are `TIMESTAMP_NTZ`, which Spark never rebases, and every
+/// other leaf is the identity ([`RebasePolicy::Corrected`]).
+fn leaf_policies(
+ dt: &DataType,
+ next_leaf: &mut usize,
+ policies: &FileRebasePolicies,
+ out: &mut Vec<RebasePolicy>,
+) {
+ match dt {
+ DataType::Date32 => {
+ out.push(policies.date_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Timestamp(_, Some(_)) => {
+ out.push(policies.timestamp_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Struct(fields) => {
+ for f in fields {
+ leaf_policies(f.data_type(), next_leaf, policies, out);
+ }
+ }
+ // Mirrors `leaf_count` variant for variant, so a rebase-affected leaf inside a nested
+ // type `rebase_array` cannot rebuild (views, run-end, union -- never produced from a
+ // parquet schema) still gets its real policy and makes `rebase_array` refuse loudly
+ // instead of being stamped the identity.
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_policies(f.data_type(), next_leaf, policies, out),
+ DataType::Dictionary(_, value) => leaf_policies(value, next_leaf, policies, out),
+ DataType::RunEndEncoded(_, value) => {
+ leaf_policies(value.data_type(), next_leaf, policies, out)
+ }
+ DataType::Union(fields, _) => {
+ for (_, f) in fields.iter() {
+ leaf_policies(f.data_type(), next_leaf, policies, out);
+ }
+ }
+ _ => {
+ *next_leaf += 1;
+ out.push(RebasePolicy::Corrected);
+ }
+ }
+}
+
+/// Wraps every column reference in `expr` whose physical file type contains a rebase-affected
+/// leaf under a policy that needs handling with a [`SparkDatetimeRebaseExpr`] carrying that
+/// column's per-leaf policies, so both the per-file projection and the pushed-down predicate
+/// evaluate rebased values. Columns whose leaves are all the identity -- unaffected types,
+/// affected types under [`RebasePolicy::Corrected`], or leaves the query does not read (see
+/// [`FileRebasePolicies::restrict_to_requested`]) -- pass through unwrapped. (The pruning
+/// predicates derived from the wrapped predicate treat the wrapper as an opaque expression and
+/// skip pruning on those columns -- conservative, since file-level statistics are in the
+/// file's own calendar.)
+pub(crate) fn wrap_datetime_rebase(
+ expr: Arc<dyn PhysicalExpr>,
+ physical_schema: &SchemaRef,
+ policies: &FileRebasePolicies,
+) -> DataFusionResult<Arc<dyn PhysicalExpr>> {
+ expr.transform(|e| {
+ let Some(col) = e.downcast_ref::<Column>() else {
+ return Ok(Transformed::no(e));
+ };
+ // Missing columns were already replaced with literals; any surviving reference is
+ // physical-schema-indexed. Out-of-range means a non-file column (defensive): skip.
+ let Some(field) = physical_schema.fields().get(col.index()) else {
+ return Ok(Transformed::no(e));
+ };
+ // This column's first leaf ordinal: the leaves of every preceding top-level field.
+ let mut next_leaf: usize = physical_schema.fields()[..col.index()]
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ let mut column_leaf_policies = Vec::with_capacity(leaf_count(field.data_type()));
+ leaf_policies(
+ field.data_type(),
+ &mut next_leaf,
+ policies,
+ &mut column_leaf_policies,
+ );
+ if column_leaf_policies
+ .iter()
+ .all(|p| *p == RebasePolicy::Corrected)
+ {
+ return Ok(Transformed::no(e));
+ }
+ Ok(Transformed::yes(Arc::new(SparkDatetimeRebaseExpr {
+ child: e,
+ field: Arc::clone(field),
+ leaf_policies: column_leaf_policies,
+ }) as Arc<dyn PhysicalExpr>))
+ })
+ .map(|t| t.data)
+}
+
+/// Applies a file's calendar-rebase policies to one column: rebases exactly where possible,
+/// raises on ancient values it cannot rebase, and passes modern values (the identity under
+/// every policy) through untouched. Nested columns are rebuilt leaf by leaf with nulls and
+/// offsets preserved. See the module doc for the policy table.
+#[derive(Debug, Eq)]
+struct SparkDatetimeRebaseExpr {
+ child: Arc<dyn PhysicalExpr>,
+ /// The physical file field this expression reads (type preserved by the rebase).
+ field: FieldRef,
+ /// One policy per primitive leaf of `field`'s type, in depth-first order (a single entry
+ /// for a flat column). At least one is not [`RebasePolicy::Corrected`].
+ leaf_policies: Vec<RebasePolicy>,
+}
+
+impl SparkDatetimeRebaseExpr {
+ /// The refusal error, as an [`ArrowError`] so `try_unary` closures can raise it directly;
+ /// it converts into a `DataFusionError` at the `?` in `evaluate`.
+ fn rebase_error(&self, detail: &str) -> ArrowError {
+ ArrowError::ComputeError(format!(
+ "Native scan cannot rebase ancient values in column '{}': the file was written \
+ with the legacy (hybrid Julian/Gregorian) calendar, or does not declare which \
+ calendar it used, and {detail}. Reading it natively would return silently \
+ shifted values; disable the native Delta scan \
+ (spark.comet.scan.delta.enabled=false) to let Spark read this table",
+ self.field.name(),
+ ))
+ }
+
+ fn internal_error(&self, detail: impl Display) -> DataFusionError {
+ DataFusionError::Internal(format!(
+ "SparkDatetimeRebaseExpr on column '{}': {detail}",
+ self.field.name()
+ ))
+ }
+
+ /// Rebases a timestamp column written at a fixed zero-offset zone: shift the nominal day
+ /// with the exact date table, keep the time of day. Matches Spark's
+ /// `rebaseJulianToGregorianMicros` for UTC, where the hybrid calendar's day boundaries sit
+ /// exactly on multiples of a day and no timezone transition can apply (UTC's last switch
+ /// instant in Spark's rebase table is the 1582-10-15 cutover itself).
+ fn rebase_timestamp_utc(&self, v: i64, units_per_day: i64) -> Result<i64, ArrowError> {
+ // Compare in days, not units: the cutover day times a nanosecond day does not fit i64.
+ let day = v.div_euclid(units_per_day);
+ if day >= LAST_SWITCH_JULIAN_DAY as i64 {
+ return Ok(v);
+ }
+ let time_of_day = v - day * units_per_day;
+ let day = i32::try_from(day).map_err(|_| {
+ self.rebase_error("the value is outside the rebaseable timestamp range")
+ })?;
+ let rebased = rebase_julian_to_gregorian_days(day) as i64;
+ rebased
+ .checked_mul(units_per_day)
+ .and_then(|d| d.checked_add(time_of_day))
+ .ok_or_else(|| self.rebase_error("the rebased value overflows the timestamp range"))
+ }
+
+ /// Whether `array` holds no null and no value before the Gregorian cutover, so no rebase
+ /// and no rejection applies and the batch can pass through untouched. The vectorised
+ /// minimum makes this one cheap pass on null-free values; arrays with nulls take the
+ /// per-value path below, which skips nulls without a second pass.
+ fn all_modern<T: ArrowPrimitiveType<Native = i64>>(
+ array: &PrimitiveArray<T>,
+ cutover: i64,
+ ) -> bool {
+ array.null_count() == 0 && arrow::compute::min(array).is_none_or(|min| min >= cutover)
Review Comment:
> Keeping the vectorised `min` only when the array is null-free, and falling back to the validity-aware loop this commit replaced, gets the best of both ... Could that assertion go back?
Done in f0351b9. `all_modern` takes the hybrid for both the timestamp and the date arm, and the pass-through test again asserts `Arc::ptr_eq` for the masked dates, plus a masked timestamp batch with an ancient value under the null slot, under all three policies. The test fails on the old guard and passes now; the module's 41 tests and clippy are clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4207414827
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so its rewrite (via
+ * [[CometScanWithPlanData]]) survives plan copies -- a transient field would be dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning (the CometIcebergNativeScanExec lesson).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node
+ // automatically gets a helper consistent with ITS runtimeFilters, avoiding the #3510 class of
+ // bug where a stored helper field desyncs from rewritten filters. Costs one extra file listing
+ // per executed instance; correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below, and AQE's UI
+ // plan-walk calls `.metrics` on every node MID-PLANNING, including while a DPP subquery is
+ // still an adaptive placeholder or a partition filter holds an unresolved ScalarSubquery (see
+ // `hasUnevaluableSubqueryFilter` below). That's safe ONLY because constructing `scanHelper` is a
+ // cheap case-class build with no file listing, and core's `CometScanExec.metrics` touches only
+ // `wrapped.driverMetrics` (populated by Spark's own planning) plus a static metric-node
+ // constructor -- neither file listing nor subquery resolution. If core's `metrics` ever touches
+ // either, forcing `scanHelper` here would resurrect the AQE mid-planning crashes this invariant
+ // prevents.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element, so checking runtimeFilters
+ // here avoids constructing/forcing the derived scanHelper just to read partitioning. The
+ // InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep in sync. The
+ // ScalarSubquery case is probed rather than treated as permanently unevaluable: Spark exposes no
+ // public finished/updated flag on ExecSubqueryExpression, but `eval()` doubles as one -- it only
+ // reads the cached `result` behind a `require(updated, ...)` guard, while the subquery is
+ // actually run by `updateResult()` (invoked separately during prepare/AQE), never by `eval()`.
+ // Once resolved, outputPartitioning below reports the real perPartitionData.length instead of
+ // staying at zero -- a fused native parent's buildNativeContext requires that count to match.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case s: ScalarSubquery => !isScalarSubqueryResolved(s)
+ case _ => false
+ })
+
+ // `eval()` never triggers the subquery's execution: on a resolved subquery it is a pure cached
+ // read of `result` (verified against bytecode: `Predef.require(updated(), ...)` then a plain
+ // field read), so this probe is safe to call repeatedly, including from AQE's mid-planning plan
+ // walks. Pre-resolution, the ONLY throw is `require`'s `IllegalArgumentException`; catch exactly
+ // that, since anything else escaping is a genuine bug we must not mask as unpartitioned.
+ private def isScalarSubqueryResolved(s: ScalarSubquery): Boolean =
+ try {
+ s.eval()
+ true
+ } catch {
+ case _: IllegalArgumentException => false
+ }
+
+ private def isAdaptivePlaceholder(p: SparkPlan): Boolean = p match {
+ case ReusedSubqueryExec(inner) => isAdaptivePlaceholder(inner)
+ case _: CometSubqueryAdaptiveBroadcastExec => true
+ case _: SubqueryAdaptiveBroadcastExec => true
+ case _ => false
+ }
+
+ override lazy val outputOrdering: Seq[SortOrder] = originalPlan.outputOrdering
+
+ override def dynamicPruningFilters: Seq[Expression] = runtimeFilters
+
+ override def withDynamicPruningFilters(filters: Seq[Expression]): SparkPlan = {
+ // A real copy: runtimeFilters is a constructor field included in equality, so the copy
+ // survives enclosing-block rebuilds, and the derived scanHelper picks up the rewritten
+ // filters automatically.
+ copy(runtimeFilters = filters)
Review Comment:
> Could this scan override `sparkFallback`, following `CometNativeScanExec`, to preserve the current partition filters and logical link?
Done in 2fe0af902. `CometDeltaNativeScanExec.sparkFallback` now restores `originalPlan` with `runtimeFilters` as its partition filters, which are the same filters AQE rewrote on the live scan, and carries over the logical link. Data filters stay as the original scan had them: the live list also holds subquery predicates taken from the parent `Filter`, which Spark left out of its own scan.
`CometDeltaNativeScanSuite` has your query as a regression, with DPP on and transition reversion forced, run with AQE both on and off. The AQE run failed with the `SubqueryAdaptiveBroadcastExec` error before the change. It now matches Spark and checks that the restored scan is a `FileSourceScanExec` whose pruning subquery is an executable broadcast, not the adaptive placeholder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4162650461
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1913 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ // Unlike core, a required Variant root stays declined: this path lacks core's Variant gates.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
Review Comment:
[P2] Preserve the fallback for split-dependent metadata. This allowlist accepts `file_block_start` and `file_block_length`, and Delta claims the scan before core’s metadata guard runs. For a split Delta file, Spark assigns a row group to the split containing its midpoint, whereas DataFusion uses its first page. Consequently, selecting `_metadata.file_block_start` returns incorrect values: the reproduced 5,000-row table reports `20480` through Spark/Delta and `0` through the current native reader. The base kept these Delta reads on Spark. Please explicitly decline these two metadata columns, as core already does, until native split attribution matches Spark.
Evidence: With Spark 3.5.9/Delta 3.3.2, write `spark.range(0,5000).selectExpr('id', "concat('value_', cast(id as string)) as s").coalesce(1)` as Delta, set `spark.sql.files.maxPartitionBytes=4096`, and select `id, _metadata.file_block_start, _metadata.file_block_length`. Spark produced 12 partitions and `(start=20480, length=4096, count=5000)`. A disposable test calling the exact-head `init_datasource_exec` on the same file and 4096-byte ranges, with the partition constants supplied by `partition2Proto`, produced `(start=0, length=4096, rows=5000)`. Logs: `/tmp/5365-a2df-block-spark.log` and `/tmp/5365-a2df-current-block-native.log`. Delta admission and serialization were source-traced; full Comet/JNI execution was not run.
##########
native/core/src/parquet/datetime_rebase.rs:
##########
@@ -0,0 +1,3192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Per-file datetime calendar-rebase handling for the parquet scan.
+//!
+//! Spark 2.4 and earlier wrote dates and timestamps in the hybrid Julian + Gregorian calendar;
+//! Spark 3.0+ uses the proleptic Gregorian calendar and records the calendar policy of every
+//! file it writes in the parquet footer's key-value metadata (`org.apache.spark.version`,
+//! `org.apache.spark.legacyDateTime`, `org.apache.spark.legacyINT96`,
+//! `org.apache.spark.timeZone`). Spark's reader resolves the rebase policy from EACH FILE's
+//! writer metadata (`DataSourceUtils.datetimeRebaseSpec` / `int96RebaseSpec`) -- the session's
+//! `spark.sql.parquet.datetimeRebaseModeInRead` conf only applies to files whose metadata does
+//! not decide the policy on its own -- so a reader that ignores the metadata silently returns
+//! values shifted by up to ten days for dates before 1582-10-15 (e.g. `1500-01-01` reads as
+//! `1500-01-10`).
+//!
+//! This module mirrors that per-file resolution: [`resolve_file_rebase_policies`] computes the
+//! date / INT64-timestamp / INT96-timestamp policies from a file's arrow schema metadata (the
+//! parquet key-value pairs survive the parquet -> arrow schema conversion), and
+//! [`wrap_datetime_rebase`] wraps the per-file rewritten expressions' column references in a
+//! [`SparkDatetimeRebaseExpr`] that rebases values exactly where that is possible without the
+//! JVM's historical timezone tables (dates always; timestamps for a fixed UTC writer zone) and
+//! refuses -- rather than silently corrupting -- ancient values it cannot rebase. Nested
+//! columns are rebuilt leaf by leaf (struct / list / map / fixed-size list / dictionary), each
+//! leaf under its own policy, with nulls and offsets preserved. Modern values are always the
+//! identity under every policy: from 1582-10-15 onward for dates, and from
+//! [`LAST_SWITCH_JULIAN_TS_SECONDS`] (1900-01-01T00:00:00Z, Spark's
+//! `RebaseDateTime.lastSwitchJulianTs`) onward for timestamps.
+//!
+//! Spark applies `datetimeRebaseSpec` to INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns
+//! and `int96RebaseSpec` to INT96 columns. The two physical types are indistinguishable in the
+//! arrow schema DataFusion hands the expression adapter (both surface as `Timestamp(us, "UTC")`
+//! after INT96 coercion), so Comet's parquet reader factory stamps the file's INT96 leaf
+//! ordinals -- taken from the parquet footer's own `SchemaDescriptor` -- into the key-value
+//! metadata under [`INT96_LEAVES_METADATA_KEY`] before the arrow schema is derived (see
+//! [`stamp_int96_leaves`] and `eager_page_index_reader_factory.rs`), and the adapter attributes
+//! every timestamp leaf to its spec from that stamp. Without a stamp, the two specs are merged:
+//! agreement decides, disagreement degrades to [`RebasePolicy::CheckAncient`].
+//!
+//! The wrapper sits BENEATH the schema adapter's nested narrowing (the struct -> struct convert
+//! that keeps only the requested children), which is what keeps those ordinals physical -- but
+//! it means the wrapper sees every physical child, requested or not. Spark only ever decodes
+//! the requested nested schema, so [`FileRebasePolicies::restrict_to_requested`] marks the
+//! physical leaves the narrowing drops as the identity: an unrequested ancient `s.ts` never
+//! blocks `select s.d`, exactly as in Spark.
+//!
+//! The same pairing decides a timestamp leaf's policy by the type the query READS it as, since
+//! `ParquetVectorUpdaterFactory.getUpdater` keys on the requested Spark type, not the parquet
+//! annotation: a leaf read as `TIMESTAMP_NTZ` never rebases (INT96 or INT64; Spark 4.x's
+//! `BinaryToSQLTimestampUpdater` / `LongUpdater` consult no mode, and Spark 3.x refuses the
+//! INT96 and adjusted-INT64 pairings outright, which Comet's `allow_timestamp_ltz_to_ntz` gate
+//! reproduces); a leaf read as `TIMESTAMP` rebases under the datetime spec even when the file
+//! declares it `isAdjustedToUTC=false` (`isTimestampTypeMatched` checks the unit only); and a
+//! `DATE` leaf keeps the date policy whether read as `DATE` or, on Spark 4.x, as
+//! `TIMESTAMP_NTZ` (`DateToTimestampNTZWithRebaseUpdater`).
+//!
+//! Currently only enabled by the Delta scan arms via
+//! `SparkParquetOptions::rebase_from_file_metadata`, which also carries the session read modes
+//! ([`SessionRebaseModes`], forwarded from the JVM) that decide the policy for files without
+//! Spark writer metadata; the plain NativeScan keeps its documented no-rebase behavior (see
+//! the compatibility guide and issue #5010).
+
+use std::collections::HashMap;
+use std::fmt::{self, Display};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Date32Array, FixedSizeListArray, GenericListArray, MapArray,
+ OffsetSizeTrait, PrimitiveArray, RecordBatch, StructArray,
+};
+use arrow::datatypes::{
+ ArrowPrimitiveType, ArrowTimestampType, DataType, Date32Type, FieldRef, Schema, SchemaRef,
+ TimeUnit, TimestampMicrosecondType, TimestampMillisecondType, TimestampNanosecondType,
+ TimestampSecondType,
+};
+use arrow::error::ArrowError;
+use datafusion::common::tree_node::{Transformed, TreeNode};
+use datafusion::common::{DataFusionError, Result as DataFusionResult};
+use datafusion::physical_expr::expressions::Column;
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::ColumnarValue;
+use parquet::basic::Type as ParquetPhysicalType;
+use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaData};
+use parquet::schema::types::SchemaDescriptor;
+
+use super::name_fold::fold_names;
+use super::parquet_support::field_id;
+
+/// Footer key naming the Spark release that wrote the file; absent for non-Spark writers.
+const SPARK_VERSION_METADATA_KEY: &str = "org.apache.spark.version";
+/// Present (empty value) when the file's dates and INT64 timestamps were written with
+/// `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_DATETIME_KEY: &str = "org.apache.spark.legacyDateTime";
+/// Present (empty value) when the file's INT96 timestamps were written with
+/// `spark.sql.parquet.int96RebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_INT96_KEY: &str = "org.apache.spark.legacyINT96";
+/// The writer session's time zone, stamped alongside either legacy flag.
+const SPARK_TIMEZONE_KEY: &str = "org.apache.spark.timeZone";
+
+/// Key-value metadata entry Comet's parquet reader factory adds to a file's footer metadata
+/// (in memory only, never written back) so the expression adapter can tell INT96 timestamp
+/// columns from INT64 ones after both have been coerced to the same arrow type. Value:
+/// `"<leaf count>:<comma-separated INT96 leaf ordinals>"`, where leaves are the file's
+/// primitive columns in `SchemaDescriptor::columns()` order -- the same depth-first order
+/// parquet-rs assigns arrow leaves, so an arrow-side depth-first walk lines up with it. The
+/// leaf count lets the reader detect a stamp that does not describe the schema it is paired
+/// with (see [`Int96Attribution::from_schema`]).
+pub(crate) const INT96_LEAVES_METADATA_KEY: &str = "comet.int96_leaf_columns";
+
+/// Day of the Gregorian cutover (1582-10-15) as days since the epoch; rebasing is the identity
+/// from this day onward. Same value as Spark's `RebaseDateTime.lastSwitchJulianDay`.
+const LAST_SWITCH_JULIAN_DAY: i32 = -141427;
+
+/// Spark's `RebaseDateTime.lastSwitchJulianTs` (and `lastSwitchGregorianTs`) in seconds since
+/// the epoch: 1900-01-01T00:00:00Z. Spark derives it as the latest switch instant across every
+/// zone in its `julian-gregorian-rebase-micros.json` table (`getLastSwitchTs`, which also
+/// asserts the calendars' difference is zero for every zone from then on): most zones ran on
+/// local mean time before 1900, so the last instant at which rebasing changes a value in ANY
+/// zone is 1900-01-01T00:00:00Z, not the 1582 cutover. `createTimestampRebaseFuncInRead`
+/// under `EXCEPTION` throws exactly for `micros < lastSwitchJulianTs` (after converting
+/// `TIMESTAMP_MILLIS` to micros), and `rebaseJulianToGregorianMicros` is the identity from it
+/// onward in every zone. The value is in seconds so it scales exactly to any timestamp unit.
+pub(crate) const LAST_SWITCH_JULIAN_TS_SECONDS: i64 = -2_208_988_800;
+
+/// The per-century differences between the Julian and proleptic Gregorian calendars, and the
+/// Julian-calendar switch days at which each difference starts to apply. Copied verbatim from
+/// Spark's `RebaseDateTime.julianGregDiffs` / `julianGregDiffSwitchDay` (which Spark generated
+/// from `localRebaseJulianToGregorianDays`); `rebase_julian_to_gregorian_days` must stay
+/// value-for-value equal to Spark's `rebaseJulianToGregorianDays`.
+const JULIAN_GREG_DIFFS: [i32; 14] = [2, 1, 0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, 0];
+const JULIAN_GREG_DIFF_SWITCH_DAY: [i32; 14] = [
+ -719164, -682945, -646420, -609895, -536845, -500320, -463795, -390745, -354220, -317695,
+ -244645, -208120, -171595, -141427,
+];
+
+/// Proleptic-Gregorian days since 1970-01-01 for a nominal civil date, via Howard Hinnant's
+/// `days_from_civil`. `d` may exceed the month's length; the excess rolls into the following
+/// month exactly like `LocalDate.of(y, m, 1).plusDays(d - 1)` in Spark's
+/// `localRebaseJulianToGregorianDays` (how the non-existent proleptic date `1000-02-29`,
+/// valid in the Julian calendar, lands on `1000-03-01`).
+fn days_from_civil(y: i64, m: i64, d: i64) -> i64 {
+ let y = if m <= 2 { y - 1 } else { y };
+ let era = y.div_euclid(400);
+ let yoe = y - era * 400; // [0, 399]
+ let mp = (m + 9) % 12; // [0, 11], March = 0
+ let doy = (153 * mp + 2) / 5 + d - 1;
+ let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
+ era * 146097 + doe - 719468
+}
+
+/// Julian-calendar civil date `(year, month, day)` for a day count since 1970-01-01 that labels
+/// days in the Julian calendar (astronomical year numbering: 1 BCE is year 0). Standard
+/// Julian-day-number conversion (E.G. Richards' algorithm), exact for any day.
+fn julian_day_to_civil(days: i64) -> (i64, i64, i64) {
+ // Integer (noon) Julian Day Number of this civil day: 1970-01-01 is JDN 2440588.
+ let jdn = days + 2_440_588;
+ let f = jdn + 1401;
+ let e = 4 * f + 3;
+ let g = e.rem_euclid(1461) / 4;
+ let h = 5 * g + 2;
+ let day = h.rem_euclid(153) / 5 + 1;
+ let month = (h / 153 + 2).rem_euclid(12) + 1;
+ let year = e.div_euclid(1461) - 4716 + (14 - month) / 12;
+ (year, month, day)
+}
+
+/// Exact port of Spark's `RebaseDateTime.rebaseJulianToGregorianDays`: reinterprets a day count
+/// written in the hybrid Julian + Gregorian calendar as the proleptic Gregorian day count of the
+/// same nominal civil date. Identity for days from 1582-10-15 onward. Days before the tables'
+/// range (before Julian `0001-01-01`) take the calendar-arithmetic path, mirroring Spark's
+/// `localRebaseJulianToGregorianDays` fallback.
+pub(crate) fn rebase_julian_to_gregorian_days(days: i32) -> i32 {
+ if days < JULIAN_GREG_DIFF_SWITCH_DAY[0] {
+ let (y, m, d) = julian_day_to_civil(days as i64);
+ (days_from_civil(y, m, 1) + (d - 1)) as i32
+ } else {
+ // Spark's rebaseDays: linear search from the most recent switch day.
+ let mut i = JULIAN_GREG_DIFF_SWITCH_DAY.len();
+ loop {
+ i -= 1;
+ if i == 0 || days >= JULIAN_GREG_DIFF_SWITCH_DAY[i] {
+ break;
+ }
+ }
+ days + JULIAN_GREG_DIFFS[i]
+ }
+}
+
+/// Timezone strings from `org.apache.spark.timeZone` that denote a fixed zero-offset zone in
+/// both `java.util.TimeZone` and `java.time`. Only for these is timestamp rebasing the pure
+/// nominal-date shift [`SparkDatetimeRebaseExpr::rebase_timestamp_utc`] computes; any other (or
+/// absent) zone needs the JVM's historical timezone tables and stays on the
+/// refuse-ancient-values path.
+const UTC_EQUIVALENT_TIMEZONES: [&str; 6] = ["UTC", "Etc/UTC", "GMT", "Etc/GMT", "Z", "+00:00"];
+
+/// How the writer's session time zone (if recorded) affects timestamp rebasing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum WriterTimeZone {
+ /// A fixed zero-offset zone: rebasing reduces to the exact nominal-date shift.
+ Utc,
+ /// Any other zone, or none recorded (pre-3.0 files): ancient values cannot be rebased
+ /// without the JVM's historical timezone data.
+ OtherOrUnknown,
+}
+
+/// One session-level datetime rebase read mode (a `LegacyBehaviorPolicy` value of
+/// `spark.sql.parquet.datetimeRebaseModeInRead` / `int96RebaseModeInRead`), consulted by
+/// [`resolve_file_rebase_policies`] ONLY for files whose footer metadata does not decide the
+/// policy on its own -- exactly the `getOrElse` fallback in Spark's
+/// `DataSourceUtils.getRebaseSpec`. Files that carry `org.apache.spark.version` ignore these
+/// modes entirely, on every Spark version.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) enum RebaseReadMode {
+ /// Refuse ancient values (Spark raises `SparkUpgradeException`); maps to
+ /// [`RebasePolicy::CheckAncient`]. The default mirrors the conservative posture used
+ /// before the conf was plumbed through (and Spark 3.x's own conf default).
+ #[default]
+ Exception,
+ /// Read values as proleptic Gregorian without rebasing.
+ Corrected,
+ /// Rebase from the hybrid Julian + Gregorian calendar.
+ Legacy,
+}
+
+impl RebaseReadMode {
+ /// Parses a `LegacyBehaviorPolicy` conf value. `SQLConf` validates and upper-cases the
+ /// session conf, but a per-relation `datetimeRebaseMode` option arrives verbatim, so the
+ /// match is case-insensitive. Anything unrecognized -- including the empty string a proto
+ /// producer that predates the field sends -- falls back to [`RebaseReadMode::Exception`],
+ /// which refuses ancient values rather than silently corrupting them.
+ pub(crate) fn from_conf_value(value: &str) -> Self {
+ match value.to_ascii_uppercase().as_str() {
+ "CORRECTED" => RebaseReadMode::Corrected,
+ "LEGACY" => RebaseReadMode::Legacy,
+ _ => RebaseReadMode::Exception,
+ }
+ }
+}
+
+/// The session's effective datetime rebase read modes, one per spec class (INT64
+/// dates/timestamps vs INT96 timestamps), forwarded from the JVM at planning time.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) struct SessionRebaseModes {
+ /// `spark.sql.parquet.datetimeRebaseModeInRead` (or the relation's `datetimeRebaseMode`).
+ pub datetime: RebaseReadMode,
+ /// `spark.sql.parquet.int96RebaseModeInRead` (or the relation's `int96RebaseMode`).
+ pub int96: RebaseReadMode,
+}
+
+/// Calendar policy of one file's date or timestamp columns, resolved from writer metadata the
+/// same way Spark's `DataSourceUtils.getRebaseSpec` resolves it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum RebasePolicy {
+ /// Written in the proleptic Gregorian calendar; values pass through untouched.
+ Corrected,
+ /// Written in the hybrid Julian + Gregorian calendar; values must be rebased.
+ Legacy(WriterTimeZone),
+ /// Policy could not be pinned down (contradictory flags, or a non-Spark writer under the
+ /// `EXCEPTION` read mode): modern values -- identical under either calendar -- pass,
+ /// ancient values raise. Mirrors Spark's `EXCEPTION` behavior (`SparkUpgradeException`).
+ CheckAncient,
+}
+
+/// Which of a file's leaf columns are physically INT96, from the stamp the parquet reader
+/// factory adds under [`INT96_LEAVES_METADATA_KEY`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) enum Int96Attribution {
+ /// No stamp, or a stamp whose leaf count does not match the schema it arrived with: the
+ /// INT64 and INT96 timestamp specs cannot be told apart per column and are merged.
+ Unknown,
+ /// Sorted leaf ordinals (depth-first over the file schema's primitive columns) that are
+ /// INT96; every other timestamp leaf is INT64.
+ Known(Vec<usize>),
+}
+
+impl Int96Attribution {
+ /// Parses the stamp out of `schema`'s metadata and validates its leaf count against the
+ /// schema's own depth-first leaf count, so a stamp that does not describe this schema (a
+ /// crafted footer key, or a cached-metadata mismatch) degrades to [`Self::Unknown`].
+ fn from_schema(schema: &Schema) -> Self {
+ let Some(stamp) = schema.metadata().get(INT96_LEAVES_METADATA_KEY) else {
+ return Int96Attribution::Unknown;
+ };
+ let Some((count, ordinals)) = stamp.split_once(':') else {
+ return Int96Attribution::Unknown;
+ };
+ let schema_leaves: usize = schema
+ .fields()
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ if count.parse::<usize>().ok() != Some(schema_leaves) {
+ return Int96Attribution::Unknown;
+ }
+ let parsed: Option<Vec<usize>> = if ordinals.is_empty() {
+ Some(Vec::new())
+ } else {
+ ordinals
+ .split(',')
+ .map(|o| o.parse::<usize>().ok().filter(|o| *o < schema_leaves))
+ .collect()
+ };
+ match parsed {
+ Some(mut leaves) => {
+ leaves.sort_unstable();
+ Int96Attribution::Known(leaves)
+ }
+ None => Int96Attribution::Unknown,
+ }
+ }
+
+ /// `Some(true)` / `Some(false)` when the leaf is known to be INT96 / INT64, `None` when
+ /// the attribution is unknown.
+ fn is_int96(&self, leaf: usize) -> Option<bool> {
+ match self {
+ Int96Attribution::Unknown => None,
+ Int96Attribution::Known(leaves) => Some(leaves.binary_search(&leaf).is_ok()),
+ }
+ }
+}
+
+/// The [`INT96_LEAVES_METADATA_KEY`] value describing `schema`: its leaf count and the
+/// ordinals of its INT96 primitive columns.
+pub(crate) fn int96_leaf_stamp(schema: &SchemaDescriptor) -> String {
+ let ordinals: Vec<String> = schema
+ .columns()
+ .iter()
+ .enumerate()
+ .filter(|(_, column)| column.physical_type() == ParquetPhysicalType::INT96)
+ .map(|(ordinal, _)| ordinal.to_string())
+ .collect();
+ format!("{}:{}", schema.num_columns(), ordinals.join(","))
+}
+
+/// Returns a copy of `metadata` whose key-value metadata carries the [`int96_leaf_stamp`] of
+/// its own schema, or `None` when it already does (the common case after the first open of a
+/// file, since the caller caches the stamped copy). Any pre-existing entry under the key --
+/// a file cannot legitimately carry one -- is replaced, never trusted. Only the file-level
+/// key-value list changes; row groups and page indexes are carried over as-is. The parquet
+/// API cannot carry a file decryptor, nor `FileMetaData`'s crate-private encryption fields
+/// (encryption algorithm, footer signing key metadata), across this rebuild, so callers must
+/// not stamp opens that supply decryption properties -- and the only consumer, the Delta
+/// scan, declines every encrypted-parquet configuration before planning, so a parquet
+/// modular encryption file never reaches this path with or without those properties.
+pub(crate) fn stamp_int96_leaves(metadata: &ParquetMetaData) -> Option<ParquetMetaData> {
+ let file_metadata = metadata.file_metadata();
+ let stamp = int96_leaf_stamp(file_metadata.schema_descr());
+ let existing = file_metadata
+ .key_value_metadata()
+ .and_then(|kvs| kvs.iter().find(|kv| kv.key == INT96_LEAVES_METADATA_KEY))
+ .and_then(|kv| kv.value.as_deref());
+ if existing == Some(stamp.as_str()) {
+ return None;
+ }
+ let mut key_values: Vec<KeyValue> = file_metadata
+ .key_value_metadata()
+ .map(|kvs| {
+ kvs.iter()
+ .filter(|kv| kv.key != INT96_LEAVES_METADATA_KEY)
+ .cloned()
+ .collect()
+ })
+ .unwrap_or_default();
+ key_values.push(KeyValue::new(INT96_LEAVES_METADATA_KEY.to_string(), stamp));
+ let stamped_file_metadata = FileMetaData::new(
+ file_metadata.version(),
+ file_metadata.num_rows(),
+ file_metadata.created_by().map(str::to_string),
+ Some(key_values),
+ file_metadata.schema_descr_ptr(),
+ file_metadata.column_orders().cloned(),
+ );
+ Some(
+ ParquetMetaData::new(stamped_file_metadata, metadata.row_groups().to_vec())
+ .into_builder()
+ .set_column_index(metadata.column_index().cloned())
+ .set_offset_index(metadata.offset_index().cloned())
+ .build(),
+ )
+}
+
+/// Per-file rebase policies for the three affected column classes, plus the INT96
+/// attribution that selects between the two timestamp specs per leaf.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct FileRebasePolicies {
+ /// `DATE` columns, governed by `org.apache.spark.legacyDateTime` alone.
+ pub date: RebasePolicy,
+ /// INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns, adjusted to UTC or not: the
+ /// datetime spec (same resolution as `date`), as Spark's `ParquetVectorUpdaterFactory`
+ /// selects for INT64 read as `TIMESTAMP`.
+ pub int64_timestamp: RebasePolicy,
+ /// INT96 columns: the INT96 spec (`org.apache.spark.legacyINT96`, min version 3.1.0).
+ pub int96_timestamp: RebasePolicy,
+ /// Which timestamp leaves are INT96. See [`Int96Attribution`].
+ pub int96_leaves: Int96Attribution,
+ /// Sorted depth-first leaf ordinals -- over the physical file schema, the same ordinals
+ /// `int96_leaves` uses -- that the query does not read: nested children the schema
+ /// adapter's struct narrowing drops before any value leaves the scan. Spark never decodes
+ /// them either, so their policy is the identity whatever the file's calendar. Empty until
+ /// [`Self::restrict_to_requested`] runs (every leaf requested).
+ pub unrequested_leaves: Vec<usize>,
+ /// Sorted physical leaf ordinals of timezone-carrying timestamps (INT96, or INT64 with
+ /// `isAdjustedToUTC=true`) the query reads as `TIMESTAMP_NTZ`. Spark decodes those with
+ /// `BinaryToSQLTimestampUpdater` / `LongUpdater`, which never rebase, so their policy is
+ /// the identity whatever the file's calendar. Filled by [`Self::restrict_to_requested`].
+ pub ntz_requested_leaves: Vec<usize>,
+ /// Sorted physical leaf ordinals of timezone-free INT64 timestamps
+ /// (`isAdjustedToUTC=false`) the query reads as `TIMESTAMP`. Spark's INT64 branch checks
+ /// only the unit (`isTimestampTypeMatched`) and hands a `TimestampType` request to
+ /// `LongWithRebaseUpdater` under the datetime spec, adjusted or not, so these leaves take
+ /// the same policy as adjusted INT64 leaves. Filled by [`Self::restrict_to_requested`].
+ pub ltz_requested_leaves: Vec<usize>,
+}
+
+/// The leaf ordinals [`push_unrequested_leaves`] records while pairing a physical type with
+/// the type the query reads it as. Each list is emitted in depth-first order, so it is already
+/// sorted for the binary searches in [`FileRebasePolicies`].
+#[derive(Debug, Default)]
+struct LeafPairing {
+ unrequested: Vec<usize>,
+ ntz_requested: Vec<usize>,
+ ltz_requested: Vec<usize>,
+}
+
+impl FileRebasePolicies {
+ /// True when some policy is not the plain proleptic-Gregorian pass-through, i.e. when the
+ /// per-column wrap in [`wrap_datetime_rebase`] can install anything at all.
+ pub(crate) fn any_rebase_needed(&self) -> bool {
+ self.date != RebasePolicy::Corrected
+ || self.int64_timestamp != RebasePolicy::Corrected
+ || self.int96_timestamp != RebasePolicy::Corrected
+ }
+
+ fn is_requested(&self, leaf: usize) -> bool {
+ self.unrequested_leaves.binary_search(&leaf).is_err()
+ }
+
+ /// The policy of the `Date32` leaf at depth-first ordinal `leaf`: the file's date policy,
+ /// or the identity when the query does not read that leaf.
+ fn date_policy(&self, leaf: usize) -> RebasePolicy {
+ if self.is_requested(leaf) {
+ self.date
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+
+ /// The policy of the timezone-carrying timestamp leaf at depth-first ordinal `leaf`: the
+ /// identity when the query does not read it or reads it as `TIMESTAMP_NTZ` (Spark's NTZ
+ /// updaters never rebase; on Spark 3.x the pairing is refused before any rebase decision,
+ /// which Comet's `allow_timestamp_ltz_to_ntz` gate reproduces); otherwise its physical
+ /// type's spec when the attribution is known, or else the two specs merged -- agreement
+ /// decides, disagreement degrades to [`RebasePolicy::CheckAncient`], which still passes
+ /// every modern value and refuses only ancient ones.
+ fn timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) || self.ntz_requested_leaves.binary_search(&leaf).is_ok() {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ Some(false) => self.int64_timestamp,
+ None if self.int64_timestamp == self.int96_timestamp => self.int64_timestamp,
+ None => RebasePolicy::CheckAncient,
+ }
+ }
+
+ /// The policy of the timezone-free timestamp leaf at depth-first ordinal `leaf` (INT64 with
+ /// `isAdjustedToUTC=false`): the identity unless the query reads it as `TIMESTAMP`, which
+ /// Spark decodes with `LongWithRebaseUpdater` under the datetime spec exactly like an
+ /// adjusted INT64 leaf. The stamp is still consulted so a leaf it names INT96 follows the
+ /// INT96 spec; without a stamp the physical type itself proves INT64, so the two specs are
+ /// not merged.
+ fn tz_free_timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) || self.ltz_requested_leaves.binary_search(&leaf).is_err() {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ _ => self.int64_timestamp,
+ }
+ }
+
+ /// These policies with every physical leaf the query does not read marked the identity,
+ /// and every timestamp leaf whose requested type differs from its physical one in timezone
+ /// presence recorded, so [`leaf_policies`] can pick the policy Spark's
+ /// `ParquetVectorUpdaterFactory.getUpdater` picks for the REQUESTED type.
+ /// `requested` pairs each top-level field of `physical_schema` (by position) with the type
+ /// of the logical field the schema adapter narrows it to -- `None` for a column without a
+ /// logical counterpart, whose leaves are left as they are (no expression reads it anyway).
+ /// Nested children pair the way the adapter's struct convert selects them (see
+ /// [`push_unrequested_leaves`]); the INT96 attribution is untouched, since the ordinals
+ /// stay physical. `requested` is parallel to the schema's fields; should a caller pass a
+ /// shorter slice, the trailing columns simply keep every leaf (the safe direction).
+ pub(crate) fn restrict_to_requested(
+ mut self,
+ physical_schema: &Schema,
+ requested: &[Option<&DataType>],
+ case_sensitive: bool,
+ use_field_id: bool,
+ ) -> Self {
+ debug_assert_eq!(requested.len(), physical_schema.fields().len());
+ let matching = FieldMatching {
+ case_sensitive,
+ use_field_id,
+ };
+ let mut next_leaf = 0;
+ let mut pairing = LeafPairing::default();
+ for (field, requested) in physical_schema.fields().iter().zip(requested) {
+ match requested {
+ Some(logical) => push_unrequested_leaves(
+ field.data_type(),
+ logical,
+ &mut next_leaf,
+ matching,
+ &mut pairing,
+ ),
+ None => next_leaf += leaf_count(field.data_type()),
+ }
+ }
+ // Emitted in depth-first order, so already sorted for the binary searches.
+ self.unrequested_leaves = pairing.unrequested;
+ self.ntz_requested_leaves = pairing.ntz_requested;
+ self.ltz_requested_leaves = pairing.ltz_requested;
+ self
+ }
+}
+
+/// The field-matching rules of the schema adapter's nested narrowing
+/// (`parquet_convert_struct_to_struct`): names fold per `case_sensitive`, and Parquet field ids
+/// select fields when `use_field_id` is set.
+#[derive(Debug, Clone, Copy)]
+struct FieldMatching {
+ case_sensitive: bool,
+ use_field_id: bool,
+}
+
+/// Appends to `out.unrequested` the depth-first leaf ordinals of `physical` (counting from
+/// `next_leaf`, which advances past every leaf of `physical`) that reading it as `requested`
+/// drops, and records in `out.ntz_requested` / `out.ltz_requested` the timestamp leaves whose
+/// requested type has the opposite timezone presence (a timezone-carrying leaf read as
+/// `TIMESTAMP_NTZ`, a timezone-free leaf read as `TIMESTAMP`); the unit is irrelevant to
+/// either, as it is to Spark's `isTimestampTypeMatched`.
+///
+/// Recurses through exactly the pairings `parquet_convert_array` narrows, and no others: a
+/// struct child is dropped only when NO requested child selects it by either rule the struct
+/// convert uses -- folded name, or Parquet field id when ids are in play -- and an ambiguous
+/// child (several requested children select it) is kept; `List` pairs with `List` by element
+/// type, and `Map` with a `Map` of the same key ordering by its entries, positionally. Any
+/// other pairing -- a `LargeList` / `FixedSizeList` / dictionary, a map whose ordering
+/// differs, or a shape mismatch -- is handed to arrow's cast or passed through whole by the
+/// convert, so it keeps every leaf under its physical type's policy. Keeping a superset of
+/// what the narrowing reads is always safe (a spurious check at worst); dropping a leaf the
+/// narrowing reads would skip its rebase, so every doubt resolves to "requested". Timestamp
+/// leaves inside those pass-through shapes are never recorded either, so they keep the
+/// physical rule (a spurious check for an NTZ request, no rebase for a `TIMESTAMP` request of
+/// a timezone-free leaf); Spark's requested schemas never take those arrow shapes.
+fn push_unrequested_leaves(
+ physical: &DataType,
+ requested: &DataType,
+ next_leaf: &mut usize,
+ matching: FieldMatching,
+ out: &mut LeafPairing,
+) {
+ match (physical, requested) {
+ (DataType::Timestamp(_, Some(_)), DataType::Timestamp(_, None)) => {
+ out.ntz_requested.push(*next_leaf);
+ *next_leaf += 1;
+ }
+ (DataType::Timestamp(_, None), DataType::Timestamp(_, Some(_))) => {
+ out.ltz_requested.push(*next_leaf);
+ *next_leaf += 1;
+ }
+ (DataType::Struct(physical_fields), DataType::Struct(requested_fields)) => {
+ let names: Vec<&str> = physical_fields
+ .iter()
+ .chain(requested_fields.iter())
+ .map(|f| f.name().as_str())
+ .collect();
+ // A fold failure means the names could not be compared at all; keeping every leaf
+ // requested is the safe superset, the same as the pass-through pairings below.
+ let Ok(folded) = fold_names(&names, matching.case_sensitive) else {
+ *next_leaf += leaf_count(physical);
+ return;
+ };
+ let (physical_folded, requested_folded) = folded.split_at(physical_fields.len());
+ for (i, child) in physical_fields.iter().enumerate() {
+ let child_id = if matching.use_field_id {
+ field_id(child)
+ } else {
+ None
+ };
+ let mut selectors = requested_fields.iter().enumerate().filter(|(j, r)| {
+ requested_folded[*j] == physical_folded[i]
+ || (child_id.is_some() && field_id(r) == child_id)
+ });
+ match (selectors.next(), selectors.next()) {
+ (None, _) => {
+ let n = leaf_count(child.data_type());
+ out.unrequested.extend(*next_leaf..*next_leaf + n);
+ *next_leaf += n;
+ }
+ (Some((_, requested_child)), None) => push_unrequested_leaves(
+ child.data_type(),
+ requested_child.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (Some(_), Some(_)) => *next_leaf += leaf_count(child.data_type()),
+ }
+ }
+ }
+ (DataType::List(physical_item), DataType::List(requested_item)) => push_unrequested_leaves(
+ physical_item.data_type(),
+ requested_item.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (
+ DataType::Map(physical_entries, physical_sorted),
+ DataType::Map(requested_entries, requested_sorted),
+ ) if physical_sorted == requested_sorted => {
+ match (physical_entries.data_type(), requested_entries.data_type()) {
+ (DataType::Struct(physical_kv), DataType::Struct(requested_kv))
+ if physical_kv.len() == requested_kv.len() =>
+ {
+ for (p, r) in physical_kv.iter().zip(requested_kv.iter()) {
+ push_unrequested_leaves(
+ p.data_type(),
+ r.data_type(),
+ next_leaf,
+ matching,
+ out,
+ );
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
Review Comment:
[P2] Suppress temporal rebasing when Spark requests a supported numeric type. The catch-all pairing leaves timestamp/date leaves governed by their physical temporal policy even when requested as `BIGINT` or `INT`. A concrete supported case is metadata-free `TIMESTAMP(NANOS,true)` Parquet inferred as `BIGINT` with `spark.sql.legacy.parquet.nanosAsLong=true`, then converted to Delta. Under `datetimeRebaseModeInRead=EXCEPTION`, Spark returns the stored integer, but the new Delta rebase wrapper rejects an 1800-era value and fails the query. Spark’s numeric updaters bypass calendar conversion. Please account for numeric requested types in the leaf policy, preserving genuine temporal rebasing, and add a converted-Parquet regression.
Evidence: Wrote metadata-free `message m { required int64 v (TIMESTAMP(NANOS,true)); }` containing `-5364662400000000000`. Spark 3.5.9/Delta 3.3.2 inferred `BIGINT` and returned that integer before and after `CONVERT TO DELTA ... NO STATISTICS` under EXCEPTION mode. The exact-head native scan builder with requested `Int64` returned it with rebasing disabled or CORRECTED, but metadata rebasing plus EXCEPTION failed with `Native scan cannot rebase ancient values in column 'v'`. MICROS-to-BIGINT and DATE-to-INT controls reproduced the same discrepancy. Logs: `/tmp/5365-a2df-current-numeric-reproduction.log` and `/tmp/5365-a2df-current-spark35-delta-reproduction.log`. Spark 3.5.9/4.0.4/4.1.3 `ParquetVectorUpdaterFactory` selects numeric updaters before temporal handling. Full Comet/Delta/JNI execution was not run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3874367477
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,548 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. Strip the parquet.field.id metadata
+ // createPhysicalSchema also stamps: files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations.
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
Review Comment:
You were right about which API this is. Spark keys footers by toLowerCase(Locale.ROOT), not equalsIgnoreCase, so that's what the matcher does now, and your three shapes flip accordingly. To stop chasing these one at a time I diffed the entire codepoint space against JDK 17: 95 codepoints differ purely from Unicode version skew between the JDK and the Rust tables. Those are pinned with a checked-in fixture and full-sweep tests on both sides, plus contextual sigma cases, which turned out to matter since the newer Rust tables treat some of those codepoints as cased and that flips the final sigma decision. If your harness still finds a disagreement I'd genuinely like to see it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3871824730
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme (finding 8).
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
Review Comment:
Went after the underlying cause this time instead of the specific keys. Two changes: the option extraction now forwards substituted values (it was forwarding raw literals, which is where every ${...} case came from), and the per-case credential gates are replaced by one comparison of Hadoop's effective value against native's resolution for every key native reads, declining on any difference. Your ${review.provider} shape is covered, and your earlier ${review.access} probe now claims and reads correctly against MinIO since both sides see the same resolved values. If you have more shapes handy I'd appreciate you throwing them at this version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5441843040
> **Note on this review:** this was generated by an LLM (Claude Code) at my request while I worked through a review backlog. I have not verified the individual findings myself. Please treat everything below as suggestions to evaluate rather than as authoritative review feedback, and push back on anything that is wrong or already handled.
> A note on scope: this is an 11,000-line change and I have reviewed it at the level of structure, risk, and process rather than line by line. @sunchao has already gone through it in detail, so I have tried to raise things that a structural read surfaces rather than repeat that.
The core insight is a good one, and it is the thing that makes this PR much more attractive than the two it supersedes: delta-spark has already done log replay, snapshot resolution, and partition pruning by the time `CometScanRule` sees the `FileSourceScanExec`, so there is no Delta planning to do natively and the scan can reuse the existing `ParquetSource` path wholesale. Getting row-group pruning, page-index pruning, and filter pushdown for free, with deletion vectors composing into `ParquetAccessPlan` so DV skips and page skips intersect, is exactly the right design. Crediting both prior efforts is the right thing to do too.
Five things.
**`delta` is in the default feature set**
```toml
-default = ["hdfs-opendal"]
+default = ["hdfs-opendal", "delta"]
```
The comment argues this is inert without the contrib jar and the config, which I believe. But it still means every default build of `libcomet` links `roaring` and `crc32fast` and carries 1,800 lines of DV decoding, including for the large majority of users who will never touch Delta. The existing `contrib-delta` feature right below it is explicitly documented as "Default builds carry zero Delta surface", so the two features now state opposite philosophies in adjacent comments.
What is the argument for defaulting `delta` on rather than making it opt-in like `contrib-delta`? If it is "otherwise nobody can use the contrib jar without a custom build", that is a real concern and worth stating explicitly, but it should be a deliberate decision recorded in the description rather than a one-line default change.
**`delta_dv.rs` is 1,818 lines of binary format parsing**
Deletion-vector bitmaps are an on-disk format with a CRC, produced by delta-spark and by other Delta writers. This is the highest-risk file in the PR: a parsing bug is either a wrong answer (rows that should be deleted come back) or a panic on malformed input.
Two questions. Is the decoder validated against DV files produced by delta-spark itself, or only against ones this code round-trips? And what happens on a truncated, CRC-mismatched, or otherwise malformed DV file: a clean error, or a panic or out-of-bounds read? Delta tables are read from object storage where partial reads happen, so the malformed path matters.
**Reflection surface**
`DeltaScanSupport.scala` is 1,079 lines and `IcebergReflection`-style reflection against delta-spark internals is inherently version-fragile. Which delta-spark versions is this tested against, and what happens on a version whose internals moved: a clean decline through the gate, or a `NoSuchMethodError` at plan time? The decline-gate philosophy inherited from #4366 suggests the former, but with that much reflection it is worth stating explicitly and having a test that simulates a missing method.
**CI cost**
A new `delta_contrib_test.yml` workflow plus ~4,300 lines of test suites. What does that add to CI wall clock and runner minutes, and is it gated by `compute-changes.py` so it only runs when Delta or native code changes? The `compute-changes.py` diff suggests yes, but the numbers should be in the description, especially given #5459 is separately trying to reduce Iceberg CI time.
**Benchmark numbers**
The description starts a "Local benchmark (20M rows, ..." sentence that appears to be cut off in what I read. Whatever the numbers are, they are the justification for the whole PR and should be prominent: Comet versus Spark on a Delta table with and without deletion vectors, and ideally with and without page-index pruning so the DV-composes-with-page-skips claim is demonstrated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5488167570
@sunchao any updates on the changes here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5503242012
> cc @andygrove @comphead @parthchandra @viirya @peterxcli as well, since this is a fairly large PR and would benefit from a few extra sets of eyes.
If anyone knows any fixes, they can also submit to this branch as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3920748407
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -76,6 +77,583 @@ fn schema_has_field_ids(schema: &SchemaRef) -> bool {
schema.fields().iter().any(|f| parse_field_id(f).is_some())
}
+// ---------------------------------------------------------------------------------------------
+// JVM-shipped case tables: reproduce the PLANNING JVM's `String.toLowerCase(Locale.ROOT)`,
+// which Spark's Parquet footer field matching is built on.
+//
+// The data arrives on `NativeScanCommon` (populated by `JvmCaseTables.scala` when
+// case_sensitive = false), generated from the very JVM that plans the query, so the native
+// matcher is correct BY CONSTRUCTION for whatever JDK runs Spark. The one contextual mapping
+// (`Locale.ROOT` has exactly one: Greek capital sigma U+03A3) cannot be a per-codepoint table
+// entry, so its condition is ported as an algorithm over a shipped per-codepoint
+// classification -- see `JvmCaseTables::lowercase`.
+// ---------------------------------------------------------------------------------------------
+
+// Sigma-scan classes: the wire contract shared with `JvmCaseTables.scala`. The classes are
+// the UAX#29-style word-break classes (ALetter, Numeric, MidLetter, MidNum, MidNumLet,
+// Extend, Format) as the PLANNING JVM's legacy break iterator actually realizes them --
+// probed per codepoint from `BreakIterator.isBoundary` on the JVM side -- plus classes for
+// its pre-UAX#29 extensions (the danda and the supplementary-plane behaviors of its UTF-16
+// DFA), with cased variants split out via the JDK's `isCased`. Any codepoint outside every
+// shipped range -- and any class value this build does not know -- is a word boundary: the
+// sigma context scan stops there, which is also the safe reading for class values added by a
+// NEWER JVM-side generator.
+const CLASS_ALETTER_CASED: u8 = 1;
+const CLASS_ALETTER: u8 = 2;
+const CLASS_NUMERIC: u8 = 3;
+const CLASS_MID_LETTER: u8 = 4;
+const CLASS_MID_NUM: u8 = 5;
+const CLASS_MID_NUM_LET: u8 = 6;
+/// Cased supplementary char: attaches to the preceding word and closes it (and forms a word
+/// of its own at raw text start).
+const CLASS_SUPP_CASED: u8 = 7;
+/// U+0964/U+0965: word-terminal, chains only into digits.
+const CLASS_DANDA: u8 = 8;
+/// U+0345, the one cased combining mark: cased only when its run is attached to a word.
+const CLASS_EXTEND_CASED: u8 = 9;
+/// Cased digit-base (Nl Roman numerals): joins like CLASS_ALETTER_CASED when reached
+/// directly, but bridges only mid-num punctuation, never mid-letter.
+const CLASS_NUMERIC_CASED: u8 = 10;
+/// Non-cased Mn/Me marks: riders that attach only to genuine letter/digit bases.
+const CLASS_EXTEND: u8 = 11;
+/// Word-forming non-cased supplementary letter: a genuine letter-base that closes the word
+/// immediately after itself.
+const CLASS_SUPP_LETTER: u8 = 12;
+/// Cf format characters: fully transparent (WB4-style) -- deleted from the sequence before
+/// the scans run, so a pure-format rider chain bridges mid punctuation ("AΣ-<ZWJ>b" is one
+/// word exactly like "AΣ-b").
+const CLASS_FORMAT: u8 = 13;
+/// Supplementary chars that attach to the preceding word but never form one themselves
+/// (supplementary combining marks, tag characters): a cased mark riding on one belongs to
+/// the sigma's word only when the run hangs off a real base (`supp_mn_anchor`).
+const CLASS_SUPP_MN: u8 = 14;
+/// Word-forming supplementary digit: like CLASS_SUPP_LETTER except a riding cased mark
+/// carries only across mid-num (digit-context) punctuation, never mid-letter.
+const CLASS_SUPP_NUM: u8 = 15;
+/// Not on the wire: the absence of a class.
+const CLASS_BOUNDARY: u8 = 0;
+
+const CAPITAL_SIGMA: char = '\u{03A3}';
+const SMALL_SIGMA: char = '\u{03C3}';
+const SMALL_FINAL_SIGMA: char = '\u{03C2}';
+
+/// What a supplementary-mark run ultimately hangs off (see `supp_mn_anchor`).
+#[derive(PartialEq, Eq, Clone, Copy)]
+enum SuppMnAnchor {
+ None,
+ Letter,
+ Digit,
+}
+
+/// The planning JVM's case data, parsed once per scan from `NativeScanCommon` and attached to
+/// [`SparkParquetOptions`]. Two tables:
+///
+/// - `lower`: every codepoint the JVM lowercases non-identically, with its full (possibly
+/// multi-char, e.g. U+0130 -> "i" + U+0307) replacement; codepoints absent here lowercase
+/// to themselves;
+/// - `class_ranges`: sorted, disjoint `(start, end, class)` codepoint ranges holding the
+/// word-break classification the JVM probed from its own `BreakIterator`.
+///
+/// `lowercase` applies Java's algorithm over that data: per codepoint, U+03A3 takes its
+/// contextual final/non-final form via the ported `isFinalCased` condition -- word-boundary
+/// based (the JDK's legacy break-iterator word rules), NOT the Unicode-standard Final_Sigma
+/// case-ignorable skip, so e.g. "A1Σ" lowers to "a1ς" -- and every other codepoint takes its
+/// table replacement. `JvmCaseTables.mirrorLowercase` on the Scala side is the line-for-line
+/// mirror of this function over the same generated data; the JVM-side parity suite proves the
+/// pair equal to the running JDK's `String.toLowerCase(Locale.ROOT)` across the full codepoint
+/// space (calibrated to zero mismatches on JDK 17, 21, and 25).
+#[derive(Debug)]
+pub struct JvmCaseTables {
+ /// Non-identity lowercase mappings: codepoint -> full replacement string.
+ lower: HashMap<char, String>,
+ /// Sorted, disjoint (start, end, class) inclusive codepoint ranges for the sigma scan.
+ class_ranges: Vec<(u32, u32, u8)>,
+ /// Precomputed content hash so `SparkParquetOptions`'s derived `Hash` stays cheap.
+ fingerprint: u64,
+}
+
+impl PartialEq for JvmCaseTables {
+ fn eq(&self, other: &Self) -> bool {
+ self.fingerprint == other.fingerprint
+ && self.class_ranges == other.class_ranges
+ && self.lower == other.lower
+ }
+}
+
+impl Eq for JvmCaseTables {}
+
+impl Hash for JvmCaseTables {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ // Equal contents always produce the equal (deterministically computed) fingerprint,
+ // so hashing only the fingerprint is consistent with `PartialEq`.
+ state.write_u64(self.fingerprint);
+ }
+}
+
+fn is_letter_base(cls: u8) -> bool {
+ cls == CLASS_ALETTER_CASED
+ || cls == CLASS_ALETTER
+ || cls == CLASS_SUPP_CASED
+ || cls == CLASS_SUPP_LETTER
+}
+
+fn is_digit_base(cls: u8) -> bool {
+ cls == CLASS_NUMERIC || cls == CLASS_NUMERIC_CASED
+}
+
+impl JvmCaseTables {
+ /// Parse the proto representation: `lower_cp`/`lower_repl` are index-aligned, and
+ /// `class_ranges` holds (start, end, class) triples. Malformed input (length mismatch,
+ /// trailing partial triple, out-of-range codepoints) is dropped entry-by-entry rather
+ /// than rejected: every dropped entry degrades one codepoint to identity/boundary
+ /// behavior instead of failing the scan.
+ pub fn from_proto(lower_cp: &[u32], lower_repl: &[String], class_ranges: &[u32]) -> Self {
+ let mut lower = HashMap::with_capacity(lower_cp.len());
+ for (cp, repl) in lower_cp.iter().zip(lower_repl.iter()) {
+ if let Some(c) = char::from_u32(*cp) {
+ lower.insert(c, repl.clone());
+ }
+ }
+ let ranges: Vec<(u32, u32, u8)> = class_ranges
+ .as_chunks::<3>()
+ .0
+ .iter()
+ .filter(|t| t[0] <= t[1] && t[1] <= 0x10FFFF && u8::try_from(t[2]).is_ok())
+ .map(|t| (t[0], t[1], t[2] as u8))
+ .collect();
+
+ let mut hasher = DefaultHasher::new();
+ for (start, end, class) in &ranges {
+ hasher.write_u32(*start);
+ hasher.write_u32(*end);
+ hasher.write_u8(*class);
+ }
+ let mut lower_sorted: Vec<(&char, &String)> = lower.iter().collect();
+ lower_sorted.sort_by_key(|(c, _)| **c);
+ for (c, repl) in lower_sorted {
+ hasher.write_u32(*c as u32);
+ hasher.write(repl.as_bytes());
+ }
+
+ Self {
+ lower,
+ class_ranges: ranges,
+ fingerprint: hasher.finish(),
+ }
+ }
+
+ /// Sigma-scan class of `c`; `CLASS_BOUNDARY` when no shipped range covers it.
+ fn class_of(&self, c: char) -> u8 {
+ let cp = c as u32;
+ let mut lo = 0usize;
+ let mut hi = self.class_ranges.len();
+ while lo < hi {
+ let mid = (lo + hi) / 2;
+ let (start, end, class) = self.class_ranges[mid];
+ if cp < start {
+ hi = mid;
+ } else if cp > end {
+ lo = mid + 1;
+ } else {
+ return class;
+ }
+ }
+ CLASS_BOUNDARY
+ }
+
+ /// First position at or beyond `start` (stepping by `step`, i.e. -1 backward / +1
+ /// forward) whose class is not CLASS_EXTEND. Returns `None` if the scan runs off the
+ /// array without finding one.
+ fn skip_extends(&self, cps: &[char], start: isize, step: isize) -> Option<usize> {
+ let mut k = start;
+ while k >= 0 && (k as usize) < cps.len() {
+ if self.class_of(cps[k as usize]) != CLASS_EXTEND {
+ return Some(k as usize);
+ }
+ k += step;
+ }
+ None
+ }
+
+ /// As [`Self::skip_extends`], but also skips CLASS_EXTEND_CASED, reporting whether one
+ /// was walked: a cased mark (U+0345) crossed while looking for a base is itself cased
+ /// whenever the landing validates the run.
+ fn skip_extends_tracking_cased(
+ &self,
+ cps: &[char],
+ start: isize,
+ step: isize,
+ ) -> (Option<usize>, bool) {
+ let mut k = start;
+ let mut saw_cased = false;
+ while k >= 0 && (k as usize) < cps.len() {
+ let cls = self.class_of(cps[k as usize]);
+ if cls != CLASS_EXTEND && cls != CLASS_EXTEND_CASED {
+ return (Some(k as usize), saw_cased);
+ }
+ if cls == CLASS_EXTEND_CASED {
+ saw_cased = true;
+ }
+ k += step;
+ }
+ (None, saw_cased)
+ }
+
+ /// What the supplementary-mark run at `k` (CLASS_SUPP_MN) ultimately hangs off, walking
+ /// down through further marks and supplementary chars: a letter-flavored base, a
+ /// digit-flavored base, or nothing word-forming. A cased mark riding the run belongs to
+ /// the sigma's word only per this anchor.
+ fn supp_mn_anchor(&self, cps: &[char], k: usize) -> SuppMnAnchor {
+ let mut m = k as isize - 1;
+ while m >= 0 {
+ let cls = self.class_of(cps[m as usize]);
+ if cls != CLASS_SUPP_MN && cls != CLASS_EXTEND && cls != CLASS_EXTEND_CASED {
+ break;
+ }
+ m -= 1;
+ }
+ if m < 0 {
+ return SuppMnAnchor::None;
+ }
+ match self.class_of(cps[m as usize]) {
+ CLASS_ALETTER_CASED | CLASS_ALETTER | CLASS_SUPP_CASED | CLASS_SUPP_LETTER => {
+ SuppMnAnchor::Letter
+ }
+ CLASS_NUMERIC | CLASS_NUMERIC_CASED | CLASS_SUPP_NUM => SuppMnAnchor::Digit,
+ _ => SuppMnAnchor::None,
+ }
+ }
+
+ /// Backward half of the ported `isFinalCased`: is there a cased letter before position
+ /// `i` within the sigma's word? Runs over the FORMAT-FILTERED sequence; `leading_format`
+ /// says whether format chars were filtered off the raw text start.
+ fn scan_back_finds_cased(&self, cps: &[char], i: usize, leading_format: bool) -> bool {
+ let mut last_letter = true; // the sigma itself is a letter
+ let mut j = i as isize - 1;
+ while j >= 0 {
+ match self.class_of(cps[j as usize]) {
+ CLASS_ALETTER_CASED | CLASS_NUMERIC_CASED => return true,
+ CLASS_ALETTER => {
+ last_letter = true;
+ j -= 1;
+ }
+ CLASS_NUMERIC => {
+ last_letter = false;
+ j -= 1;
+ }
+ CLASS_EXTEND => {
+ // Non-cased marks attach only to a real base below them; anything else
+ // (mid punctuation, danda, boundary, text start) leaves the run
+ // unattached.
+ let Some(k) = self.skip_extends(cps, j, -1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ let is_continuer = b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_EXTEND_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_NUM;
+ if !is_continuer {
+ return false;
+ }
+ j = k as isize;
+ }
+ CLASS_SUPP_CASED => {
+ // Closes the preceding word, so the scan stops -- except at RAW text
+ // start (no filtered-out leading format chars), where the DFA keeps it
+ // joined to what follows.
+ return j == 0 && !leading_format;
+ }
+ CLASS_SUPP_LETTER | CLASS_SUPP_MN | CLASS_SUPP_NUM => {
+ // Attach/close and never themselves cased; nothing beyond is reachable.
+ return false;
+ }
+ CLASS_EXTEND_CASED => {
+ // Cased combining mark (U+0345): cased when its run hangs off a base --
+ // a BMP letter/digit, a word-forming supplementary char (which closes a
+ // word right below the mark, merging the mark into the sigma's
+ // segment), or an ANCHORED supplementary mark.
+ let (Some(k), _) = self.skip_extends_tracking_cased(cps, j - 1, -1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ if b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_NUM
+ {
+ return true;
+ }
+ if b == CLASS_SUPP_MN {
+ return self.supp_mn_anchor(cps, k) != SuppMnAnchor::None;
+ }
+ return false;
+ }
+ CLASS_DANDA => {
+ // Backward across a danda: the word part before it must end in letters
+ // (grammar: letters, optional danda, then number+word chains) -- or
+ // carry a riding cased mark on a word-forming base, or be a cased
+ // supplementary char at text start -- and the danda itself chains only
+ // into digits after it.
+ if last_letter {
+ return false;
+ }
+ let (Some(k), saw_cased_mark) =
+ self.skip_extends_tracking_cased(cps, j - 1, -1)
+ else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ if b == CLASS_ALETTER_CASED {
+ return true;
+ }
+ if b == CLASS_SUPP_CASED {
+ return saw_cased_mark || (k == 0 && !leading_format);
+ }
+ if b == CLASS_SUPP_LETTER {
+ return saw_cased_mark;
+ }
+ if b == CLASS_SUPP_MN {
+ return saw_cased_mark
+ && self.supp_mn_anchor(cps, k) == SuppMnAnchor::Letter;
+ }
+ if b != CLASS_ALETTER {
+ return false;
+ }
+ if saw_cased_mark {
+ return true;
+ }
+ last_letter = true;
+ j = k as isize;
+ }
+ cls @ (CLASS_MID_LETTER | CLASS_MID_NUM | CLASS_MID_NUM_LET) => {
+ // `<mid-letter><let>` / `<mid-num><digit>` require a genuine
+ // letter/digit base before the punctuation; scanning backward
+ // legitimately walks marks-then-base (marks trail their base). A cased
+ // mark walked over rides whatever the punctuation hangs off, including
+ // a context-matching anchored supplementary mark or supplementary
+ // digit.
+ let mw_ok = cls == CLASS_MID_LETTER || cls == CLASS_MID_NUM_LET;
+ let mn_ok = cls == CLASS_MID_NUM || cls == CLASS_MID_NUM_LET;
+ let (Some(real_pos), saw_cased_mark) =
+ self.skip_extends_tracking_cased(cps, j - 1, -1)
+ else {
+ return false;
+ };
+ let b = self.class_of(cps[real_pos]);
+ if last_letter
+ && mw_ok
+ && saw_cased_mark
+ && b == CLASS_SUPP_MN
+ && self.supp_mn_anchor(cps, real_pos) == SuppMnAnchor::Letter
+ {
+ return true;
+ }
+ if !last_letter
+ && mn_ok
+ && saw_cased_mark
+ && (b == CLASS_SUPP_NUM
+ || (b == CLASS_SUPP_MN
+ && self.supp_mn_anchor(cps, real_pos) == SuppMnAnchor::Digit))
+ {
+ return true;
+ }
+ let bridge_valid = (last_letter && mw_ok && is_letter_base(b))
+ || (!last_letter && mn_ok && is_digit_base(b));
+ if !bridge_valid {
+ return false;
+ }
+ if saw_cased_mark {
+ return true;
+ }
+ j = real_pos as isize;
+ }
+ _ => return false,
+ }
+ }
+ false
+ }
+
+ /// Forward half of the ported `isFinalCased`: is there a cased letter after position `i`
+ /// within the sigma's word? Runs over the FORMAT-FILTERED sequence.
+ fn scan_fwd_finds_cased(&self, cps: &[char], i: usize) -> bool {
+ let mut last_letter = true;
+ let mut j = i + 1;
+ while j < cps.len() {
+ match self.class_of(cps[j]) {
+ CLASS_ALETTER_CASED | CLASS_NUMERIC_CASED => return true,
+ CLASS_ALETTER => {
+ last_letter = true;
+ j += 1;
+ }
+ CLASS_NUMERIC => {
+ last_letter = false;
+ j += 1;
+ }
+ CLASS_EXTEND => {
+ // A mark run trailing the anchor is properly attached in text order, so
+ // the run stays open past it, including onto mid punctuation on its far
+ // side.
+ let Some(k) = self.skip_extends(cps, j as isize, 1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ let is_continuer = b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_EXTEND_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_MN
+ || b == CLASS_SUPP_NUM
+ || b == CLASS_DANDA
+ || b == CLASS_MID_LETTER
+ || b == CLASS_MID_NUM
+ || b == CLASS_MID_NUM_LET;
+ if !is_continuer {
+ return false;
+ }
+ j = k;
+ }
+ CLASS_SUPP_CASED | CLASS_EXTEND_CASED => {
+ // Attaches to the current word, so the scan sees it (cased).
+ return true;
+ }
+ CLASS_SUPP_LETTER | CLASS_SUPP_MN | CLASS_SUPP_NUM => {
+ // Attach to the current word and close it; never themselves cased, and
+ // nothing beyond is reachable.
+ return false;
+ }
+ // The danda attaches only to a word part that ends in letters (reached
+ // after digits the word is already closed) and continues only into a digit
+ // -- unless that digit is itself cased (a Roman numeral), which resolves
+ // the scan immediately.
+ CLASS_DANDA if !last_letter => return false,
+ CLASS_DANDA
+ if j + 1 < cps.len() && self.class_of(cps[j + 1]) == CLASS_NUMERIC_CASED =>
+ {
+ return true;
+ }
+ CLASS_DANDA if j + 1 < cps.len() && self.class_of(cps[j + 1]) == CLASS_NUMERIC => {
+ last_letter = false;
+ j += 2;
+ }
+ cls @ (CLASS_MID_LETTER | CLASS_MID_NUM | CLASS_MID_NUM_LET) => {
+ // `<mid-letter><let>` / `<mid-num><digit>` require a genuine
+ // letter/digit base IMMEDIATELY after the punctuation -- unlike the
+ // backward scan, marks here are never skipped past: a mark directly
+ // after the punctuation is attached to the punctuation, not a base, so
+ // it blocks the bridge. (Format chars are already filtered out, which
+ // is what lets "AΣ-<ZWJ>b" bridge exactly like "AΣ-b".)
+ let mw_ok = cls == CLASS_MID_LETTER || cls == CLASS_MID_NUM_LET;
+ let mn_ok = cls == CLASS_MID_NUM || cls == CLASS_MID_NUM_LET;
+ if j + 1 >= cps.len() {
+ return false;
+ }
+ let b = self.class_of(cps[j + 1]);
+ if (last_letter && mw_ok && is_letter_base(b))
+ || (!last_letter && mn_ok && is_digit_base(b))
+ {
+ j += 1;
+ } else {
+ return false;
+ }
+ }
+ _ => return false,
+ }
+ }
+ false
+ }
+
+ /// Lowercase `s` the way the planning JVM's `String.toLowerCase(Locale.ROOT)` does.
+ pub fn lowercase(&self, s: &str) -> String {
+ let raw: Vec<char> = s.chars().collect();
+ // Built lazily on the first sigma: the format-filtered sequence the scans run over
+ // (WB4-style: the legacy break iterator's `<ignore>` class loops on every DFA
+ // state), the raw->filtered index map, and whether format chars led the raw text.
+ let mut filtered: Option<(Vec<char>, Vec<usize>, bool)> = None;
+ let mut out = String::with_capacity(s.len());
+ for (i, &c) in raw.iter().enumerate() {
+ if c == CAPITAL_SIGMA {
+ // The condition consults the ORIGINAL neighbors, exactly as the JDK scans
+ // `src`, not the partially-lowered output. The final/non-final target chars
+ // are Unicode-stable (pinned in `ConditionalSpecialCasing`'s entry table).
+ let (f, idx, leading_format) = filtered.get_or_insert_with(|| {
+ let mut f = Vec::with_capacity(raw.len());
+ let mut idx = vec![0usize; raw.len()];
+ for (k, &rc) in raw.iter().enumerate() {
+ idx[k] = f.len();
+ if self.class_of(rc) != CLASS_FORMAT {
+ f.push(rc);
+ }
+ }
+ let leading_format = self.class_of(raw[0]) == CLASS_FORMAT;
+ (f, idx, leading_format)
+ });
+ let fi = idx[i];
+ let is_final = self.scan_back_finds_cased(f, fi, *leading_format)
+ && !self.scan_fwd_finds_cased(f, fi);
+ out.push(if is_final {
+ SMALL_FINAL_SIGMA
+ } else {
+ SMALL_SIGMA
+ });
+ } else if let Some(repl) = self.lower.get(&c) {
+ out.push_str(repl);
+ } else {
+ out.push(c);
+ }
+ }
+ out
+ }
+}
+
+/// Lowercase `s` for case-insensitive field matching. With tables (populated whenever
+/// `case_sensitive = false`, shared by the core Parquet scan and the Delta contrib scan) this
+/// reproduces the planning JVM's `String.toLowerCase(Locale.ROOT)` exactly.
+///
+/// Without tables, fall back to Rust's `str::to_lowercase`, which agrees with Java on all
+/// simple mappings and differs only where the two Unicode snapshots or the sigma context
+/// diverge. This is a real, live path, not just a defensive default: the Iceberg native scan
+/// (`SparkPhysicalExprAdapterFactory::new(_, None)`) defaults `case_sensitive = false` and
+/// always reaches this fallback for its schema name remap, alongside
+/// `parquet_convert_struct_to_struct`'s general struct-cast matching and Rust-only unit tests
+/// that construct `SparkParquetOptions` directly.
+pub(crate) fn java_lowercase(s: &str, tables: Option<&JvmCaseTables>) -> String {
Review Comment:
Resolved by dropping our folding entirely: the branch now rides main's name_fold from #5602, which has the ASCII fast path you suggested plus schema-wide pre-folding, so none of this code exists here anymore. Your measurements matched what we saw when we profiled it independently, thanks for the rigor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3859142786
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
Review Comment:
Good catch, that was a real correctness bug. The partition schema now goes through the same createPhysicalSchema path as the data schemas, in both the plain and DV builders, so partition constants live in the physical namespace and can't collide with retained physical data names. Your exact history reproduced the corruption on Spark 3.5 as well ((100,100) twice) and now returns (1,100),(2,100) natively. Regressions cover that history, a DV variant, and a rename without collision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4081054647
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ *
+ * `private[delta]` (not `private`): [[DeltaScanSupport.declineReason]]'s non-ASCII
+ * case-insensitive name gate reuses this exact conversion to compute the names native sees
+ * under column mapping, rather than re-deriving physical names with separate logic.
+ */
+ private[delta] def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. Strip the parquet.field.id metadata
+ // createPhysicalSchema also stamps: files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations.
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully. `memo` is the same claim-memo instance
+ * [[DeltaScanSupport.declineReason]] populated on this claim; its `hadoopConf` and
+ * `dvDescriptors` are reused here rather than recomputed.
+ */
+ def convert(
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaScanSupport.DeltaClaimMemo): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = memo.hadoopConf
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = toPhysical(scanExec, relation.partitionSchema),
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Already forced by declineReason on this claim; reused rather than deserialized again.
+ val dvDescriptors = memo.dvDescriptors
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme.
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
+
+ val common = commonBuilder.build()
+ // Effective session rebase read modes, resolved through ParquetOptions exactly as
+ // ParquetFileFormat.buildReaderWithPartitionValues resolves them (per-relation
+ // `datetimeRebaseMode` / `int96RebaseMode` options win over the session conf, whose
+ // per-Spark-version default -- EXCEPTION on 3.x, CORRECTED on 4.0 -- SQLConf supplies).
+ // Native consults them only for files whose footer metadata does not decide the rebase
+ // policy on its own, mirroring DataSourceUtils.getRebaseSpec's modeByConfig fallback.
+ val parquetReadOptions =
+ new org.apache.spark.sql.execution.datasources.parquet.ParquetOptions(
+ relation.options,
+ scanExec.conf)
+ val deltaCommon = OperatorOuterClass.DeltaSparkScanCommon
+ .newBuilder()
+ .setTableRoot(tableRoot)
+ .setColumnMappingMode(columnMappingMode(scanExec))
+ .setSourceKey(DeltaPlanDataInjector.sourceKey(tableRoot, common))
+ .setDatetimeRebaseModeInRead(parquetReadOptions.datetimeRebaseModeInRead)
+ .setInt96RebaseModeInRead(parquetReadOptions.int96RebaseModeInRead)
+ .build()
+ val deltaScan = OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaCommon)
+ Operator
+ .newBuilder()
+ .setPlanId(scanExec.id)
+ .setContribScan(DeltaSparkScanEnvelope.pack(deltaScan.build()))
+ .build()
+ }
+ }
+
+ /**
+ * One representative store URI per distinct object-store authority this scan's partitions may
+ * need options for: the data-file authority (`firstFileUri`), the table root unconditionally
+ * (UUID-relative DV sidecars resolve against it), and every distinct on-disk DV authority from
+ * `descriptors` (inline DVs carry no external URI and are filtered out). Deduping by authority
+ * rather than full URI keeps this O(distinct authorities) instead of O(files), keeping the
+ * FIRST URI seen per authority so `firstFileUri`/the table root win over a same-authority DV
+ * path.
+ */
+ private[delta] def storeUris(
+ descriptors: Seq[DeletionVectorDescriptor],
+ tableRootPath: Path,
+ firstFileUri: Option[java.net.URI]): Seq[java.net.URI] = {
+ val dvAuthorityUris = descriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(tableRootPath).toUri)
+ val candidates = firstFileUri.toSeq ++ Seq(tableRootPath.toUri) ++ dvAuthorityUris
+ val byAuthority = scala.collection.mutable.LinkedHashMap.empty[String, java.net.URI]
+ candidates.foreach(uri =>
+ byAuthority.getOrElseUpdate(DeltaScanSupport.uriAuthority(uri), uri))
+ byAuthority.values.toSeq
+ }
+
+ /**
+ * Unions `NativeConfig.extractObjectStoreOptions` over every `uris` authority. Safe to union
+ * rather than pick one: extracted keys are scheme-disjoint prefixes (`fs.s3a.*` vs
+ * `fs.azure.*`, ...), so options for different schemes never collide, and re-extracting the
+ * same scheme from two URIs is idempotent.
+ */
+ private[delta] def mergedObjectStoreOptions(
+ hadoopConf: org.apache.hadoop.conf.Configuration,
+ uris: Seq[java.net.URI]): Map[String, String] =
+ uris.foldLeft(Map.empty[String, String]) { (merged, uri) =>
+ merged ++ NativeConfig.extractObjectStoreOptions(hadoopConf, uri)
+ }
+
+ /**
+ * Harvest subquery-bearing predicates for this scan from its covering FilterExec. Spark 3.x
+ * strips them from a scan's `dataFilters` at planning (`FileSourceStrategy` routes them to the
+ * post-scan filter only), while Spark 4.x keeps them in `dataFilters`; collecting them here at
+ * claim time gives the execution-time resolve-and-push path the same inputs on every version,
+ * and the dedup below keeps Spark 4.x from carrying duplicates. Reference containment alone
+ * does not prove pushing a predicate down is semantics-preserving, so `spineToScan` also
+ * requires every intervening operator to commute with the push (an intervening
+ * LIMIT/Sort/Aggregate/join etc. stops the walk and leaves the filter where Spark placed it:
+ * missed pruning only).
+ */
+ def subqueryFiltersFromParent(
+ plan: org.apache.spark.sql.execution.SparkPlan,
+ scanExec: FileSourceScanExec): Seq[org.apache.spark.sql.catalyst.expressions.Expression] = {
+ import org.apache.spark.sql.catalyst.expressions.{PlanExpression, SubqueryExpression}
+ import org.apache.spark.sql.execution.{FilterExec, ProjectExec, SparkPlan}
+
+ // Whether every node from `node` down to `scanExec` is one pushdown can safely cross: a
+ // deterministic ProjectExec is 1:1 on rows and a deterministic FilterExec only removes rows,
+ // so moving a predicate over the scan's output through either preserves semantics -- mirroring
+ // Spark's own PushPredicateThroughNonJoin/CollapseProject rules. A nondeterministic node (or
+ // anything else: LIMIT/TopN, Sort, Aggregate, Window, joins, ...) can change which rows survive
+ // to matter, so it stops the walk and the filter is left uncollected (missed pruning only).
+ def spineToScan(node: SparkPlan): Boolean = node match {
+ case n if n eq scanExec => true
+ case p: ProjectExec if p.projectList.forall(_.deterministic) => spineToScan(p.child)
+ case f: FilterExec if f.condition.deterministic => spineToScan(f.child)
+ case _ => false
+ }
+
+ // Nearest FilterExec whose spine down to the scan is Project/Filter-only (the DV shape
+ // interposes such nodes between them, so do not require a direct parent-child edge).
+ val filtersAboveScan = plan.collect {
+ case f: FilterExec if spineToScan(f.child) => f
+ }
+ filtersAboveScan.lastOption
+ .map { f =>
+ splitConjunctivePredicates(f.condition)
+ .filter(_.deterministic)
+ .filter(_.references.subsetOf(scanExec.outputSet))
+ .filter(p =>
+ SubqueryExpression.hasSubquery(p) || p.exists(_.isInstanceOf[PlanExpression[_]]))
+ .filterNot(p => scanExec.dataFilters.exists(_.semanticEquals(p)))
+ }
+ .getOrElse(Seq.empty)
+ }
+
+ /**
+ * Execution-time scalar-subquery data filters of a scan. `hasResolvedFilters` is true whenever
+ * pushdown is enabled and such filters exist, whether or not they bind or serialize; `protos`
+ * holds only the ones that serialized.
+ */
+ case class ResolvedSubqueryFilters(
+ hasResolvedFilters: Boolean,
+ protos: Seq[org.apache.comet.serde.ExprOuterClass.Expr])
+
+ private val NoResolvedSubqueryFilters = ResolvedSubqueryFilters(false, Seq.empty)
+
+ /**
+ * Resolve scalar-subquery data filters at execution time and serialize them for native
+ * pushdown, mirroring `CometNativeScanExec.serializedPartitionData`. `supportedDataFilters`
+ * excludes PlanExpressions at planning time (subquery results do not exist yet), so these
+ * bounds reach the native reader only through this path. Filters that fail to serialize are
+ * skipped: Spark keeps a covering FilterExec above the scan, so this is missed pruning only.
+ * Their presence is still reported, since native keys the safe timestamp conversion on the scan
+ * being filtered at all, as the core scan does for its resolved filters.
+ *
+ * Known core-parity limitation: when fused under a parent native operator,
+ * `ensureSubqueriesResolved` has already called `updateResult()` on these subqueries and this
+ * path calls it again (`ScalarSubquery.updateResult` re-executes unconditionally); benign here
+ * since the subquery's snapshot is pinned at analysis, but wasteful. Fix belongs in core.
+ */
+ def resolvedSubqueryFilters(
+ dataFilters: Seq[org.apache.spark.sql.catalyst.expressions.Expression],
+ output: Seq[org.apache.spark.sql.catalyst.expressions.Attribute],
+ requiredSchema: StructType,
+ conf: org.apache.spark.sql.internal.SQLConf): ResolvedSubqueryFilters = {
+ if (!conf.getConf(org.apache.spark.sql.internal.SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ return NoResolvedSubqueryFilters
+ }
+ val subqueryFilters = dataFilters.filter(_.exists(_.isInstanceOf[ExecScalarSubquery]))
+ if (subqueryFilters.isEmpty) {
+ return NoResolvedSubqueryFilters
+ }
+ // Same binding guard as the DV shape's plan-time filters: references limited to the
+ // data-column prefix of the output, where positions agree with the native read schema.
+ // Guard BEFORE updateResult so discarded filters never execute their subqueries.
+ val strippedLen = requiredSchema.count(f => !internalColumnNames.contains(f.name))
+ val dataColIds = output.take(strippedLen).map(_.exprId).toSet
+ val pushableFilters =
+ subqueryFilters.filter(_.references.forall(r => dataColIds.contains(r.exprId)))
+ pushableFilters.foreach(_.foreach {
+ case s: ExecScalarSubquery => s.updateResult()
+ case _ =>
+ })
+ val protos = pushableFilters
+ .flatMap { filter =>
+ // MergeScalarSubqueries can fuse several scalar subqueries into one struct-returning
+ // subquery accessed via GetStructField; fold that whole subtree to a literal (a bare
+ // GetStructField-over-Literal would not serialize).
+ val resolved = filter.transform {
+ case g @ org.apache.spark.sql.catalyst.expressions
+ .GetStructField(_: ExecScalarSubquery, _, _) =>
+ Literal.create(g.eval(null), g.dataType)
+ case s: ExecScalarSubquery =>
+ Literal.create(s.eval(null), s.dataType)
+ }
+ val proto = exprToProto(resolved, output)
+ if (proto.isEmpty) {
+ logWarning(s"Could not serialize resolved scalar subquery filter: $resolved")
+ }
+ proto
+ }
+ ResolvedSubqueryFilters(hasResolvedFilters = true, protos)
+ }
+
+ /**
+ * Allocate the partition-schema slots for the DV shape's internal columns
+ * (`internalColumnNames`), with names collision-free against the physical data schema, the
+ * physical partition schema, and the constant-metadata slots already allocated for this scan
+ * (plus each other): DataFusion substitutes partition constants BY NAME, so an unprefixed,
+ * un-uniquified slot could collide with a real column and silently replace its data with the
+ * bookkeeping constant. `buildDvScanCommon` keys `internalIndexByName` by each field's ORIGINAL
+ * name from `requiredSchema`, so the renaming here only changes the proto's field name.
+ */
+ private[delta] def allocateUniqueInternalFields(
+ requiredSchema: StructType,
+ physicalDataSchema: StructType,
+ physicalPartitionSchema: StructType,
+ constantMetadataFields: Seq[StructField]): Seq[StructField] = {
+ val reserved = scala.collection.mutable.LinkedHashSet[String]()
+ reserved ++= physicalDataSchema.fields.map(_.name)
+ reserved ++= physicalPartitionSchema.fields.map(_.name)
+ reserved ++= constantMetadataFields.map(_.name)
+ requiredSchema.fields.toSeq
+ .filter(f => internalColumnNames.contains(f.name))
+ .map { f =>
+ var name = s"$deltaConstFieldPrefix${f.name}"
+ while (reserved.contains(name)) {
+ name = name + "_"
+ }
+ reserved += name
+ StructField(name, f.dataType, f.nullable)
+ }
+ }
+
+ /**
+ * DV shape common builder. Layout invariants (declined by DeltaScanSupport when violated): scan
+ * output = requiredSchema attrs (data columns, then the internal columns as a suffix) followed
+ * by partition and constant-metadata columns. The parquet read schema strips the internal
+ * columns; they are appended to the partition schema as per-file constants, so the projection
+ * vector routes them from the constants block.
+ */
+ private def buildDvScanCommon(
Review Comment:
> Could ordinary-column construction in this path share the same builder as `CometNativeScan.buildNativeScanCommon`?
Tracked in #5945. The deletion-vector path keeps its own construction here; the issue records the view-without-generated-columns approach you outline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4081142447
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ *
+ * `private[delta]` (not `private`): [[DeltaScanSupport.declineReason]]'s non-ASCII
+ * case-insensitive name gate reuses this exact conversion to compute the names native sees
+ * under column mapping, rather than re-deriving physical names with separate logic.
+ */
+ private[delta] def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
Review Comment:
> Could `toPhysical` delegate to `DeltaParquetFileFormat.prepareSchemaForRead` instead of duplicating `createPhysicalSchema` and name-mode field-ID stripping?
Tracked in #5949, together with keeping the physical read schemas apart from the logical output types in `common.fields`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4081037622
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1838 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
Review Comment:
> Could the input-file-expression, vectorized-reader-setting, and nested-default checks come from a shared native-reader admission helper?
Tracked in #5944, with the parity test across ordinary Parquet and Delta as its acceptance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5908003633
@sunchao could you approve a CI run on `bdc463119`? This head merges main to resolve a conflict with #6351 in `CometNativeScan`: the timezone normalization now sits in `populateScanConfFlags`, so the core scan and both Delta scans get it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5328427972
Hi @sunchao,
On #4952 as the foundation: we already share more than it might look like. This PR builds on part 1 of that same breakup (#4700's CometScanWithPlanData / PlanDataInjector SPI) and keeps #4366's contrib shape, decline-gate philosophy, and test catalog, with co-authored-by credit to both earlier efforts. The remaining overlap is contrib infrastructure, and I'm glad to reconcile it once #4952 lands: adopt its contrib-delta profile and feature naming, the per-Spark delta.version matrix, the verify-gate script, and unify the proto slot (this PR is at 119, #4952 at 118). For the claim hook I'd suggest the generic CometScanRuleExtension SPI from this PR, since it keeps core free of Delta-specific code and the kernel path can register through it the same way.
I do see the two read paths as different layers rather than one thing to converge on. By the time CometScanRule sees the scan, delta-spark has already done log replay, time travel, and partition pruning, so this path reuses Comet's existing native parquet scan and gets row-group pruning, page-index pruning, and filter pushdown for free. DVs become ParquetAccessPlans that DataFusion intersects with page-index pruning, so DV skips and page skips compose in one scan. As far as I know no vectorized Delta reader does all of that today, including kernel's, which has no page-index pruning. I'd want convergence to keep this as the default read path, with the kernel path covering what JVM planning can't reach (DSv2, non-Spark frontends, likely CDF and row tracking).
On splitting: I'd push back on slicing by feature, for two reasons. First, the features aren't independent. Several decline gates only exist because DVs, column mapping, and Delta's own suites ran together. For example, Delta's findTouchedFiles scan looks like a plain read, and if a basic-reads slice claims it, DELETE silently rewrites files instead of writing DVs. Second, the proof is holistic: this branch runs Delta's own suites at 1156/1156 and the contrib suites at 39/39 on Spark 3.5, 4.0, and 4.1. Feature slices would decline most tables and couldn't run that meaningfully. What I can do is split along review surfaces instead: core SPI additions, native DV decode with its unit tests, the contrib module and read path, and the regression harness and CI, keeping the read path itself (DVs, column mapping, gates) as one reviewable unit. If it lands whole, Comet ships the only vectorized Delta reader with complete skipping.
The Spark 4 milestone is already met, the suites are green on 4.0 and 4.1 today. Row tracking and CDF are out of scope here and seem like a natural place for the kernel work to lead. Happy to set up a chat with you and @schenksj to work out the details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5389347576
> ### [P2] Also handle plain long-form S3 bucket credentials
> The long-form JCEKS cases are fixed, but the same compatibility gap remains for **plain configuration values** at `514f71e0`.
>
> With `fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` and only these plain keys configured:
>
> ```
> fs.s3a.bucket.mybucket.fs.s3a.access.key
> fs.s3a.bucket.mybucket.fs.s3a.secret.key
> ```
>
> [`bucketCredentialAliasReason`](https://github.com/apache/datafusion-comet/blob/514f71e0d09f4b0e85964014afae17363fa2dba1/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala#L712-L713) returns `None` because no credential-provider path is set. Hadoop resolves these keys, but `NativeConfig` forwards them unchanged and the [native lookup](https://github.com/apache/datafusion-comet/blob/514f71e0d09f4b0e85964014afae17363fa2dba1/native/core/src/parquet/objectstore/s3.rs#L299-L318) reads only short bucket keys and globals. The native Simple provider therefore has no credentials. If different plain global credentials are also configured, Hadoop uses the long bucket aliases while native uses those globals instead. Claiming the Delta scan exposes this mismatch to reads that previously used Hadoop successfully.
>
> I verified both cases with actual Hadoop 3.4.1 `SimpleAWSCredentialsProvider` and the unchanged current admission/extraction methods, before and after `S3AUtils.propagateBucketOptions`, plus exact-source Rust lookup tests. Short-only and global-only controls agree. This was not a full Comet/JNI scan or a live S3 request.
>
> Could we conservatively decline these configurations, or normalize the effective credentials using Hadoop's precedence, and add plain-long-only and global-shadowing regressions?
>
> This is a remaining case of the [existing S3-authentication P2](https://github.com/apache/datafusion-comet/pull/5365#discussion_r3835076674), posted here for visibility rather than as another inline thread.
@sunchao
Fixed at 6d20ad69f. The gate now also compares plain values: for each bucket it computes Hadoop's effective credentials (short overrides long, then long, then global) against what the native lookup would use (short then global) and declines whenever a long-form key is set and the two diverge. Pure conf-map reads, so still zero I/O. Added plain-long-only and global-shadowing regressions plus controls for short-overrides-long and long-equals-global. Also rebased onto latest main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5389343727
Fixed at 6d20ad69f. The gate now also compares plain values: for each bucket it computes Hadoop's effective credentials (short overrides long, then long, then global) against what the native lookup would use (short then global) and declines whenever a long-form key is set and the two diverge. Pure conf-map reads, so still zero I/O. Added plain-long-only and global-shadowing regressions plus controls for short-overrides-long and long-equals-global. Also rebased onto latest main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5399704303
### [P2] Avoid forcing scalar partition subqueries from outputPartitioning
The adaptive-broadcast guard fixes the earlier DPP shape, but [this getter](https://github.com/apache/datafusion-comet/blob/6d20ad69fd39f52c35230d2fa5373aa215e7ed5e/contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala#L75-L91) still forces `perPartitionData` when a partition filter contains an unresolved `ScalarSubquery` rather than `InSubqueryExec`.
With AQE/coalescing enabled, a Delta `fact` table partitioned by `p`, and Parquet `dim`/`other` tables, this shape reaches the failure:
```sql
SELECT id, p FROM fact
WHERE p = (SELECT max(p) FROM dim WHERE country = 'yes')
UNION ALL
SELECT cast(count(*) AS int) AS id, p FROM other GROUP BY p
```
A reduced Spark 4.0.2 / Delta 4.0.0 planning harness, using the current production contribution registry and keeping the parent operators on Spark, reproduces:
```text
ValidateRequirements
-> CometDeltaNativeScanExec.outputPartitioning
-> serializedPartitionData / getFilePartitions
-> ScalarSubquery.eval
-> requirement failed: Subquery ... has not finished
```
The claim has already returned successfully, so the registry's exception fallback cannot catch this. `doPrepare`/execution-time subquery resolution also occurs too late. Stock Spark completes the query; changing only the scratch probe's getter to return non-forcing `UnknownPartitioning(0)` gets past this validation and reaches the custom AQE optimizer. No Comet/JNI reader executed in the probe.
Could we keep partitioning inspection from evaluating unresolved scalar partition filters as well, with a regression covering this mixed `UNION ALL`/shuffle shape? This is a remaining case of the [existing AQE P2](https://github.com/apache/datafusion-comet/pull/5365#discussion_r3835076676).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5399703778
### [P2] Use Hadoop's long-before-short bucket credential precedence
The long-only and divergent-global cases from the [previous comment](https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5389049677) are fixed at `6d20ad69`, but conflicting **long and short bucket keys** still pass the new guard.
The [comparison](https://github.com/apache/datafusion-comet/blob/6d20ad69fd39f52c35230d2fa5373aa215e7ed5e/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala#L670-L674) computes Hadoop's effective value as `short.orElse(long).orElse(global)`. Actual Hadoop 3.4.1 resolves the long bucket form first; a nonempty long value is retained rather than overridden by the short form.
For example, with distinct fixture credentials:
```text
fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
fs.s3a.bucket.mybucket.fs.s3a.access.key=long-ak
fs.s3a.bucket.mybucket.fs.s3a.secret.key=long-sk
fs.s3a.bucket.mybucket.access.key=short-ak
fs.s3a.bucket.mybucket.secret.key=short-sk
```
The current admission helper returns `None`. Hadoop's `SimpleAWSCredentialsProvider` selects the long pair, while native selects the short pair. A Delta read that works through Hadoop can therefore fail authentication or use a different identity after being claimed.
I verified this using actual Hadoop 3.4.1 and the current admission/config-extraction methods, both before and after `S3AUtils.propagateBucketOptions`, plus exact-source Rust lookup tests. No live S3 request or full JNI scan was used.
Could the plain-value comparison use Hadoop's actual long-before-short precedence, and could the divergent-short regression check the real provider instead of asserting that the short form wins? This is a remaining case of the existing S3-authentication P2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5344589643
> Thanks @dwsmith1983 Your proposed split by review surface sounds reasonable. I agree that the reader, its safety gates, and the essential DML/fallback tests should stay together. Thanks also for being open to aligning with #4952 once it lands. We can leave row tracking and CDF for later discussions rather than expand this PR’s scope. The main additional point I’d like us to settle is keeping experimental Delta support explicitly opt-in.
@sunchao
Yeah, agreed on explicit opt-in. It's mostly already set up that way. All the Delta code lives in a separate comet-contrib-delta jar that never gets bundled into comet-spark, so a stock Comet install has no Delta surface at all. If we publish that jar with releases, trying it out is just --packages and a conf, nobody has to build from source. Right now the conf defaults to on when the jar is present though, so I'll flip spark.comet.scan.delta.enabled to default false to make the opt-in explicit.
The one spot where I'd differ from #4952's gate is the native binary. The Delta bits in libcomet are tiny (DV decoding plus a hand-off to the existing parquet scan, no delta-kernel dependency) and can't be reached without the jar and the conf. I'd rather keep them in the default build than make people compile their own native binary to try an experimental feature. Sound reasonable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5605696104
Both items are in the head (a48544e45).
Selected paths: `declineReason` now probes every distinct selected data-file and deletion-vector URI rather than their parent directories. The probe is a URL parse in the native library with no I/O, measured at 0.75 microseconds per file over 200,000 paths, so the per-file cost sits below the scan's own per-file work. The converted case is a test: a Parquet directory with one data file renamed to carry a newline, `CONVERT TO DELTA` keeping the basename, and the query falls back to Spark with the matching answer and a reason naming that file. On the previous head that table was claimed natively.
Duplicate classes: the root pom no longer carries the comet-common exception. The exclusion alternative packages fine but fails the reactor test lifecycle with `NoClassDefFoundError: org/apache/comet/CometRuntimeException`, because a reactor test run resolves comet-spark from `target/classes` unshaded and the exclusion removes the only copy of those classes, so that option is out. The exception now lives in the contrib module's own enforcer execution, same execution id and `combine.children="append"`, so it extends the inherited rule for that module only. Validated with the delta profile: clean package, and clean test with DeltaScanContribSuite at 134; without the profile, validate shows no contrib module and no exception; the effective pom of the contrib module lists the root's dependencies plus comet-common; and the delta build gate script passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3943821181
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1789 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // An opted-in S3-compliant alias scheme (fs.comet.s3Compliant.schemes) is declined before the
+ // generic scheme gate below so the reason says why: core's native scan reads it through the
+ // S3 client, but the S3 divergence gates further down model Hadoop's S3AFileSystem only.
+ val aliasReason =
+ s3CompliantAliasSchemeReason(hadoopConf, scanExec.relation.location.rootPaths.map(_.toUri))
+ if (aliasReason.isDefined) {
+ return aliasReason
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Always zero-I/O (plain propagated-conf read, no keystore): native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Zero-I/O, conf-only, like the proxy gate above: Hadoop's AssumedRoleCredentialProvider
+ // sends fs.s3a.assumed.role.policy as the session policy of its STS AssumeRole request,
+ // while native's assumed-role provider never reads the key -- a claimed scan would assume
+ // the role WITHOUT the configured session restriction, silently widening permissions.
+ val rolePolicyReason =
+ assumedRolePolicyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (rolePolicyReason.isDefined) {
+ return rolePolicyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * parsed exactly like core's scan gate (`NativeConfig.parseSchemeSet`: split on commas,
+ * trimmed, lowercased) and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) => NativeConfig.parseSchemeSet(s)
+ case None => Set("hdfs")
+ }
+
+ /**
+ * Decline reason when any of `uris` uses a scheme opted in as an S3-compliant alias through
+ * `fs.comet.s3Compliant.schemes` (e.g. `blob`), or `None`. Core's native Parquet scan admits
+ * such a scheme and reads it through its S3 client, with `NativeConfig` translating the vendor
+ * `fs.<scheme>.<authority>.*` keys into `fs.s3a.bucket.*` options. Spark, however, reads the
+ * same table through the vendor's own Hadoop FileSystem, not `S3AFileSystem`, and every S3
+ * divergence gate in this object ([[s3ConfigDivergenceReason]] and its siblings) is verified
+ * against `S3AFileSystem`'s consumers only. With no model of how the vendor filesystem resolves
+ * its configuration, whether native and Spark would agree cannot be decided, so the scan is
+ * declined rather than claimed on a guess. Selected data-file and deletion-vector URIs under an
+ * alias scheme are declined by the generic scheme gates, which never admit an alias (see
+ * [[unsupportedSchemes]]).
+ */
+ private[delta] def s3CompliantAliasSchemeReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val aliases = NativeConfig.resolveS3CompliantSchemes(hadoopConf)
+ if (aliases.isEmpty) {
+ return None
+ }
+ val found = uris
+ .flatMap(uri => Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)))
+ .filter(aliases.contains)
+ .distinct
+ if (found.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support S3-compliant alias filesystem scheme(s) " +
+ s"${found.sorted.mkString(", ")} (${CometConf.COMET_S3_COMPLIANT_SCHEMES_KEY}): " +
+ "Spark reads them through a vendor filesystem whose S3 configuration resolution the " +
+ "native scan's S3AFileSystem divergence model cannot verify")
+ }
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source. The alias
+ * set handed to core's gate is deliberately empty: an `fs.comet.s3Compliant.schemes` alias is
+ * never admitted here (see [[s3CompliantAliasSchemeReason]]), even though core admits it.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri, Set.empty)
Review Comment:
Right, the rebase over #5314 dropped that check on the Delta side. Fixed in f1c9b082d, which also carries the DataFusion 55 and Arrow 59.2 rebase. `objectStoreAcceptsPath` is widened to `private[comet]` the same way `isNativelyReadableScheme` is, and the contrib applies it at two points in `declineReason`: on the table root URIs right after the root scheme gate, and on the distinct parent directories of the selected data files and deletion vectors right after the selected-file scheme gate, so a wide selection probes once per directory rather than per file. libhdfs schemes skip it as in core; aliases are declined earlier. The reason string renders the URI with any userinfo masked.
Regressions: unit tests in `DeltaScanContribSuite` for the rejected root, an ordinary path, and a libhdfs-exempt path with the same character; end to end in `CometDeltaNativeScanSuite`, a table under a directory whose name contains a newline falls back to Spark with a matching answer, and a shallow clone whose source files sit under such a directory falls back through the selected-directories gate. On the previous head the first of those planned a `CometDeltaNativeScan` over `dir%0A/data`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3917732904
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -76,6 +77,583 @@ fn schema_has_field_ids(schema: &SchemaRef) -> bool {
schema.fields().iter().any(|f| parse_field_id(f).is_some())
}
+// ---------------------------------------------------------------------------------------------
+// JVM-shipped case tables: reproduce the PLANNING JVM's `String.toLowerCase(Locale.ROOT)`,
+// which Spark's Parquet footer field matching is built on.
+//
+// The data arrives on `NativeScanCommon` (populated by `JvmCaseTables.scala` when
+// case_sensitive = false), generated from the very JVM that plans the query, so the native
+// matcher is correct BY CONSTRUCTION for whatever JDK runs Spark. The one contextual mapping
+// (`Locale.ROOT` has exactly one: Greek capital sigma U+03A3) cannot be a per-codepoint table
+// entry, so its condition is ported as an algorithm over a shipped per-codepoint
+// classification -- see `JvmCaseTables::lowercase`.
+// ---------------------------------------------------------------------------------------------
+
+// Sigma-scan classes: the wire contract shared with `JvmCaseTables.scala`. The classes are
+// the UAX#29-style word-break classes (ALetter, Numeric, MidLetter, MidNum, MidNumLet,
+// Extend, Format) as the PLANNING JVM's legacy break iterator actually realizes them --
+// probed per codepoint from `BreakIterator.isBoundary` on the JVM side -- plus classes for
+// its pre-UAX#29 extensions (the danda and the supplementary-plane behaviors of its UTF-16
+// DFA), with cased variants split out via the JDK's `isCased`. Any codepoint outside every
+// shipped range -- and any class value this build does not know -- is a word boundary: the
+// sigma context scan stops there, which is also the safe reading for class values added by a
+// NEWER JVM-side generator.
+const CLASS_ALETTER_CASED: u8 = 1;
+const CLASS_ALETTER: u8 = 2;
+const CLASS_NUMERIC: u8 = 3;
+const CLASS_MID_LETTER: u8 = 4;
+const CLASS_MID_NUM: u8 = 5;
+const CLASS_MID_NUM_LET: u8 = 6;
+/// Cased supplementary char: attaches to the preceding word and closes it (and forms a word
+/// of its own at raw text start).
+const CLASS_SUPP_CASED: u8 = 7;
+/// U+0964/U+0965: word-terminal, chains only into digits.
+const CLASS_DANDA: u8 = 8;
+/// U+0345, the one cased combining mark: cased only when its run is attached to a word.
+const CLASS_EXTEND_CASED: u8 = 9;
+/// Cased digit-base (Nl Roman numerals): joins like CLASS_ALETTER_CASED when reached
+/// directly, but bridges only mid-num punctuation, never mid-letter.
+const CLASS_NUMERIC_CASED: u8 = 10;
+/// Non-cased Mn/Me marks: riders that attach only to genuine letter/digit bases.
+const CLASS_EXTEND: u8 = 11;
+/// Word-forming non-cased supplementary letter: a genuine letter-base that closes the word
+/// immediately after itself.
+const CLASS_SUPP_LETTER: u8 = 12;
+/// Cf format characters: fully transparent (WB4-style) -- deleted from the sequence before
+/// the scans run, so a pure-format rider chain bridges mid punctuation ("AΣ-<ZWJ>b" is one
+/// word exactly like "AΣ-b").
+const CLASS_FORMAT: u8 = 13;
+/// Supplementary chars that attach to the preceding word but never form one themselves
+/// (supplementary combining marks, tag characters): a cased mark riding on one belongs to
+/// the sigma's word only when the run hangs off a real base (`supp_mn_anchor`).
+const CLASS_SUPP_MN: u8 = 14;
+/// Word-forming supplementary digit: like CLASS_SUPP_LETTER except a riding cased mark
+/// carries only across mid-num (digit-context) punctuation, never mid-letter.
+const CLASS_SUPP_NUM: u8 = 15;
+/// Not on the wire: the absence of a class.
+const CLASS_BOUNDARY: u8 = 0;
+
+const CAPITAL_SIGMA: char = '\u{03A3}';
+const SMALL_SIGMA: char = '\u{03C3}';
+const SMALL_FINAL_SIGMA: char = '\u{03C2}';
+
+/// What a supplementary-mark run ultimately hangs off (see `supp_mn_anchor`).
+#[derive(PartialEq, Eq, Clone, Copy)]
+enum SuppMnAnchor {
+ None,
+ Letter,
+ Digit,
+}
+
+/// The planning JVM's case data, parsed once per scan from `NativeScanCommon` and attached to
+/// [`SparkParquetOptions`]. Two tables:
+///
+/// - `lower`: every codepoint the JVM lowercases non-identically, with its full (possibly
+/// multi-char, e.g. U+0130 -> "i" + U+0307) replacement; codepoints absent here lowercase
+/// to themselves;
+/// - `class_ranges`: sorted, disjoint `(start, end, class)` codepoint ranges holding the
+/// word-break classification the JVM probed from its own `BreakIterator`.
+///
+/// `lowercase` applies Java's algorithm over that data: per codepoint, U+03A3 takes its
+/// contextual final/non-final form via the ported `isFinalCased` condition -- word-boundary
+/// based (the JDK's legacy break-iterator word rules), NOT the Unicode-standard Final_Sigma
+/// case-ignorable skip, so e.g. "A1Σ" lowers to "a1ς" -- and every other codepoint takes its
+/// table replacement. `JvmCaseTables.mirrorLowercase` on the Scala side is the line-for-line
+/// mirror of this function over the same generated data; the JVM-side parity suite proves the
+/// pair equal to the running JDK's `String.toLowerCase(Locale.ROOT)` across the full codepoint
+/// space (calibrated to zero mismatches on JDK 17, 21, and 25).
+#[derive(Debug)]
+pub struct JvmCaseTables {
+ /// Non-identity lowercase mappings: codepoint -> full replacement string.
+ lower: HashMap<char, String>,
+ /// Sorted, disjoint (start, end, class) inclusive codepoint ranges for the sigma scan.
+ class_ranges: Vec<(u32, u32, u8)>,
+ /// Precomputed content hash so `SparkParquetOptions`'s derived `Hash` stays cheap.
+ fingerprint: u64,
+}
+
+impl PartialEq for JvmCaseTables {
+ fn eq(&self, other: &Self) -> bool {
+ self.fingerprint == other.fingerprint
+ && self.class_ranges == other.class_ranges
+ && self.lower == other.lower
+ }
+}
+
+impl Eq for JvmCaseTables {}
+
+impl Hash for JvmCaseTables {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ // Equal contents always produce the equal (deterministically computed) fingerprint,
+ // so hashing only the fingerprint is consistent with `PartialEq`.
+ state.write_u64(self.fingerprint);
+ }
+}
+
+fn is_letter_base(cls: u8) -> bool {
+ cls == CLASS_ALETTER_CASED
+ || cls == CLASS_ALETTER
+ || cls == CLASS_SUPP_CASED
+ || cls == CLASS_SUPP_LETTER
+}
+
+fn is_digit_base(cls: u8) -> bool {
+ cls == CLASS_NUMERIC || cls == CLASS_NUMERIC_CASED
+}
+
+impl JvmCaseTables {
+ /// Parse the proto representation: `lower_cp`/`lower_repl` are index-aligned, and
+ /// `class_ranges` holds (start, end, class) triples. Malformed input (length mismatch,
+ /// trailing partial triple, out-of-range codepoints) is dropped entry-by-entry rather
+ /// than rejected: every dropped entry degrades one codepoint to identity/boundary
+ /// behavior instead of failing the scan.
+ pub fn from_proto(lower_cp: &[u32], lower_repl: &[String], class_ranges: &[u32]) -> Self {
+ let mut lower = HashMap::with_capacity(lower_cp.len());
+ for (cp, repl) in lower_cp.iter().zip(lower_repl.iter()) {
+ if let Some(c) = char::from_u32(*cp) {
+ lower.insert(c, repl.clone());
+ }
+ }
+ let ranges: Vec<(u32, u32, u8)> = class_ranges
+ .as_chunks::<3>()
+ .0
+ .iter()
+ .filter(|t| t[0] <= t[1] && t[1] <= 0x10FFFF && u8::try_from(t[2]).is_ok())
+ .map(|t| (t[0], t[1], t[2] as u8))
+ .collect();
+
+ let mut hasher = DefaultHasher::new();
+ for (start, end, class) in &ranges {
+ hasher.write_u32(*start);
+ hasher.write_u32(*end);
+ hasher.write_u8(*class);
+ }
+ let mut lower_sorted: Vec<(&char, &String)> = lower.iter().collect();
+ lower_sorted.sort_by_key(|(c, _)| **c);
+ for (c, repl) in lower_sorted {
+ hasher.write_u32(*c as u32);
+ hasher.write(repl.as_bytes());
+ }
+
+ Self {
+ lower,
+ class_ranges: ranges,
+ fingerprint: hasher.finish(),
+ }
+ }
+
+ /// Sigma-scan class of `c`; `CLASS_BOUNDARY` when no shipped range covers it.
+ fn class_of(&self, c: char) -> u8 {
+ let cp = c as u32;
+ let mut lo = 0usize;
+ let mut hi = self.class_ranges.len();
+ while lo < hi {
+ let mid = (lo + hi) / 2;
+ let (start, end, class) = self.class_ranges[mid];
+ if cp < start {
+ hi = mid;
+ } else if cp > end {
+ lo = mid + 1;
+ } else {
+ return class;
+ }
+ }
+ CLASS_BOUNDARY
+ }
+
+ /// First position at or beyond `start` (stepping by `step`, i.e. -1 backward / +1
+ /// forward) whose class is not CLASS_EXTEND. Returns `None` if the scan runs off the
+ /// array without finding one.
+ fn skip_extends(&self, cps: &[char], start: isize, step: isize) -> Option<usize> {
+ let mut k = start;
+ while k >= 0 && (k as usize) < cps.len() {
+ if self.class_of(cps[k as usize]) != CLASS_EXTEND {
+ return Some(k as usize);
+ }
+ k += step;
+ }
+ None
+ }
+
+ /// As [`Self::skip_extends`], but also skips CLASS_EXTEND_CASED, reporting whether one
+ /// was walked: a cased mark (U+0345) crossed while looking for a base is itself cased
+ /// whenever the landing validates the run.
+ fn skip_extends_tracking_cased(
+ &self,
+ cps: &[char],
+ start: isize,
+ step: isize,
+ ) -> (Option<usize>, bool) {
+ let mut k = start;
+ let mut saw_cased = false;
+ while k >= 0 && (k as usize) < cps.len() {
+ let cls = self.class_of(cps[k as usize]);
+ if cls != CLASS_EXTEND && cls != CLASS_EXTEND_CASED {
+ return (Some(k as usize), saw_cased);
+ }
+ if cls == CLASS_EXTEND_CASED {
+ saw_cased = true;
+ }
+ k += step;
+ }
+ (None, saw_cased)
+ }
+
+ /// What the supplementary-mark run at `k` (CLASS_SUPP_MN) ultimately hangs off, walking
+ /// down through further marks and supplementary chars: a letter-flavored base, a
+ /// digit-flavored base, or nothing word-forming. A cased mark riding the run belongs to
+ /// the sigma's word only per this anchor.
+ fn supp_mn_anchor(&self, cps: &[char], k: usize) -> SuppMnAnchor {
+ let mut m = k as isize - 1;
+ while m >= 0 {
+ let cls = self.class_of(cps[m as usize]);
+ if cls != CLASS_SUPP_MN && cls != CLASS_EXTEND && cls != CLASS_EXTEND_CASED {
+ break;
+ }
+ m -= 1;
+ }
+ if m < 0 {
+ return SuppMnAnchor::None;
+ }
+ match self.class_of(cps[m as usize]) {
+ CLASS_ALETTER_CASED | CLASS_ALETTER | CLASS_SUPP_CASED | CLASS_SUPP_LETTER => {
+ SuppMnAnchor::Letter
+ }
+ CLASS_NUMERIC | CLASS_NUMERIC_CASED | CLASS_SUPP_NUM => SuppMnAnchor::Digit,
+ _ => SuppMnAnchor::None,
+ }
+ }
+
+ /// Backward half of the ported `isFinalCased`: is there a cased letter before position
+ /// `i` within the sigma's word? Runs over the FORMAT-FILTERED sequence; `leading_format`
+ /// says whether format chars were filtered off the raw text start.
+ fn scan_back_finds_cased(&self, cps: &[char], i: usize, leading_format: bool) -> bool {
+ let mut last_letter = true; // the sigma itself is a letter
+ let mut j = i as isize - 1;
+ while j >= 0 {
+ match self.class_of(cps[j as usize]) {
+ CLASS_ALETTER_CASED | CLASS_NUMERIC_CASED => return true,
+ CLASS_ALETTER => {
+ last_letter = true;
+ j -= 1;
+ }
+ CLASS_NUMERIC => {
+ last_letter = false;
+ j -= 1;
+ }
+ CLASS_EXTEND => {
+ // Non-cased marks attach only to a real base below them; anything else
+ // (mid punctuation, danda, boundary, text start) leaves the run
+ // unattached.
+ let Some(k) = self.skip_extends(cps, j, -1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ let is_continuer = b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_EXTEND_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_NUM;
+ if !is_continuer {
+ return false;
+ }
+ j = k as isize;
+ }
+ CLASS_SUPP_CASED => {
+ // Closes the preceding word, so the scan stops -- except at RAW text
+ // start (no filtered-out leading format chars), where the DFA keeps it
+ // joined to what follows.
+ return j == 0 && !leading_format;
+ }
+ CLASS_SUPP_LETTER | CLASS_SUPP_MN | CLASS_SUPP_NUM => {
+ // Attach/close and never themselves cased; nothing beyond is reachable.
+ return false;
+ }
+ CLASS_EXTEND_CASED => {
+ // Cased combining mark (U+0345): cased when its run hangs off a base --
+ // a BMP letter/digit, a word-forming supplementary char (which closes a
+ // word right below the mark, merging the mark into the sigma's
+ // segment), or an ANCHORED supplementary mark.
+ let (Some(k), _) = self.skip_extends_tracking_cased(cps, j - 1, -1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ if b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_NUM
+ {
+ return true;
+ }
+ if b == CLASS_SUPP_MN {
+ return self.supp_mn_anchor(cps, k) != SuppMnAnchor::None;
+ }
+ return false;
+ }
+ CLASS_DANDA => {
+ // Backward across a danda: the word part before it must end in letters
+ // (grammar: letters, optional danda, then number+word chains) -- or
+ // carry a riding cased mark on a word-forming base, or be a cased
+ // supplementary char at text start -- and the danda itself chains only
+ // into digits after it.
+ if last_letter {
+ return false;
+ }
+ let (Some(k), saw_cased_mark) =
+ self.skip_extends_tracking_cased(cps, j - 1, -1)
+ else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ if b == CLASS_ALETTER_CASED {
+ return true;
+ }
+ if b == CLASS_SUPP_CASED {
+ return saw_cased_mark || (k == 0 && !leading_format);
+ }
+ if b == CLASS_SUPP_LETTER {
+ return saw_cased_mark;
+ }
+ if b == CLASS_SUPP_MN {
+ return saw_cased_mark
+ && self.supp_mn_anchor(cps, k) == SuppMnAnchor::Letter;
+ }
+ if b != CLASS_ALETTER {
+ return false;
+ }
+ if saw_cased_mark {
+ return true;
+ }
+ last_letter = true;
+ j = k as isize;
+ }
+ cls @ (CLASS_MID_LETTER | CLASS_MID_NUM | CLASS_MID_NUM_LET) => {
+ // `<mid-letter><let>` / `<mid-num><digit>` require a genuine
+ // letter/digit base before the punctuation; scanning backward
+ // legitimately walks marks-then-base (marks trail their base). A cased
+ // mark walked over rides whatever the punctuation hangs off, including
+ // a context-matching anchored supplementary mark or supplementary
+ // digit.
+ let mw_ok = cls == CLASS_MID_LETTER || cls == CLASS_MID_NUM_LET;
+ let mn_ok = cls == CLASS_MID_NUM || cls == CLASS_MID_NUM_LET;
+ let (Some(real_pos), saw_cased_mark) =
+ self.skip_extends_tracking_cased(cps, j - 1, -1)
+ else {
+ return false;
+ };
+ let b = self.class_of(cps[real_pos]);
+ if last_letter
+ && mw_ok
+ && saw_cased_mark
+ && b == CLASS_SUPP_MN
+ && self.supp_mn_anchor(cps, real_pos) == SuppMnAnchor::Letter
+ {
+ return true;
+ }
+ if !last_letter
+ && mn_ok
+ && saw_cased_mark
+ && (b == CLASS_SUPP_NUM
+ || (b == CLASS_SUPP_MN
+ && self.supp_mn_anchor(cps, real_pos) == SuppMnAnchor::Digit))
+ {
+ return true;
+ }
+ let bridge_valid = (last_letter && mw_ok && is_letter_base(b))
+ || (!last_letter && mn_ok && is_digit_base(b));
+ if !bridge_valid {
+ return false;
+ }
+ if saw_cased_mark {
+ return true;
+ }
+ j = real_pos as isize;
+ }
+ _ => return false,
+ }
+ }
+ false
+ }
+
+ /// Forward half of the ported `isFinalCased`: is there a cased letter after position `i`
+ /// within the sigma's word? Runs over the FORMAT-FILTERED sequence.
+ fn scan_fwd_finds_cased(&self, cps: &[char], i: usize) -> bool {
+ let mut last_letter = true;
+ let mut j = i + 1;
+ while j < cps.len() {
+ match self.class_of(cps[j]) {
+ CLASS_ALETTER_CASED | CLASS_NUMERIC_CASED => return true,
+ CLASS_ALETTER => {
+ last_letter = true;
+ j += 1;
+ }
+ CLASS_NUMERIC => {
+ last_letter = false;
+ j += 1;
+ }
+ CLASS_EXTEND => {
+ // A mark run trailing the anchor is properly attached in text order, so
+ // the run stays open past it, including onto mid punctuation on its far
+ // side.
+ let Some(k) = self.skip_extends(cps, j as isize, 1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ let is_continuer = b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_EXTEND_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_MN
+ || b == CLASS_SUPP_NUM
+ || b == CLASS_DANDA
+ || b == CLASS_MID_LETTER
+ || b == CLASS_MID_NUM
+ || b == CLASS_MID_NUM_LET;
+ if !is_continuer {
+ return false;
+ }
+ j = k;
+ }
+ CLASS_SUPP_CASED | CLASS_EXTEND_CASED => {
+ // Attaches to the current word, so the scan sees it (cased).
+ return true;
+ }
+ CLASS_SUPP_LETTER | CLASS_SUPP_MN | CLASS_SUPP_NUM => {
+ // Attach to the current word and close it; never themselves cased, and
+ // nothing beyond is reachable.
+ return false;
+ }
+ // The danda attaches only to a word part that ends in letters (reached
+ // after digits the word is already closed) and continues only into a digit
+ // -- unless that digit is itself cased (a Roman numeral), which resolves
+ // the scan immediately.
+ CLASS_DANDA if !last_letter => return false,
+ CLASS_DANDA
+ if j + 1 < cps.len() && self.class_of(cps[j + 1]) == CLASS_NUMERIC_CASED =>
+ {
+ return true;
+ }
+ CLASS_DANDA if j + 1 < cps.len() && self.class_of(cps[j + 1]) == CLASS_NUMERIC => {
+ last_letter = false;
+ j += 2;
+ }
+ cls @ (CLASS_MID_LETTER | CLASS_MID_NUM | CLASS_MID_NUM_LET) => {
+ // `<mid-letter><let>` / `<mid-num><digit>` require a genuine
+ // letter/digit base IMMEDIATELY after the punctuation -- unlike the
+ // backward scan, marks here are never skipped past: a mark directly
+ // after the punctuation is attached to the punctuation, not a base, so
+ // it blocks the bridge. (Format chars are already filtered out, which
+ // is what lets "AΣ-<ZWJ>b" bridge exactly like "AΣ-b".)
+ let mw_ok = cls == CLASS_MID_LETTER || cls == CLASS_MID_NUM_LET;
+ let mn_ok = cls == CLASS_MID_NUM || cls == CLASS_MID_NUM_LET;
+ if j + 1 >= cps.len() {
+ return false;
+ }
+ let b = self.class_of(cps[j + 1]);
+ if (last_letter && mw_ok && is_letter_base(b))
+ || (!last_letter && mn_ok && is_digit_base(b))
+ {
+ j += 1;
+ } else {
+ return false;
+ }
+ }
+ _ => return false,
+ }
+ }
+ false
+ }
+
+ /// Lowercase `s` the way the planning JVM's `String.toLowerCase(Locale.ROOT)` does.
+ pub fn lowercase(&self, s: &str) -> String {
+ let raw: Vec<char> = s.chars().collect();
+ // Built lazily on the first sigma: the format-filtered sequence the scans run over
+ // (WB4-style: the legacy break iterator's `<ignore>` class loops on every DFA
+ // state), the raw->filtered index map, and whether format chars led the raw text.
+ let mut filtered: Option<(Vec<char>, Vec<usize>, bool)> = None;
+ let mut out = String::with_capacity(s.len());
+ for (i, &c) in raw.iter().enumerate() {
+ if c == CAPITAL_SIGMA {
+ // The condition consults the ORIGINAL neighbors, exactly as the JDK scans
+ // `src`, not the partially-lowered output. The final/non-final target chars
+ // are Unicode-stable (pinned in `ConditionalSpecialCasing`'s entry table).
+ let (f, idx, leading_format) = filtered.get_or_insert_with(|| {
+ let mut f = Vec::with_capacity(raw.len());
+ let mut idx = vec![0usize; raw.len()];
+ for (k, &rc) in raw.iter().enumerate() {
+ idx[k] = f.len();
+ if self.class_of(rc) != CLASS_FORMAT {
+ f.push(rc);
+ }
+ }
+ let leading_format = self.class_of(raw[0]) == CLASS_FORMAT;
+ (f, idx, leading_format)
+ });
+ let fi = idx[i];
+ let is_final = self.scan_back_finds_cased(f, fi, *leading_format)
+ && !self.scan_fwd_finds_cased(f, fi);
+ out.push(if is_final {
+ SMALL_FINAL_SIGMA
+ } else {
+ SMALL_SIGMA
+ });
+ } else if let Some(repl) = self.lower.get(&c) {
+ out.push_str(repl);
+ } else {
+ out.push(c);
+ }
+ }
+ out
+ }
+}
+
+/// Lowercase `s` for case-insensitive field matching. With tables (populated whenever
+/// `case_sensitive = false`, shared by the core Parquet scan and the Delta contrib scan) this
+/// reproduces the planning JVM's `String.toLowerCase(Locale.ROOT)` exactly.
+///
+/// Without tables, fall back to Rust's `str::to_lowercase`, which agrees with Java on all
+/// simple mappings and differs only where the two Unicode snapshots or the sigma context
+/// diverge. This is a real, live path, not just a defensive default: the Iceberg native scan
+/// (`SparkPhysicalExprAdapterFactory::new(_, None)`) defaults `case_sensitive = false` and
+/// always reaches this fallback for its schema name remap, alongside
+/// `parquet_convert_struct_to_struct`'s general struct-cast matching and Rust-only unit tests
+/// that construct `SparkParquetOptions` directly.
+pub(crate) fn java_lowercase(s: &str, tables: Option<&JvmCaseTables>) -> String {
Review Comment:
[P2] Thanks for measuring this. I confirmed the shared Parquet path at `020f0399` and ran an allocation-only probe of the exact matcher with JDK 21 ASCII mappings. A search across 200 names made 20,100 comparisons and 160,800 allocation or reallocation calls, versus zero with a fast path guarded by `a.is_ascii() && b.is_ascii()`. This was not a Parquet or Spark timing run.
I agree this should be addressed before merge while retaining JVM lowercasing whenever either name is non-ASCII. One qualification on frequency: DataFusion skips remapping when the schemas are identical and there is no predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3917353053
##########
native/proto/src/proto/operator.proto:
##########
@@ -77,6 +77,8 @@ message Operator {
// zero runtime cost.
ContribScan contrib_scan = 200;
}
+
+ reserved "delta_scan";
Review Comment:
`delta_scan` was never on `main`. Field 119 is and always was `window_group_limit`, and `delta_scan` only existed in earlier revisions of this branch. Since this reserves the name rather than a field number it does not buy wire compatibility either, and it reads as though a released field was removed. Could it just be dropped?
##########
contrib/delta-spark/src/test/scala/org/apache/comet/contrib/delta/JvmLowercaseParitySuite.scala:
##########
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.util.Locale
+
+import org.scalatest.funsuite.AnyFunSuite
+
+import org.apache.comet.serde.operator.JvmCaseTables
+
+/**
+ * Self-validating proof that `JvmCaseTables.mirrorLowercase` -- a line-for-line mirror of the
+ * native `JvmCaseTables::lowercase` in `native/core/src/parquet/schema_adapter.rs`, run over the
+ * tables `JvmCaseTables` generates from the RUNNING JVM -- reproduces this JVM's
+ * `String.toLowerCase(Locale.ROOT)`. Because both the tables and the expectations come from the
+ * same running JVM, this suite passes on ANY supported JDK by construction.
+ *
+ * Needs no `SparkSession`, so it extends `AnyFunSuite` directly rather than `CometDeltaTestBase`.
+ */
+class JvmLowercaseParitySuite extends AnyFunSuite {
Review Comment:
This suite proves that `JvmCaseTables.mirrorLowercase` equals `String.toLowerCase(Locale.ROOT)`, but `mirrorLowercase` is not the code that runs. The Rust `JvmCaseTables::lowercase` is, and its only coverage is the hand-written 27-range `jdk17_test_tables` in `schema_adapter.rs`. Two hand-written ports of an algorithm this intricate will drift eventually, and the drift would surface as silently wrong column matching rather than a failure.
I checked the Rust side directly and it passes cleanly, as described in my top-level comment, so this is not a bug report. It is that the check does not exist in the repo. Could it live here? Writing the JVM-generated tables plus the expected outputs to a test resource and reading them from a Rust test would verify the real implementation and would make the 539-line Scala mirror unnecessary.
Separately, since what this validates is core behavior rather than Delta behavior, could the suite move to `spark/src/test`? The `delta` filter in `compute-changes.py` does include `spark/src/main/**` and `native/**/src/**`, so it fires today, but anyone building or testing Comet without `-Pdelta` never runs it, and if the Delta contrib is ever renamed or dropped the verification goes with it.
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala:
##########
@@ -125,166 +150,233 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with CometTypeS
scan: CometScanExec,
builder: Operator.Builder,
childOp: OperatorOuterClass.Operator*): Option[OperatorOuterClass.Operator] = {
- val nativeScanBuilder = OperatorOuterClass.NativeScan.newBuilder()
+ // Extract object store options from first file (S3 configs apply to all files in scan).
+ // Use selectedPartitions (static) instead of getFilePartitions() because at planning time
+ // DPP subqueries haven't been resolved yet. Object store options don't depend on DPP.
+ val firstFileUri = scan.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ // Collect S3/cloud storage configurations
+ val hadoopConf = scan.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scan.relation.options)
+
+ buildNativeScanCommon(
+ source = scan.simpleStringWithNodeId(),
+ output = scan.output,
+ requiredSchema = scan.requiredSchema,
+ dataSchema = scan.relation.dataSchema,
+ partitionSchema = scan.relation.partitionSchema,
+ fileConstantMetadataColumns = scan.wrapped.fileConstantMetadataColumns,
+ dataFilters = scan.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scan.conf) match {
+ case Some(commonBuilder) =>
+ // Sink operators don't have children
+ builder.clearChildren()
+ val nativeScanBuilder = OperatorOuterClass.NativeScan.newBuilder()
+ // Set common data in NativeScan (file_partition will be populated at execution time)
+ nativeScanBuilder.setCommon(commonBuilder.build())
+ Some(builder.setNativeScan(nativeScanBuilder).build())
+ case None =>
+ // There are unsupported scan type
+ withFallbackReason(
+ scan,
+ s"unsupported Comet operator: ${scan.nodeName}, due to unsupported data types above")
+ None
+ }
+ }
+
+ /**
+ * Build the `NativeScanCommon` proto shared by the core parquet scan and contrib scans that
+ * delegate to the same native parquet machinery (e.g. a Delta scan contrib, which passes
+ * physical-name schemas under column mapping). Returns `None` when an output data type cannot
+ * be serialized; the caller is responsible for tagging a fallback reason.
+ *
+ * Visibility note: `private[comet]` means a contrib caller must live under an
+ * `org.apache.comet.*` package (the same constraint `PlanDataInjector` implementers have).
+ */
+ private[comet] def buildNativeScanCommon(
+ source: String,
+ output: Seq[Attribute],
+ requiredSchema: StructType,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ fileConstantMetadataColumns: Seq[AttributeReference],
+ dataFilters: Seq[Expression],
+ firstFileUri: Option[URI],
+ hadoopConf: Configuration,
+ conf: SQLConf): Option[OperatorOuterClass.NativeScanCommon.Builder] = {
val commonBuilder = OperatorOuterClass.NativeScanCommon.newBuilder()
// Set source in common (used as part of injection key)
- commonBuilder.setSource(scan.simpleStringWithNodeId())
+ commonBuilder.setSource(source)
- val scanTypes = scan.output.flatten { attr =>
+ val scanTypes = output.flatten { attr =>
serializeDataType(attr.dataType)
}
- if (scanTypes.length == scan.output.length) {
- commonBuilder.addAllFields(scanTypes.asJava)
-
- // Sink operators don't have children
- builder.clearChildren()
-
- if (scan.conf.getConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
- val dataFilters = new ListBuffer[Expr]()
- for (filter <- scan.supportedDataFilters) {
- exprToProto(filter, scan.output) match {
- case Some(proto) => dataFilters += proto
- case _ =>
- logWarning(s"Unsupported data filter $filter")
- }
+ if (scanTypes.length != output.length) {
+ // There are unsupported scan types
+ return None
+ }
+ commonBuilder.addAllFields(scanTypes.asJava)
+
+ if (conf.getConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ val filterProtos = new ListBuffer[Expr]()
+ for (filter <- dataFilters) {
+ exprToProto(filter, output) match {
+ case Some(proto) => filterProtos += proto
+ case _ =>
+ logWarning(s"Unsupported data filter $filter")
}
- commonBuilder.addAllDataFilters(dataFilters.asJava)
- }
-
- val possibleDefaultValues = getExistenceDefaultValues(scan.requiredSchema)
- if (possibleDefaultValues.exists(_ != null)) {
- // Our schema has default values. Serialize two lists, one with the default values
- // and another with the indexes in the schema so the native side can map missing
- // columns to these default values.
- val (defaultValues, indexes) = possibleDefaultValues.iterator.zipWithIndex
- .filter { case (expr, _) => expr != null }
- .map { case (expr, index) =>
- // ResolveDefaultColumnsUtil.getExistenceDefaultValues has evaluated these
- // expressions and they should now just be literals.
- (Literal(expr), index.toLong.asInstanceOf[java.lang.Long])
- }
- .toList
- .unzip
- commonBuilder.addAllDefaultValues(
- defaultValues.flatMap(exprToProto(_, scan.output)).asJava)
- commonBuilder.addAllDefaultValuesIndexes(indexes.asJava)
}
+ commonBuilder.addAllDataFilters(filterProtos.asJava)
+ }
- // Extract object store options from first file (S3 configs apply to all files in scan).
- // Use selectedPartitions (static) instead of getFilePartitions() because at planning time
- // DPP subqueries haven't been resolved yet. Object store options don't depend on DPP.
- val firstFileUri = scan.selectedPartitions
- .flatMap(_.files.headOption)
- .headOption
- .map(_.getPath.toUri)
-
- // Constant metadata columns (file_path, file_name, file_size, file_block_start,
- // file_block_length, file_modification_time) are known before opening the file and
- // constant for every row read from it, exactly like partition columns. Spark places
- // them immediately after partition columns in `scan.output`
- // (FileSourceStrategy.scala: readDataColumns ++ generatedMetadataColumns ++
- // partitionColumns ++ constantMetadataColumns), so appending them after the real
- // partition schema here keeps the two in lockstep.
- val constantMetadataFields = scan.wrapped.fileConstantMetadataColumns.map(attr =>
- StructField(s"$constantMetadataFieldPrefix${attr.name}", attr.dataType, attr.nullable))
- val partitionSchemaFields = scan.relation.partitionSchema.fields.toSeq ++
- constantMetadataFields
- val partitionSchema = schema2Proto(partitionSchemaFields)
- val requiredSchema = schema2Proto(scan.requiredSchema)
-
- // Spark's required schema can prune a Variant column, including a Variant nested under an
- // unrequested struct. The complete relation schema still contains that unsupported type,
- // and serializing it would throw even though the native reader never needs those bytes.
- // Keep ordinary fields unchanged and replace a requested Variant-bearing root with its
- // already-validated, pruned required field. A requested actual Variant never reaches this
- // point because CometScanRule keeps those scans on Spark.
- val nativeDataSchema = StructType(scan.relation.dataSchema.fields.flatMap { field =>
- if (containsVariantType(field.dataType)) {
- scan.requiredSchema.fields.find(requiredField =>
- scan.conf.resolver(requiredField.name, field.name))
- } else {
- Some(field)
- }
- })
- val dataSchema = schema2Proto(nativeDataSchema)
-
- val dataSchemaIndexes = scan.requiredSchema.map(field => {
- nativeDataSchema.fieldIndex(field.name)
- })
- val partitionSchemaIndexes = nativeDataSchema.fields.length until
- (nativeDataSchema.length + partitionSchemaFields.length)
-
- val projectionVector = (dataSchemaIndexes ++ partitionSchemaIndexes).map(idx =>
- idx.toLong.asInstanceOf[java.lang.Long])
-
- commonBuilder.addAllProjectionVector(projectionVector.asJava)
-
- // In `CometScanRule`, we ensure partitionSchema (including constant metadata columns)
- // is supported.
- assert(partitionSchema.length == partitionSchemaFields.length)
-
- commonBuilder.addAllDataSchema(dataSchema.asJava)
- commonBuilder.addAllRequiredSchema(requiredSchema.asJava)
- commonBuilder.addAllPartitionSchema(partitionSchema.asJava)
- commonBuilder.setSessionTimezone(scan.conf.getConfString("spark.sql.session.timeZone"))
- commonBuilder.setCaseSensitive(scan.conf.getConf[Boolean](SQLConf.CASE_SENSITIVE))
-
- // SPARK-53535 (Spark 4.1+): when reading a struct whose requested fields are all
- // missing in the Parquet file, the new default preserves the parent struct's
- // nullness from the file (so non-null parents materialize as a struct of all-null
- // fields). Pre-4.1 Spark hardcodes the legacy behavior (whole struct null), which
- // matches the Comet default we use as fallback.
- val returnNullStructConfKey =
- "spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing"
- val returnNullStructDefault = if (isSpark41Plus) "false" else "true"
- commonBuilder.setReturnNullStructIfAllFieldsMissing(
- scan.conf.getConfString(returnNullStructConfKey, returnNullStructDefault).toBoolean)
-
- // Field-ID matching: only ask the native side to do extra work when the conf is on AND
- // the requested schema actually carries IDs. Spark's ParquetReadSupport applies the same
- // gate before invoking matchIdField.
- val useFieldId =
- scan.conf.getConf(SQLConf.PARQUET_FIELD_ID_READ_ENABLED) &&
- ParquetUtils.hasFieldIds(scan.requiredSchema)
- commonBuilder.setUseFieldId(useFieldId)
- commonBuilder.setIgnoreMissingFieldId(
- scan.conf.getConf(SQLConf.IGNORE_MISSING_PARQUET_FIELD_ID))
-
- commonBuilder.setAllowTypePromotion(CometConf.COMET_SCHEMA_EVOLUTION_ENABLED)
- commonBuilder.setAllowTimestampLtzToNtz(CometConf.COMET_ALLOW_TIMESTAMP_LTZ_AS_NTZ)
-
- // Collect S3/cloud storage configurations
- val hadoopConf = scan.relation.sparkSession.sessionState
- .newHadoopConfWithOptions(scan.relation.options)
-
- commonBuilder.setEncryptionEnabled(CometParquetUtils.encryptionEnabled(hadoopConf))
-
- firstFileUri.foreach { uri =>
- val objectStoreOptions =
- NativeConfig.extractObjectStoreOptions(hadoopConf, uri)
- objectStoreOptions.foreach { case (key, value) =>
- commonBuilder.putObjectStoreOptions(key, value)
+ val possibleDefaultValues = getExistenceDefaultValues(requiredSchema)
+ if (possibleDefaultValues.exists(_ != null)) {
+ // Our schema has default values. Serialize two lists, one with the default values
+ // and another with the indexes in the schema so the native side can map missing
+ // columns to these default values.
+ val (defaultValues, indexes) = possibleDefaultValues.iterator.zipWithIndex
+ .filter { case (expr, _) => expr != null }
+ .map { case (expr, index) =>
+ // ResolveDefaultColumnsUtil.getExistenceDefaultValues has evaluated these
+ // expressions and they should now just be literals.
+ (Literal(expr), index.toLong.asInstanceOf[java.lang.Long])
}
+ .toList
+ .unzip
+ commonBuilder.addAllDefaultValues(defaultValues.flatMap(exprToProto(_, output)).asJava)
+ commonBuilder.addAllDefaultValuesIndexes(indexes.asJava)
+ }
+
+ // Constant metadata columns (file_path, file_name, file_size, file_block_start,
+ // file_block_length, file_modification_time) are known before opening the file and
+ // constant for every row read from it, exactly like partition columns. Spark places
+ // them immediately after partition columns in the scan output
+ // (FileSourceStrategy.scala: readDataColumns ++ generatedMetadataColumns ++
+ // partitionColumns ++ constantMetadataColumns), so appending them after the real
+ // partition schema here keeps the two in lockstep.
+ val constantMetadataFields = uniqueConstantMetadataFields(
+ fileConstantMetadataColumns,
+ dataSchema.fields.map(_.name).toSet ++ partitionSchema.fields.map(_.name).toSet)
+ val partitionSchemaFields = partitionSchema.fields.toSeq ++ constantMetadataFields
+ val partitionSchemaProto = schema2Proto(partitionSchemaFields)
+ val requiredSchemaProto = schema2Proto(requiredSchema)
+
+ // Spark's required schema can prune a Variant column, including a Variant nested under an
+ // unrequested struct. The complete relation schema still contains that unsupported type,
+ // and serializing it would throw even though the native reader never needs those bytes.
+ // Keep ordinary fields unchanged and replace a requested Variant-bearing root with its
+ // already-validated, pruned required field. A requested actual Variant never reaches this
+ // point because CometScanRule keeps those scans on Spark.
+ val nativeDataSchema = StructType(dataSchema.fields.flatMap { field =>
+ if (containsVariantType(field.dataType)) {
+ requiredSchema.fields.find(requiredField => conf.resolver(requiredField.name, field.name))
+ } else {
+ Some(field)
}
+ })
+ val dataSchemaProto = schema2Proto(nativeDataSchema)
- // Set common data in NativeScan (file_partition will be populated at execution time)
- nativeScanBuilder.setCommon(commonBuilder.build())
+ val dataSchemaIndexes = requiredSchema.map(field => {
+ nativeDataSchema.fieldIndex(field.name)
+ })
+ val partitionSchemaIndexes = nativeDataSchema.fields.length until
+ (nativeDataSchema.length + partitionSchemaFields.length)
- Some(builder.setNativeScan(nativeScanBuilder).build())
+ val projectionVector = (dataSchemaIndexes ++ partitionSchemaIndexes).map(idx =>
+ idx.toLong.asInstanceOf[java.lang.Long])
- } else {
- // There are unsupported scan type
- withFallbackReason(
- scan,
- s"unsupported Comet operator: ${scan.nodeName}, due to unsupported data types above")
- None
+ commonBuilder.addAllProjectionVector(projectionVector.asJava)
+
+ // In `CometScanRule`, we ensure partitionSchema (including constant metadata columns)
+ // is supported.
+ assert(partitionSchemaProto.length == partitionSchemaFields.length)
+
+ commonBuilder.addAllDataSchema(dataSchemaProto.asJava)
+ commonBuilder.addAllRequiredSchema(requiredSchemaProto.asJava)
+ commonBuilder.addAllPartitionSchema(partitionSchemaProto.asJava)
+
+ populateScanConfFlags(commonBuilder, requiredSchema, firstFileUri, hadoopConf, conf)
+
+ Some(commonBuilder)
+ }
+
+ /**
+ * Populate the configuration-derived flags of a `NativeScanCommon`: session timezone, case
+ * sensitivity, struct-nullness legacy flag, field-ID matching, type promotion, encryption, and
+ * object-store options. Shared with contrib scans that assemble their own schemas/projection
+ * (e.g. the Delta contrib's deletion-vector shape) so new flags added here reach them without
+ * drift.
+ */
+ private[comet] def populateScanConfFlags(
+ commonBuilder: OperatorOuterClass.NativeScanCommon.Builder,
+ requiredSchema: StructType,
+ firstFileUri: Option[URI],
+ hadoopConf: Configuration,
+ conf: SQLConf): Unit = {
+ commonBuilder.setSessionTimezone(conf.getConfString("spark.sql.session.timeZone"))
+ val caseSensitive = conf.getConf[Boolean](SQLConf.CASE_SENSITIVE)
+ commonBuilder.setCaseSensitive(caseSensitive)
+ if (!caseSensitive) {
+ // Ship THIS JVM's case data so native's case-insensitive footer matching reproduces
+ // this JVM's `toLowerCase(Locale.ROOT)` exactly, whatever Unicode version it bundles.
+ JvmCaseTables.populate(commonBuilder)
Review Comment:
Attaching the tables here means every `NativeScanCommon` carries its own copy with no dedup across scans in a plan. I measured 18,801 bytes per scan.
Running `select k1, d1 from t1` repeated under `UNION ALL` and summing the serialized native blocks:
| scans | `caseSensitive=true` | `caseSensitive=false` (the default) |
|---|---|---|
| 1 | 165 B | 18,966 B |
| 2 | 330 B | 37,932 B |
| 3 | 495 B | 56,898 B |
That payload rides in the Spark task binary for every stage, so a 20-table query would ship around 380 KB of identical tables.
The change to `QueryContextInternerSuite` in this PR is what led me here. That suite asserts `before > after * 4` to protect the SQL-text interning optimization, and pinning it to `spark.sql.caseSensitive=true` is what keeps it passing. On the default config it fails: I get `before=33316 after=21767 ratio=1.53` against `before=14513 after=2964 ratio=4.90` with case sensitivity on. In other words the plan is now larger than it was before interning landed, and the suite no longer covers the default configuration.
The tables are a pure function of the JDK and identical for every scan in a plan. Could they be interned at the root `Operator` the same way `sql_text_pool` already is, so a plan pays once rather than once per scan? Better still, sending them to each executor once at plugin init keyed by the fingerprint would take the per-plan cost to zero and let `QueryContextInternerSuite` go back to testing the default config.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1707 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Always zero-I/O (plain propagated-conf read, no keystore): native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ /**
+ * `CommonConfigurationKeysPublic.HADOOP_SECURITY_CREDENTIAL_CLEAR_TEXT_FALLBACK`, default
+ * `true`, verified via `javap` against `hadoop-common` 3.3.4's
+ * `Configuration#getPasswordFromConfig`: `getPassword` only falls back to reading a plaintext
+ * conf value once `getBoolean(<this key>, true)` holds -- with the flag off, a plaintext value
+ * is invisible to every `getPassword`-based resolver, even when no credential provider is
+ * configured at all.
+ */
+ private val ClearTextFallbackKey = "hadoop.security.credential.clear-text-fallback"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * How Hadoop's OWN consumer reads one of the keys compared by [[s3ConfigDivergenceReason]],
+ * which decides how [[s3KeyDivergenceReason]] computes the Hadoop-effective side of its
+ * equality check. Exactly two consumer families exist among [[AllS3ConfigKeys]] in `hadoop-aws`
+ * 3.3.4, each verified via `javap`/CFR against the real call sites (cited per key on
+ * [[S3ConfigKeyConsumers]]). The tier must mirror the key's ACTUAL consumer: resolving a
+ * [[PropagatedOptionConsumer]] key through the wider `lookupPassword` cascade is NOT fail-safe
+ * for a value-EQUALITY comparator -- a long-form alias value Hadoop itself never reads can
+ * EQUAL native's resolution while Hadoop's true propagate-then-plain-get value differs, turning
+ * a real divergence into a wrongly-claimed scan (the endpoint `${...}`-redirect shape pinned in
+ * `DeltaScanContribSuite`).
+ */
+ private[delta] sealed trait S3ConfigConsumer
+
+ /**
+ * Read via `S3AUtils#lookupPassword(bucket, conf, baseKey)`, verified via `javap` against
+ * `hadoop-aws` 3.3.4: builds `longBucketKey = "fs.s3a.bucket." + bucket + "." + baseKey` (the
+ * FULL, already-`fs.s3a`-prefixed base key appended after the bucket segment) and reads it via
+ * `Configuration#getPassword` BEFORE the short-bucket key, keeping the long value whenever
+ * `getPassword` returns non-empty and only falling through to short-then-global otherwise.
+ * `getPassword` is Hadoop-credential-provider-aware and skips plaintext conf entirely when
+ * [[ClearTextFallbackKey]] is false. Modeled by [[hadoopLookupPasswordEffective]].
+ */
+ private[delta] case object LookupPasswordConsumer extends S3ConfigConsumer
+
+ /**
+ * Read via `S3AUtils#propagateBucketOptions` followed by a plain `Configuration#get`-family
+ * call (`getTrimmed`/`getBoolean`/`getClasses`) against the propagated view: the short bucket
+ * form wins only by having overwritten the global key during propagation, the long bucket form
+ * folds into an unread `fs.s3a.fs.s3a.*` key, and neither a credential provider nor
+ * [[ClearTextFallbackKey]] is ever consulted. Modeled as a plain `Configuration#get` on the
+ * [[propagateBucketOptions]] result, which also expands `${...}` references under that
+ * propagated view exactly like the real consumer.
+ */
+ private[delta] case object PropagatedOptionConsumer extends S3ConfigConsumer
+
+ /**
+ * Every `fs.s3a.*` base key that governs whether a claimed native scan actually behaves like
+ * Hadoop's own reader would, paired with the consumer family Hadoop resolves it through -- ONE
+ * list, with each key's resolution tier declared beside it, so a key can never sit in the
+ * comparator without a deliberate classification (adding one without picking a tier does not
+ * compile). The entries are every per-bucket `fs.s3a.*` base key native's S3 client's
+ * `get_config` (s3.rs) resolves, verified directly against its call sites:
+ * `extract_s3_config_options` (endpoint.region, path.style.access, endpoint,
+ * requester.pays.enabled), `lookup_provider_class` (the Comet-specific
+ * credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name).
+ *
+ * Tier assignments, each verified via `javap`/CFR against `hadoop-aws` 3.3.4:
+ * - access.key/secret.key/session.token: `S3AUtils#getAWSAccessKeys` and
+ * `MarshalledCredentialBinding#fromFileSystem` (reached from
+ * `TemporaryAWSCredentialsProvider`) resolve all three via `S3AUtils#lookupPassword` --
+ * [[LookupPasswordConsumer]].
+ * - aws.credentials.provider and assumed.role.credentials.provider:
+ * `S3AUtils#buildAWSProviderList` -> `loadAWSProviderClasses` -> plain
+ * `Configuration#getClasses` -- [[PropagatedOptionConsumer]].
+ * - assumed.role.arn/session.name: `AssumedRoleCredentialProvider`'s constructor reads both
+ * via plain `Configuration#getTrimmed` -- [[PropagatedOptionConsumer]].
+ * - endpoint (`S3AFileSystem`: `getTrimmed`), endpoint.region (`DefaultS3ClientFactory`:
+ * `getTrimmed`), path.style.access (`S3AFileSystem`: `getBoolean`) --
+ * [[PropagatedOptionConsumer]].
+ * - requester.pays.enabled: not read anywhere in `hadoop-aws` 3.3.4 (the constant does not
+ * even exist in its `Constants` class); later releases read it via plain `getBoolean`
+ * against the propagated conf, so the plain tier is both the faithful forward model and
+ * inert on 3.3.4 -- [[PropagatedOptionConsumer]].
+ * - comet.credential.provider.class: Comet's own activation key, plain conf read on both
+ * sides, never a Hadoop key at all -- [[PropagatedOptionConsumer]].
+ *
+ * SYNC NOTE: the key list must stay a superset of native's `NATIVE_S3A_CONFIG_PROPERTIES`
+ * constant (`native/core/src/parquet/objectstore/s3.rs`, property suffixes without the
+ * `fs.s3a.` prefix) -- `DeltaScanContribSuite`'s discovery-harness test asserts this
+ * mechanically against [[AllS3ConfigKeys]]. Literal strings, not the
+ * [[AwsCredentialsProviderKey]] / [[AssumedRoleCredentialsProviderKey]] vals declared below,
+ * purely to avoid a forward reference inside this `object` body; kept textually identical to
+ * those two constants.
+ */
+ private[delta] val S3ConfigKeyConsumers: Seq[(String, S3ConfigConsumer)] = Seq(
+ "fs.s3a.access.key" -> LookupPasswordConsumer,
+ "fs.s3a.secret.key" -> LookupPasswordConsumer,
+ "fs.s3a.session.token" -> LookupPasswordConsumer,
+ "fs.s3a.aws.credentials.provider" -> PropagatedOptionConsumer,
+ "fs.s3a.assumed.role.arn" -> PropagatedOptionConsumer,
+ "fs.s3a.assumed.role.session.name" -> PropagatedOptionConsumer,
+ "fs.s3a.assumed.role.credentials.provider" -> PropagatedOptionConsumer,
+ "fs.s3a.endpoint" -> PropagatedOptionConsumer,
+ "fs.s3a.endpoint.region" -> PropagatedOptionConsumer,
+ "fs.s3a.path.style.access" -> PropagatedOptionConsumer,
+ "fs.s3a.requester.pays.enabled" -> PropagatedOptionConsumer,
+ "fs.s3a.comet.credential.provider.class" -> PropagatedOptionConsumer)
+
+ /** The compared keys alone, in [[S3ConfigKeyConsumers]] order (discovery-harness surface). */
+ private[delta] val AllS3ConfigKeys: Seq[String] = S3ConfigKeyConsumers.map(_._1)
+
+ /**
+ * The short-bucket-then-global value resolved for `baseKey` under `bucket` from `hadoopConf`,
+ * skipping an empty value at either alias exactly like [[plainValue]]. NOT used by
+ * [[s3ConfigDivergenceReason]]/[[s3KeyDivergenceReason]] any more -- every key checked there
+ * resolves per its declared [[S3ConfigKeyConsumers]] tier (see [[s3KeyDivergenceReason]]),
+ * neither of which matches this function's read. This function's one remaining caller is
+ * [[shortThenGlobalOrReason]], which reads provider-CLASS strings (from the ORIGINAL,
+ * unpropagated conf) for name-support validation in [[providerClassReason]]/
+ * [[assumedRoleProviderClassReason]] -- by the time those run, [[s3ConfigDivergenceReason]] has
+ * already proven Hadoop's and native's effective values agree for the same key, so whichever of
+ * the two (equal) values this narrower read returns does not affect correctness there. NEVER
+ * used to compute native's own effective value -- see [[nativeShortThenGlobal]] for that.
+ */
+ private def shortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket` from the ORIGINAL, unpropagated `hadoopConf` -- `NativeConfig
+ * .extractObjectStoreOptions` forwards `Configuration#get`'s substituted value for every
+ * `fs.s3a.*` entry with no bucket-option propagation step of its own, so the original conf is
+ * the right input here. Unlike [[shortThenGlobal]]/[[plainValue]], this mirrors `get_config`
+ * faithfully: PRESENCE of the short-bucket key alone -- never its emptiness -- decides whether
+ * native falls back to the global key (`get_config` is a plain `HashMap::get`, which returns
+ * `Some` for a key explicitly set to `""`), so an explicitly empty or whitespace-only
+ * short-bucket value resolves to `Some("")` here and never falls through to global -- the
+ * OPPOSITE of Hadoop's own `getPassword`/`lookupPassword` semantics (see
+ * [[hadoopLookupPasswordEffective]]), which treat empty as absent and keep trying the next
+ * alias. The ONLY function used to compute native's effective value in
+ * [[s3KeyDivergenceReason]].
+ *
+ * Deliberately does NOT apply `get_config_trimmed`'s `.trim()` here: [[s3KeyDivergenceReason]]
+ * trims both this value and Hadoop's effective value together, symmetrically, at the point they
+ * are compared, rather than one-sidedly here -- trimming only the native side would flag a
+ * spurious divergence for a value neither side's whitespace actually changes the behavior of
+ * once each side's own downstream parsing normalizes it (e.g. Hadoop's own multi-line
+ * `fs.s3a.aws.credentials.provider` default, which both Hadoop and native additionally trim per
+ * comma-separated entry after splitting), while a one-sided trim would make an
+ * otherwise-identical default value look diverged for every bucket, never claiming natively at
+ * all.
+ */
+ private def nativeShortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ Option(hadoopConf.get(shortKey)).orElse(Option(hadoopConf.get(baseKey)))
+ }
+
+ /**
+ * Faithful in-memory replica of `S3AUtils#propagateBucketOptions` (`hadoop-aws`), which
+ * `S3AFileSystem#initialize` calls FIRST, before any option or credential is read:
+ * `Configuration conf = propagateBucketOptions(originalConf, bucket); ...; setConf(conf);` --
+ * every subsequent `conf.get`/`getPassword` call in that filesystem instance, including
+ * `${...}` variable substitution, resolves against this propagated view, not the original conf.
+ * `hadoop-aws` is not on this module's runtime classpath (see the string-literal-keys note
+ * above), so `S3AUtils#propagateBucketOptions` cannot be called directly; this reproduces its
+ * logic verbatim using only `hadoop-common`'s `Configuration`:
+ * {{{
+ * public static Configuration propagateBucketOptions(Configuration source, String bucket) {
+ * final String bucketPrefix = FS_S3A_BUCKET_PREFIX + bucket + '.';
+ * final Configuration dest = new Configuration(source);
+ * for (Map.Entry<String, String> entry : source) {
+ * final String key = entry.getKey();
+ * final String value = entry.getValue(); // the (unexpanded) value
+ * if (!key.startsWith(bucketPrefix) || bucketPrefix.equals(key)) continue;
+ * final String stripped = key.substring(bucketPrefix.length());
+ * if (stripped.startsWith("bucket.") || "impl".equals(stripped)) {
+ * // ignored
+ * } else {
+ * final String generic = FS_S3A_PREFIX + stripped;
+ * dest.set(generic, value, ...); // overwrites any existing global value
+ * }
+ * }
+ * return dest;
+ * }
+ * }}}
+ * Note the LONG bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) folds to an unread
+ * `fs.s3a.fs.s3a.<key>` key here too, exactly like the real method -- `stripped` already starts
+ * with `fs.s3a.` in that case, so prepending `fs.s3a.` again produces a key nothing ever reads.
+ */
+ private def propagateBucketOptions(hadoopConf: Configuration, bucket: String): Configuration = {
+ val bucketPrefix = s"fs.s3a.bucket.$bucket."
+ val dest = new Configuration(hadoopConf)
+ hadoopConf.iterator().asScala.foreach { entry =>
+ val key = entry.getKey
+ if (key.startsWith(bucketPrefix) && key != bucketPrefix) {
+ val stripped = key.substring(bucketPrefix.length)
+ if (!stripped.startsWith("bucket.") && stripped != "impl") {
+ dest.set(s"fs.s3a.$stripped", entry.getValue)
+ }
+ }
+ }
+ dest
+ }
+
+ /**
+ * Canonical and deprecated Hadoop S3A encryption-algorithm config keys, verified via `javap`
+ * against `hadoop-aws` 3.3.4's `org.apache.hadoop.fs.s3a.Constants`: `S3_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.encryption.algorithm"` (canonical) and `SERVER_SIDE_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.server-side-encryption-algorithm"` (DEPRECATED -- note the hyphen before "algorithm",
+ * unlike the corresponding `*.key` constants below, which both use a `.key` suffix).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, same
+ * rationale as [[HadoopCredentialProviderPathKey]].
+ */
+ private val S3EncryptionAlgorithmKey = "fs.s3a.encryption.algorithm"
+ private val DeprecatedS3EncryptionAlgorithmKey = "fs.s3a.server-side-encryption-algorithm"
+
+ /**
+ * The exact strings `S3AEncryptionMethods#getMethod` accepts, verified via `javap`/CFR against
+ * `hadoop-aws` 3.3.4's `S3AEncryptionMethods` enum: `NONE("")`, `SSE_S3("AES256", serverSide =
+ * true, requiresSecret = false)`, `SSE_KMS("SSE-KMS", serverSide = true, requiresSecret =
+ * false)`, `SSE_C("SSE-C", serverSide = true, requiresSecret = true)`, `CSE_KMS("CSE-KMS",
+ * serverSide = false, requiresSecret = true)`, `CSE_CUSTOM("CSE-CUSTOM", serverSide = false,
+ * requiresSecret = true)`. `getMethod` parses case-insensitively
+ * (`values().find(_.getMethod.equalsIgnoreCase(algorithm))`), matched below the same way.
+ *
+ * ALLOWLIST, not a blocklist (replaces the former SSE-C-only blocklist): only the algorithms S3
+ * decrypts transparently on GET/HEAD given read permission alone, with NO extra request header
+ * and NO client-side step, are safe for a native scan that forwards none of Hadoop's
+ * `fs.s3a.encryption.*`/`fs.s3a.server-side-encryption*` options --
+ * - `AES256` (SSE_S3, `serverSide = true`): plain server-side encryption, transparent on GET.
+ * - `SSE-KMS` (SSE_KMS, `serverSide = true`): server-side, KMS-managed key, transparent on
+ * GET given KMS decrypt permission (no header).
+ * - `DSSE-KMS`: NOT present in this enum on `hadoop-aws` 3.3.4 (confirmed by the six values
+ * listed above) -- `S3AEncryptionMethods.getMethod("DSSE-KMS")` throws
+ * `IOException("Unknown encryption algorithm DSSE-KMS")` on this version, so
+ * `S3AUtils#buildEncryptionSecrets` (and therefore Hadoop's own reader) already fails
+ * before ever reading such a table under 3.3.4, meaning this string can never actually be
+ * the resolved value on the declared target version -- admitting it here is inert there.
+ * Included anyway, forward-compatible, for a newer `hadoop-aws` on the runtime classpath (a
+ * later Hadoop release; this module has no compile-time `hadoop-aws` dependency, see the
+ * string-literal-keys note above) where DSSE-KMS is a real, dual-layer, server-side
+ * algorithm decrypted transparently on GET the same way SSE-KMS is. Every other value
+ * declines: `SSE-C` (SSE_C is `serverSide = true` in Hadoop's own enum, but `requiresSecret
+ * \= true` -- S3 rejects a GET/HEAD for an SSE-C object outright (400 Bad Request) unless
+ * the customer key is resent as a request header on every call, so a native scan that never
+ * learns the key cannot succeed at all, where Hadoop's own reader -- whose request factory
+ * attaches the key -- would), `CSE-KMS`/`CSE-CUSTOM` (`serverSide = false`: client-side
+ * encryption decrypts object bytes locally in the SDK layer, which the native Parquet
+ * reader has no equivalent of -- it would read raw ciphertext), and any future/unknown
+ * value (a value `S3AEncryptionMethods.getMethod` itself would reject is certainly not one
+ * of the three confirmed-transparent algorithms above; declining is the only safe default
+ * for anything this gate cannot positively confirm).
+ */
+ private val AllowedEncryptionAlgorithms: Set[String] = Set("AES256", "SSE-KMS", "DSSE-KMS")
+
+ /**
+ * `bucket`'s effective encryption-algorithm key and value under `hadoopConf`, or `None` when
+ * neither the canonical nor deprecated key is set anywhere consulted. Mirrors
+ * `S3AUtils#buildEncryptionSecrets`'s real resolution order, verified via `javap`/CFR
+ * decompilation of `hadoop-aws` 3.3.4's `S3AUtils.class`:
+ * {{{
+ * String algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.server-side-encryption-algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.server-side-encryption-algorithm");
+ * }}}
+ * i.e. bucket-tier (canonical, then deprecated), THEN global-tier (canonical, then deprecated)
+ * -- the two tiers are never interleaved key-by-key, so this must stay two explicit bucket-tier
+ * lookups followed by two explicit global-tier lookups, not a single
+ * [[hadoopLookupPasswordEffective]] call per key (which would let an unset canonical bucket key
+ * fall through straight to the canonical GLOBAL value ahead of a SET deprecated bucket key, the
+ * wrong answer).
+ *
+ * THE FIX for the SSE-C long-bucket-alias gap is entirely inside the bucket tier:
+ * `lookupBucketSecret` itself is long-then-short, decompiled from `hadoop-aws` 3.3.4's
+ * `S3AUtils.class`:
+ * {{{
+ * // longBucketKey = fs.s3a.bucket.B.fs.s3a.<key>
+ * String longBucketKey = String.format(BUCKET_PATTERN, bucket, baseKey);
+ * String initialVal = getPassword(conf, longBucketKey, null, null);
+ * // shortBucketKey = fs.s3a.bucket.B.<key>
+ * String shortBucketKey = String.format(BUCKET_PATTERN, bucket, subkey);
+ * // keeps initialVal (the LONG value) if non-empty
+ * return getPassword(conf, shortBucketKey, initialVal, null);
+ * }}}
+ * i.e. the SAME long-bucket-key construction and long-wins-if-nonempty semantics as
+ * `S3AUtils#lookupPassword` (see [[LookupPasswordConsumer]]/[[hadoopLookupPasswordEffective]])
+ * -- the encryption algorithm is NOT one of the keys that flows through
+ * `S3AUtils#propagateBucketOptions` (which folds an unrelated per-bucket LONG form into an
+ * unread key). An earlier version of this function modeled the bucket tier as SHORT-only,
+ * documented as "the LONG bucket form is genuinely never consulted for this key" -- that
+ * documentation was wrong (this decompilation supersedes it): a bucket configured only via
+ * `fs.s3a.bucket.B.fs.s3a.encryption.algorithm=SSE-C` bypassed the SSE-C gate entirely, because
+ * Hadoop's own reader DOES read that long form (and picks SSE-C), while this function reported
+ * `None` (nothing set) and the allowlist check below never even ran.
+ *
+ * The canonical-vs-deprecated distinction below is frequently moot in practice: `hadoop-aws`'s
+ * `S3AFileSystem.addDeprecatedKeys()` statically registers `fs.s3a.server-side-encryption-*` as
+ * `Configuration`-level deprecated aliases of `fs.s3a.encryption.*` (verified via `javap`), a
+ * registration that lives in a static field on Hadoop's `Configuration` class -- process-wide
+ * once `S3AFileSystem`'s class has loaded anywhere in the JVM, which a real scan has always
+ * already done by the time this gate runs, since reading the S3 table at all requires loading
+ * that class. Once active, `Configuration#get` resolves either literal key to the identical
+ * value transparently, making the two-key cascade below redundant (but harmless) for that case;
+ * it remains the operative path only when nothing else in the process has loaded
+ * `S3AFileSystem` yet.
+ *
+ * ALSO walks the Hadoop-credential-provider (JCEKS) path via [[resolveViaCredentialAliases]]
+ * for each of the four lookups below, matching `lookupBucketSecret`/`lookupPassword`'s real
+ * per-alias `getPassword` calls (quoted above) exactly: both are `getPassword`, not plain
+ * `Configuration#get`, so a bucket storing the algorithm name ONLY in a JCEKS keystore is
+ * exactly as real a Hadoop deployment shape for this key as it is for the credential keys
+ * [[hadoopLookupPasswordEffective]] already covers -- there is nothing algorithm-specific that
+ * makes JCEKS storage implausible here, so an earlier version of this function skipping it
+ * (documented at the time as "the algorithm NAME is not credential-sensitive data, so storing
+ * it in a keystore is not a realistic Hadoop deployment pattern") was an unjustified, narrower
+ * read than Hadoop's own resolver actually performs, under-declining a bucket whose algorithm
+ * is keystore-only. [[resolveViaCredentialAliases]]'s Arm B/C split still means this is zero
+ * extra I/O for the common case: keystore I/O only happens when a Hadoop credential-provider
+ * path is actually configured for the bucket, contained in that function's own try/catch.
+ * `bucketTier`/`globalTier` return `Left` (propagated straight through by [[orElseTier]]) when
+ * [[resolveViaCredentialAliases]] cannot safely verify a tier at all (an S3A-scoped provider
+ * path, or a corrupt/unreadable global keystore) -- correctly short-circuiting the whole
+ * cascade with a decline rather than silently falling through to a later tier that might look
+ * unset only because the true value was unverifiable.
+ */
+ private def effectiveEncryptionAlgorithm(
+ hadoopConf: Configuration,
+ bucket: String): Either[String, Option[(String, String)]] = {
+ def bucketTier(baseKey: String): Either[String, Option[(String, String)]] = {
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(longKey, shortKey))
+ .map(_.map(baseKey -> _))
+ }
+ def globalTier(baseKey: String): Either[String, Option[(String, String)]] =
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(baseKey))
+ .map(_.map(baseKey -> _))
+
+ // Short-circuits on Left (unverifiable tier) or Right(Some(_)) (resolved); only Right(None)
+ // (tier definitively unset) falls through to `next`, mirroring buildEncryptionSecrets's
+ // sequential `if (algorithm == null) algorithm = ...` cascade exactly.
+ def orElseTier(
+ current: Either[String, Option[(String, String)]],
+ next: => Either[String, Option[(String, String)]])
+ : Either[String, Option[(String, String)]] =
+ current match {
+ case Left(reason) => Left(reason)
+ case Right(Some(value)) => Right(Some(value))
+ case Right(None) => next
+ }
+
+ orElseTier(
+ bucketTier(S3EncryptionAlgorithmKey),
+ orElseTier(
+ bucketTier(DeprecatedS3EncryptionAlgorithmKey),
+ orElseTier(
+ globalTier(S3EncryptionAlgorithmKey),
+ globalTier(DeprecatedS3EncryptionAlgorithmKey))))
+ }
+
+ private def unsupportedEncryptionAlgorithmDeclineReason(
+ bucket: String,
+ algorithmKey: String,
+ algorithm: String): String =
+ s"Native Delta scan does not support $algorithmKey=$algorithm for $bucket " +
+ "(the native S3 client only supports unencrypted objects and S3's transparent " +
+ "server-side algorithms -- AES256/SSE-S3, SSE-KMS, and DSSE-KMS decrypt on GET/HEAD given " +
+ "read permission alone, with no extra request header; SSE-C additionally requires the " +
+ "customer-provided key resent as a header on every GET/HEAD request, which the native S3 " +
+ "client's extract_s3_config_options never forwards, and CSE-KMS/CSE-CUSTOM decrypt object " +
+ "bytes client-side, a layer the native Parquet reader does not have -- any of these would " +
+ "fail outright or silently read ciphertext where Hadoop's own reader succeeds)"
+
+ /**
+ * First reason any bucket among `uris` is configured for an encryption algorithm the native S3
+ * client cannot safely read, or `None` when claimable. Allowlist-based (see
+ * [[AllowedEncryptionAlgorithms]]): only `AES256`/`SSE-KMS`/`DSSE-KMS` (and unset/empty) pass;
+ * every other resolved value -- `SSE-C`, `CSE-KMS`, `CSE-CUSTOM`, or any unrecognized future
+ * algorithm string -- declines. Deliberately NOT a blocklist keyed on `SSE-C` alone: an
+ * allowlist is safe by construction against a Hadoop release adding a new encryption method
+ * this gate has never heard of, where a blocklist would silently admit it. Never interpolates a
+ * resolved key value, only key names, the bucket, and the (non-secret) algorithm name. Declines
+ * on a `Left` from [[effectiveEncryptionAlgorithm]] too (an unverifiable credential-provider
+ * arm, e.g. an S3A-scoped provider path or a corrupt/unreadable global keystore) -- the
+ * algorithm cannot be ruled safe when it cannot be read at all.
+ */
+ private[delta] def unsupportedEncryptionAlgorithmReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ effectiveEncryptionAlgorithm(hadoopConf, bucket) match {
+ case Left(reason) => Some(reason)
+ case Right(None) => None
+ case Right(Some((key, value))) =>
+ if (!AllowedEncryptionAlgorithms.exists(_.equalsIgnoreCase(value))) {
+ Some(unsupportedEncryptionAlgorithmDeclineReason(bucket, key, value))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Canonical Hadoop S3A HTTP-proxy host config key, verified via CFR decompilation of
+ * `hadoop-aws` 3.3.4's `S3AUtils.class` (`initProxySupport`):
+ * {{{
+ * String proxyHost = conf.getTrimmed("fs.s3a.proxy.host", "");
+ * int proxyPort = conf.getInt("fs.s3a.proxy.port", -1);
+ * if (!proxyHost.isEmpty()) {
+ * ...
+ * String proxyUsername =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.username", null, null);
+ * String proxyPassword =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.password", null, null);
+ * ...
+ * }
+ * }}}
+ * `fs.s3a.proxy.host`/`fs.s3a.proxy.port` resolve via a PLAIN, non-bucket-scoped, non-JCEKS
+ * `Configuration#getTrimmed`/`getInt` call -- NOT `lookupPassword` -- against whatever conf
+ * `S3AFileSystem#initialize` already ran through `propagateBucketOptions` before
+ * `createAwsConf`/`initProxySupport` ever runs; only the SIBLING `fs.s3a.proxy.username`/
+ * `fs.s3a.proxy.password` keys go through `lookupPassword` (bucket long/short/global,
+ * JCEKS-aware). So the host is bucket-aware only via `propagateBucketOptions`'s short-bucket-
+ * form folding, never the long-bucket form, and never a credential-provider read -- the SAME
+ * shape as `endpoint`/`path.style.access` ([[PropagatedOptionConsumer]], see
+ * [[S3ConfigKeyConsumers]]'s doc), not the credential family. `hadoop-aws` 3.4.x (the Spark 4.x
+ * profiles' version) moves this code to `AWSClientConfig#createProxyConfiguration`/
+ * `#createAsyncProxyConfiguration` but keeps the exact same reads, verified via `javap` against
+ * 3.4.2: `conf.getTrimmed("fs.s3a.proxy.host", "")` for the host, `S3AUtils.lookupPassword` for
+ * username/password only.
+ *
+ * [[proxyGateReason]] therefore resolves the host EXACTLY like its real consumer -- plain
+ * `Configuration#getTrimmed` on the [[propagateBucketOptions]] result, no provider arms, no
+ * [[ClearTextFallbackKey]] handling -- rather than through the wider `lookupPassword` cascade
+ * an earlier version reused here. The wider cascade was wrong in BOTH directions for this key:
+ * with only the GLOBAL Hadoop provider path set and [[ClearTextFallbackKey]] false,
+ * `getPassword` hides a plaintext host that `getTrimmed` serves to Hadoop anyway (a missed
+ * decline, the exact bypass this gate exists to close -- native has NO HTTP-proxy support of
+ * any kind, no `fs.s3a.proxy.*` key is read anywhere in `s3.rs`); and an S3A-scoped provider
+ * path or a lone long-form bucket alias declined a bucket whose real consumer can never see a
+ * host from either source (pure over-refusal -- no keystore can supply the host to a plain
+ * `getTrimmed`, and the long form folds into the unread `fs.s3a.fs.s3a.proxy.host`).
+ */
+ private val S3ProxyHostKey = "fs.s3a.proxy.host"
+
+ private def unsupportedProxyReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key configured for $bucket (the native S3 client has " +
+ "no HTTP proxy support at all -- no fs.s3a.proxy.* key is read anywhere in its object " +
+ "store layer -- so a claimed scan would connect to S3 directly instead of routing through " +
+ "the configured proxy, either bypassing an egress/network-segmentation policy or simply " +
+ "failing to reach the endpoint)"
+
+ /**
+ * First reason any bucket among `uris` has an HTTP proxy configured via [[S3ProxyHostKey]], or
+ * `None` when claimable. Reads the host EXACTLY like its real consumer (see
+ * [[S3ProxyHostKey]]'s doc): plain `Configuration#getTrimmed` on the [[propagateBucketOptions]]
+ * result, so this gate is always zero-I/O -- no credential provider is ever consulted for the
+ * host, because none ever supplies it to Hadoop either. Ordered alongside
+ * [[unsupportedEncryptionAlgorithmReason]] among the conf-only gates, ahead of
+ * [[s3ConfigDivergenceReason]]. The try/catch guards `Configuration#get`'s
+ * `IllegalStateException` on a `${...}` substitution cycle, same as [[s3KeyDivergenceReason]].
+ * Never interpolates a resolved value: the proxy HOST is not secret, but naming it here would
+ * be a strange place to first surface it, and proxy CREDENTIALS
+ * (`fs.s3a.proxy.username`/`fs.s3a.proxy.password`, not read by this gate at all -- the whole
+ * point of gating on the host is that a non-empty host declines before any proxy credential
+ * would ever need to be forwarded) must never appear in a decline reason regardless.
+ */
+ private[delta] def proxyGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI],
+ propagatedConfCache: MutableMap[String, Configuration] = MutableMap.empty)
+ : Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ try {
+ val propagatedConf =
+ propagatedConfCache.getOrElseUpdate(
+ bucket,
+ propagateBucketOptions(hadoopConf, bucket))
+ if (propagatedConf.getTrimmed(S3ProxyHostKey, "").nonEmpty) {
+ Some(unsupportedProxyReason(bucket, S3ProxyHostKey))
+ } else {
+ None
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableValueReason(bucket, S3ProxyHostKey, e))
+ }
+ }
+ }
+ }
+
+ /**
+ * `baseKey`'s long, short, then global per-bucket aliases, in Hadoop's own resolution order.
+ */
+ private def longThenShortThenGlobalAliases(bucket: String, baseKey: String): Seq[String] = {
+ val suffix = baseKey.stripPrefix("fs.s3a.")
+ Seq(s"fs.s3a.bucket.$bucket.fs.s3a.$suffix", s"fs.s3a.bucket.$bucket.$suffix", baseKey)
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * The three-way Hadoop credential-provider-path precheck shared by every `getPassword`-based
+ * resolution below, factored out of what used to be [[hadoopLookupPasswordEffective]]'s own
Review Comment:
A few of the doc comments in this file describe how the code got here rather than what it does. This one refers to `hadoopLookupPasswordEffective`, and there are similar ones at 1139 ("Generalizes what used to be"), 1415 ("previously relied on `s3ConfigDivergenceReason` running first"), and 1489 ("the variable-reference precheck this used to run ahead of the class-support check is gone"). `DeltaScanContribSuite.scala:37` has one too, referring to "core's now-deleted extension-SPI".
Some of those symbols are not in the tree, so a reader who greps for them comes up empty, and the commits are squashed so the history is not recoverable there either. Could these be trimmed down to what the code does now, with any load-bearing context moved to the PR description?
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -76,6 +77,583 @@ fn schema_has_field_ids(schema: &SchemaRef) -> bool {
schema.fields().iter().any(|f| parse_field_id(f).is_some())
}
+// ---------------------------------------------------------------------------------------------
+// JVM-shipped case tables: reproduce the PLANNING JVM's `String.toLowerCase(Locale.ROOT)`,
+// which Spark's Parquet footer field matching is built on.
+//
+// The data arrives on `NativeScanCommon` (populated by `JvmCaseTables.scala` when
+// case_sensitive = false), generated from the very JVM that plans the query, so the native
+// matcher is correct BY CONSTRUCTION for whatever JDK runs Spark. The one contextual mapping
+// (`Locale.ROOT` has exactly one: Greek capital sigma U+03A3) cannot be a per-codepoint table
+// entry, so its condition is ported as an algorithm over a shipped per-codepoint
+// classification -- see `JvmCaseTables::lowercase`.
+// ---------------------------------------------------------------------------------------------
+
+// Sigma-scan classes: the wire contract shared with `JvmCaseTables.scala`. The classes are
+// the UAX#29-style word-break classes (ALetter, Numeric, MidLetter, MidNum, MidNumLet,
+// Extend, Format) as the PLANNING JVM's legacy break iterator actually realizes them --
+// probed per codepoint from `BreakIterator.isBoundary` on the JVM side -- plus classes for
+// its pre-UAX#29 extensions (the danda and the supplementary-plane behaviors of its UTF-16
+// DFA), with cased variants split out via the JDK's `isCased`. Any codepoint outside every
+// shipped range -- and any class value this build does not know -- is a word boundary: the
+// sigma context scan stops there, which is also the safe reading for class values added by a
+// NEWER JVM-side generator.
+const CLASS_ALETTER_CASED: u8 = 1;
+const CLASS_ALETTER: u8 = 2;
+const CLASS_NUMERIC: u8 = 3;
+const CLASS_MID_LETTER: u8 = 4;
+const CLASS_MID_NUM: u8 = 5;
+const CLASS_MID_NUM_LET: u8 = 6;
+/// Cased supplementary char: attaches to the preceding word and closes it (and forms a word
+/// of its own at raw text start).
+const CLASS_SUPP_CASED: u8 = 7;
+/// U+0964/U+0965: word-terminal, chains only into digits.
+const CLASS_DANDA: u8 = 8;
+/// U+0345, the one cased combining mark: cased only when its run is attached to a word.
+const CLASS_EXTEND_CASED: u8 = 9;
+/// Cased digit-base (Nl Roman numerals): joins like CLASS_ALETTER_CASED when reached
+/// directly, but bridges only mid-num punctuation, never mid-letter.
+const CLASS_NUMERIC_CASED: u8 = 10;
+/// Non-cased Mn/Me marks: riders that attach only to genuine letter/digit bases.
+const CLASS_EXTEND: u8 = 11;
+/// Word-forming non-cased supplementary letter: a genuine letter-base that closes the word
+/// immediately after itself.
+const CLASS_SUPP_LETTER: u8 = 12;
+/// Cf format characters: fully transparent (WB4-style) -- deleted from the sequence before
+/// the scans run, so a pure-format rider chain bridges mid punctuation ("AΣ-<ZWJ>b" is one
+/// word exactly like "AΣ-b").
+const CLASS_FORMAT: u8 = 13;
+/// Supplementary chars that attach to the preceding word but never form one themselves
+/// (supplementary combining marks, tag characters): a cased mark riding on one belongs to
+/// the sigma's word only when the run hangs off a real base (`supp_mn_anchor`).
+const CLASS_SUPP_MN: u8 = 14;
+/// Word-forming supplementary digit: like CLASS_SUPP_LETTER except a riding cased mark
+/// carries only across mid-num (digit-context) punctuation, never mid-letter.
+const CLASS_SUPP_NUM: u8 = 15;
+/// Not on the wire: the absence of a class.
+const CLASS_BOUNDARY: u8 = 0;
+
+const CAPITAL_SIGMA: char = '\u{03A3}';
+const SMALL_SIGMA: char = '\u{03C3}';
+const SMALL_FINAL_SIGMA: char = '\u{03C2}';
+
+/// What a supplementary-mark run ultimately hangs off (see `supp_mn_anchor`).
+#[derive(PartialEq, Eq, Clone, Copy)]
+enum SuppMnAnchor {
+ None,
+ Letter,
+ Digit,
+}
+
+/// The planning JVM's case data, parsed once per scan from `NativeScanCommon` and attached to
+/// [`SparkParquetOptions`]. Two tables:
+///
+/// - `lower`: every codepoint the JVM lowercases non-identically, with its full (possibly
+/// multi-char, e.g. U+0130 -> "i" + U+0307) replacement; codepoints absent here lowercase
+/// to themselves;
+/// - `class_ranges`: sorted, disjoint `(start, end, class)` codepoint ranges holding the
+/// word-break classification the JVM probed from its own `BreakIterator`.
+///
+/// `lowercase` applies Java's algorithm over that data: per codepoint, U+03A3 takes its
+/// contextual final/non-final form via the ported `isFinalCased` condition -- word-boundary
+/// based (the JDK's legacy break-iterator word rules), NOT the Unicode-standard Final_Sigma
+/// case-ignorable skip, so e.g. "A1Σ" lowers to "a1ς" -- and every other codepoint takes its
+/// table replacement. `JvmCaseTables.mirrorLowercase` on the Scala side is the line-for-line
+/// mirror of this function over the same generated data; the JVM-side parity suite proves the
+/// pair equal to the running JDK's `String.toLowerCase(Locale.ROOT)` across the full codepoint
+/// space (calibrated to zero mismatches on JDK 17, 21, and 25).
+#[derive(Debug)]
+pub struct JvmCaseTables {
+ /// Non-identity lowercase mappings: codepoint -> full replacement string.
+ lower: HashMap<char, String>,
+ /// Sorted, disjoint (start, end, class) inclusive codepoint ranges for the sigma scan.
+ class_ranges: Vec<(u32, u32, u8)>,
+ /// Precomputed content hash so `SparkParquetOptions`'s derived `Hash` stays cheap.
+ fingerprint: u64,
+}
+
+impl PartialEq for JvmCaseTables {
+ fn eq(&self, other: &Self) -> bool {
+ self.fingerprint == other.fingerprint
+ && self.class_ranges == other.class_ranges
+ && self.lower == other.lower
+ }
+}
+
+impl Eq for JvmCaseTables {}
+
+impl Hash for JvmCaseTables {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ // Equal contents always produce the equal (deterministically computed) fingerprint,
+ // so hashing only the fingerprint is consistent with `PartialEq`.
+ state.write_u64(self.fingerprint);
+ }
+}
+
+fn is_letter_base(cls: u8) -> bool {
+ cls == CLASS_ALETTER_CASED
+ || cls == CLASS_ALETTER
+ || cls == CLASS_SUPP_CASED
+ || cls == CLASS_SUPP_LETTER
+}
+
+fn is_digit_base(cls: u8) -> bool {
+ cls == CLASS_NUMERIC || cls == CLASS_NUMERIC_CASED
+}
+
+impl JvmCaseTables {
+ /// Parse the proto representation: `lower_cp`/`lower_repl` are index-aligned, and
+ /// `class_ranges` holds (start, end, class) triples. Malformed input (length mismatch,
+ /// trailing partial triple, out-of-range codepoints) is dropped entry-by-entry rather
+ /// than rejected: every dropped entry degrades one codepoint to identity/boundary
+ /// behavior instead of failing the scan.
+ pub fn from_proto(lower_cp: &[u32], lower_repl: &[String], class_ranges: &[u32]) -> Self {
+ let mut lower = HashMap::with_capacity(lower_cp.len());
+ for (cp, repl) in lower_cp.iter().zip(lower_repl.iter()) {
+ if let Some(c) = char::from_u32(*cp) {
+ lower.insert(c, repl.clone());
+ }
+ }
+ let ranges: Vec<(u32, u32, u8)> = class_ranges
+ .as_chunks::<3>()
+ .0
+ .iter()
+ .filter(|t| t[0] <= t[1] && t[1] <= 0x10FFFF && u8::try_from(t[2]).is_ok())
+ .map(|t| (t[0], t[1], t[2] as u8))
+ .collect();
+
+ let mut hasher = DefaultHasher::new();
+ for (start, end, class) in &ranges {
+ hasher.write_u32(*start);
+ hasher.write_u32(*end);
+ hasher.write_u8(*class);
+ }
+ let mut lower_sorted: Vec<(&char, &String)> = lower.iter().collect();
+ lower_sorted.sort_by_key(|(c, _)| **c);
+ for (c, repl) in lower_sorted {
+ hasher.write_u32(*c as u32);
+ hasher.write(repl.as_bytes());
+ }
+
+ Self {
+ lower,
+ class_ranges: ranges,
+ fingerprint: hasher.finish(),
+ }
+ }
+
+ /// Sigma-scan class of `c`; `CLASS_BOUNDARY` when no shipped range covers it.
+ fn class_of(&self, c: char) -> u8 {
+ let cp = c as u32;
+ let mut lo = 0usize;
+ let mut hi = self.class_ranges.len();
+ while lo < hi {
+ let mid = (lo + hi) / 2;
+ let (start, end, class) = self.class_ranges[mid];
+ if cp < start {
+ hi = mid;
+ } else if cp > end {
+ lo = mid + 1;
+ } else {
+ return class;
+ }
+ }
+ CLASS_BOUNDARY
+ }
+
+ /// First position at or beyond `start` (stepping by `step`, i.e. -1 backward / +1
+ /// forward) whose class is not CLASS_EXTEND. Returns `None` if the scan runs off the
+ /// array without finding one.
+ fn skip_extends(&self, cps: &[char], start: isize, step: isize) -> Option<usize> {
+ let mut k = start;
+ while k >= 0 && (k as usize) < cps.len() {
+ if self.class_of(cps[k as usize]) != CLASS_EXTEND {
+ return Some(k as usize);
+ }
+ k += step;
+ }
+ None
+ }
+
+ /// As [`Self::skip_extends`], but also skips CLASS_EXTEND_CASED, reporting whether one
+ /// was walked: a cased mark (U+0345) crossed while looking for a base is itself cased
+ /// whenever the landing validates the run.
+ fn skip_extends_tracking_cased(
+ &self,
+ cps: &[char],
+ start: isize,
+ step: isize,
+ ) -> (Option<usize>, bool) {
+ let mut k = start;
+ let mut saw_cased = false;
+ while k >= 0 && (k as usize) < cps.len() {
+ let cls = self.class_of(cps[k as usize]);
+ if cls != CLASS_EXTEND && cls != CLASS_EXTEND_CASED {
+ return (Some(k as usize), saw_cased);
+ }
+ if cls == CLASS_EXTEND_CASED {
+ saw_cased = true;
+ }
+ k += step;
+ }
+ (None, saw_cased)
+ }
+
+ /// What the supplementary-mark run at `k` (CLASS_SUPP_MN) ultimately hangs off, walking
+ /// down through further marks and supplementary chars: a letter-flavored base, a
+ /// digit-flavored base, or nothing word-forming. A cased mark riding the run belongs to
+ /// the sigma's word only per this anchor.
+ fn supp_mn_anchor(&self, cps: &[char], k: usize) -> SuppMnAnchor {
+ let mut m = k as isize - 1;
+ while m >= 0 {
+ let cls = self.class_of(cps[m as usize]);
+ if cls != CLASS_SUPP_MN && cls != CLASS_EXTEND && cls != CLASS_EXTEND_CASED {
+ break;
+ }
+ m -= 1;
+ }
+ if m < 0 {
+ return SuppMnAnchor::None;
+ }
+ match self.class_of(cps[m as usize]) {
+ CLASS_ALETTER_CASED | CLASS_ALETTER | CLASS_SUPP_CASED | CLASS_SUPP_LETTER => {
+ SuppMnAnchor::Letter
+ }
+ CLASS_NUMERIC | CLASS_NUMERIC_CASED | CLASS_SUPP_NUM => SuppMnAnchor::Digit,
+ _ => SuppMnAnchor::None,
+ }
+ }
+
+ /// Backward half of the ported `isFinalCased`: is there a cased letter before position
+ /// `i` within the sigma's word? Runs over the FORMAT-FILTERED sequence; `leading_format`
+ /// says whether format chars were filtered off the raw text start.
+ fn scan_back_finds_cased(&self, cps: &[char], i: usize, leading_format: bool) -> bool {
+ let mut last_letter = true; // the sigma itself is a letter
+ let mut j = i as isize - 1;
+ while j >= 0 {
+ match self.class_of(cps[j as usize]) {
+ CLASS_ALETTER_CASED | CLASS_NUMERIC_CASED => return true,
+ CLASS_ALETTER => {
+ last_letter = true;
+ j -= 1;
+ }
+ CLASS_NUMERIC => {
+ last_letter = false;
+ j -= 1;
+ }
+ CLASS_EXTEND => {
+ // Non-cased marks attach only to a real base below them; anything else
+ // (mid punctuation, danda, boundary, text start) leaves the run
+ // unattached.
+ let Some(k) = self.skip_extends(cps, j, -1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ let is_continuer = b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_EXTEND_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_NUM;
+ if !is_continuer {
+ return false;
+ }
+ j = k as isize;
+ }
+ CLASS_SUPP_CASED => {
+ // Closes the preceding word, so the scan stops -- except at RAW text
+ // start (no filtered-out leading format chars), where the DFA keeps it
+ // joined to what follows.
+ return j == 0 && !leading_format;
+ }
+ CLASS_SUPP_LETTER | CLASS_SUPP_MN | CLASS_SUPP_NUM => {
+ // Attach/close and never themselves cased; nothing beyond is reachable.
+ return false;
+ }
+ CLASS_EXTEND_CASED => {
+ // Cased combining mark (U+0345): cased when its run hangs off a base --
+ // a BMP letter/digit, a word-forming supplementary char (which closes a
+ // word right below the mark, merging the mark into the sigma's
+ // segment), or an ANCHORED supplementary mark.
+ let (Some(k), _) = self.skip_extends_tracking_cased(cps, j - 1, -1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ if b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_NUM
+ {
+ return true;
+ }
+ if b == CLASS_SUPP_MN {
+ return self.supp_mn_anchor(cps, k) != SuppMnAnchor::None;
+ }
+ return false;
+ }
+ CLASS_DANDA => {
+ // Backward across a danda: the word part before it must end in letters
+ // (grammar: letters, optional danda, then number+word chains) -- or
+ // carry a riding cased mark on a word-forming base, or be a cased
+ // supplementary char at text start -- and the danda itself chains only
+ // into digits after it.
+ if last_letter {
+ return false;
+ }
+ let (Some(k), saw_cased_mark) =
+ self.skip_extends_tracking_cased(cps, j - 1, -1)
+ else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ if b == CLASS_ALETTER_CASED {
+ return true;
+ }
+ if b == CLASS_SUPP_CASED {
+ return saw_cased_mark || (k == 0 && !leading_format);
+ }
+ if b == CLASS_SUPP_LETTER {
+ return saw_cased_mark;
+ }
+ if b == CLASS_SUPP_MN {
+ return saw_cased_mark
+ && self.supp_mn_anchor(cps, k) == SuppMnAnchor::Letter;
+ }
+ if b != CLASS_ALETTER {
+ return false;
+ }
+ if saw_cased_mark {
+ return true;
+ }
+ last_letter = true;
+ j = k as isize;
+ }
+ cls @ (CLASS_MID_LETTER | CLASS_MID_NUM | CLASS_MID_NUM_LET) => {
+ // `<mid-letter><let>` / `<mid-num><digit>` require a genuine
+ // letter/digit base before the punctuation; scanning backward
+ // legitimately walks marks-then-base (marks trail their base). A cased
+ // mark walked over rides whatever the punctuation hangs off, including
+ // a context-matching anchored supplementary mark or supplementary
+ // digit.
+ let mw_ok = cls == CLASS_MID_LETTER || cls == CLASS_MID_NUM_LET;
+ let mn_ok = cls == CLASS_MID_NUM || cls == CLASS_MID_NUM_LET;
+ let (Some(real_pos), saw_cased_mark) =
+ self.skip_extends_tracking_cased(cps, j - 1, -1)
+ else {
+ return false;
+ };
+ let b = self.class_of(cps[real_pos]);
+ if last_letter
+ && mw_ok
+ && saw_cased_mark
+ && b == CLASS_SUPP_MN
+ && self.supp_mn_anchor(cps, real_pos) == SuppMnAnchor::Letter
+ {
+ return true;
+ }
+ if !last_letter
+ && mn_ok
+ && saw_cased_mark
+ && (b == CLASS_SUPP_NUM
+ || (b == CLASS_SUPP_MN
+ && self.supp_mn_anchor(cps, real_pos) == SuppMnAnchor::Digit))
+ {
+ return true;
+ }
+ let bridge_valid = (last_letter && mw_ok && is_letter_base(b))
+ || (!last_letter && mn_ok && is_digit_base(b));
+ if !bridge_valid {
+ return false;
+ }
+ if saw_cased_mark {
+ return true;
+ }
+ j = real_pos as isize;
+ }
+ _ => return false,
+ }
+ }
+ false
+ }
+
+ /// Forward half of the ported `isFinalCased`: is there a cased letter after position `i`
+ /// within the sigma's word? Runs over the FORMAT-FILTERED sequence.
+ fn scan_fwd_finds_cased(&self, cps: &[char], i: usize) -> bool {
+ let mut last_letter = true;
+ let mut j = i + 1;
+ while j < cps.len() {
+ match self.class_of(cps[j]) {
+ CLASS_ALETTER_CASED | CLASS_NUMERIC_CASED => return true,
+ CLASS_ALETTER => {
+ last_letter = true;
+ j += 1;
+ }
+ CLASS_NUMERIC => {
+ last_letter = false;
+ j += 1;
+ }
+ CLASS_EXTEND => {
+ // A mark run trailing the anchor is properly attached in text order, so
+ // the run stays open past it, including onto mid punctuation on its far
+ // side.
+ let Some(k) = self.skip_extends(cps, j as isize, 1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ let is_continuer = b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_EXTEND_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_MN
+ || b == CLASS_SUPP_NUM
+ || b == CLASS_DANDA
+ || b == CLASS_MID_LETTER
+ || b == CLASS_MID_NUM
+ || b == CLASS_MID_NUM_LET;
+ if !is_continuer {
+ return false;
+ }
+ j = k;
+ }
+ CLASS_SUPP_CASED | CLASS_EXTEND_CASED => {
+ // Attaches to the current word, so the scan sees it (cased).
+ return true;
+ }
+ CLASS_SUPP_LETTER | CLASS_SUPP_MN | CLASS_SUPP_NUM => {
+ // Attach to the current word and close it; never themselves cased, and
+ // nothing beyond is reachable.
+ return false;
+ }
+ // The danda attaches only to a word part that ends in letters (reached
+ // after digits the word is already closed) and continues only into a digit
+ // -- unless that digit is itself cased (a Roman numeral), which resolves
+ // the scan immediately.
+ CLASS_DANDA if !last_letter => return false,
+ CLASS_DANDA
+ if j + 1 < cps.len() && self.class_of(cps[j + 1]) == CLASS_NUMERIC_CASED =>
+ {
+ return true;
+ }
+ CLASS_DANDA if j + 1 < cps.len() && self.class_of(cps[j + 1]) == CLASS_NUMERIC => {
+ last_letter = false;
+ j += 2;
+ }
+ cls @ (CLASS_MID_LETTER | CLASS_MID_NUM | CLASS_MID_NUM_LET) => {
+ // `<mid-letter><let>` / `<mid-num><digit>` require a genuine
+ // letter/digit base IMMEDIATELY after the punctuation -- unlike the
+ // backward scan, marks here are never skipped past: a mark directly
+ // after the punctuation is attached to the punctuation, not a base, so
+ // it blocks the bridge. (Format chars are already filtered out, which
+ // is what lets "AΣ-<ZWJ>b" bridge exactly like "AΣ-b".)
+ let mw_ok = cls == CLASS_MID_LETTER || cls == CLASS_MID_NUM_LET;
+ let mn_ok = cls == CLASS_MID_NUM || cls == CLASS_MID_NUM_LET;
+ if j + 1 >= cps.len() {
+ return false;
+ }
+ let b = self.class_of(cps[j + 1]);
+ if (last_letter && mw_ok && is_letter_base(b))
+ || (!last_letter && mn_ok && is_digit_base(b))
+ {
+ j += 1;
+ } else {
+ return false;
+ }
+ }
+ _ => return false,
+ }
+ }
+ false
+ }
+
+ /// Lowercase `s` the way the planning JVM's `String.toLowerCase(Locale.ROOT)` does.
+ pub fn lowercase(&self, s: &str) -> String {
+ let raw: Vec<char> = s.chars().collect();
+ // Built lazily on the first sigma: the format-filtered sequence the scans run over
+ // (WB4-style: the legacy break iterator's `<ignore>` class loops on every DFA
+ // state), the raw->filtered index map, and whether format chars led the raw text.
+ let mut filtered: Option<(Vec<char>, Vec<usize>, bool)> = None;
+ let mut out = String::with_capacity(s.len());
+ for (i, &c) in raw.iter().enumerate() {
+ if c == CAPITAL_SIGMA {
+ // The condition consults the ORIGINAL neighbors, exactly as the JDK scans
+ // `src`, not the partially-lowered output. The final/non-final target chars
+ // are Unicode-stable (pinned in `ConditionalSpecialCasing`'s entry table).
+ let (f, idx, leading_format) = filtered.get_or_insert_with(|| {
+ let mut f = Vec::with_capacity(raw.len());
+ let mut idx = vec![0usize; raw.len()];
+ for (k, &rc) in raw.iter().enumerate() {
+ idx[k] = f.len();
+ if self.class_of(rc) != CLASS_FORMAT {
+ f.push(rc);
+ }
+ }
+ let leading_format = self.class_of(raw[0]) == CLASS_FORMAT;
+ (f, idx, leading_format)
+ });
+ let fi = idx[i];
+ let is_final = self.scan_back_finds_cased(f, fi, *leading_format)
+ && !self.scan_fwd_finds_cased(f, fi);
+ out.push(if is_final {
+ SMALL_FINAL_SIGMA
+ } else {
+ SMALL_SIGMA
+ });
+ } else if let Some(repl) = self.lower.get(&c) {
+ out.push_str(repl);
+ } else {
+ out.push(c);
+ }
+ }
+ out
+ }
+}
+
+/// Lowercase `s` for case-insensitive field matching. With tables (populated whenever
+/// `case_sensitive = false`, shared by the core Parquet scan and the Delta contrib scan) this
+/// reproduces the planning JVM's `String.toLowerCase(Locale.ROOT)` exactly.
+///
+/// Without tables, fall back to Rust's `str::to_lowercase`, which agrees with Java on all
+/// simple mappings and differs only where the two Unicode snapshots or the sigma context
+/// diverge. This is a real, live path, not just a defensive default: the Iceberg native scan
+/// (`SparkPhysicalExprAdapterFactory::new(_, None)`) defaults `case_sensitive = false` and
+/// always reaches this fallback for its schema name remap, alongside
+/// `parquet_convert_struct_to_struct`'s general struct-cast matching and Rust-only unit tests
+/// that construct `SparkParquetOptions` directly.
+pub(crate) fn java_lowercase(s: &str, tables: Option<&JvmCaseTables>) -> String {
Review Comment:
This is the one I would most like to see addressed before merge.
`names_equal_ignore_case_java` replaces `eq_ignore_ascii_case` throughout the scan path, and it costs two `String` allocations plus one `HashMap<char, String>` lookup per character. `remap_physical_schema` is called once per Parquet file opened (DataFusion `opener/mod.rs:844`, guarded by `needs_rewrite`, which is true whenever there is a predicate or the projection prunes columns, so in practice always), and it does O(physical x logical) comparisons. `spark.sql.caseSensitive` defaults to `false`, so this is the default path for every Comet user, Delta or not.
I measured it against a worktree at this PR's base commit, release profile, best of 3, using identical already-lowercase ASCII column names, which is the friendliest case for the old code. Per Parquet file opened:
| columns | base | this PR | |
|---|---|---|---|
| 23 | 7.2 us | 138 us | 19x |
| 64 | 24.2 us | 912 us | 38x |
| 200 | 141 us | 8.57 ms | 61x |
| 1024 | 3.83 ms | 230 ms | 60x |
End to end through a real `DataSourceExec`, including Parquet decode:
| shape | base | this PR | |
|---|---|---|---|
| 500 files x 24 cols, project 10 | 70.4 ms | 162 ms | 2.3x |
| 100 files x 200 cols, project 20 | 28.2 ms | 338 ms | 12.0x |
A 10,000-file scan of a 200-column table would pay roughly 84 seconds of extra CPU.
There is an exact fast path available. I dumped the real table your generator produces on JDK 17 and checked it: the only ASCII entries are the 26 plain `A-Z` to `a-z` single-character mappings, and no ASCII codepoint is U+03A3. So for two pure-ASCII names, `java_lowercase(a) == java_lowercase(b)` is equivalent to `a.eq_ignore_ascii_case(b)`. Would you add that guard at the top of `names_equal_ignore_case_java`?
```rust
if a.is_ascii() && b.is_ascii() {
return a.eq_ignore_ascii_case(b);
}
```
That keeps the Kelvin sign and dotted-I cases working, since neither is ASCII, while making the overwhelmingly common case free. Hoisting the lowercased forms out of the O(P x L) loop in `remap_physical_schema` (lowercase each schema once, then compare precomputed strings) would cover the remaining non-ASCII case, and `check_column_duplicate` would benefit from the same treatment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5891089529
> Could the physical-to-requested leaf pairing choose the timestamp policy from the requested type in both directions?
Yes, in 4b7ad02. The walk that pairs each physical leaf with its requested type now records the timezone-free leaves read as `TIMESTAMP` and the timezone-carrying ones read as `TIMESTAMP_NTZ`, and the policy follows that. A metadata-free `TIMESTAMP(MICROS, false)` file read as `TIMESTAMP` now fails under `EXCEPTION` like Spark (the Delta test also checks that stock Spark throws `SparkUpgradeException` for the same table), and reads the value unchanged under `CORRECTED`.
Under `LEGACY` the native scan refuses that file rather than rebasing it: with no `org.apache.spark.timeZone` in the footer, Spark rebases in the JVM's default time zone, which the native side cannot reproduce. That is the same refusal adjusted files without a recorded writer zone already get. A file that records a UTC writer zone is rebased exactly.
The regression you suggested is `tz_free_int64_timestamps_requested_as_ltz_follow_the_datetime_spec` in `parquet_exec.rs`, plus the Delta test "non-Spark tz-free INT64 timestamps read as TIMESTAMP follow the datetime read mode".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4042635977
##########
.github/workflows/delta_contrib_test.yml:
##########
@@ -0,0 +1,169 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Delta Contrib Tests
+
+# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
+# live in the umbrella workflow.
+on:
+ workflow_call:
+
+permissions:
+ contents: read
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: rust-lld cannot resolve -ljvm against the Zulu JDK
+ # layout installed by setup-java (same rationale as pr_build_linux.yml).
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+ # The container's default locale is POSIX, which makes the JVM's file-path encoder
+ # reject non-ASCII partition directory names the suites create.
+ LANG: "C.UTF-8"
+ LC_ALL: "C.UTF-8"
+
+jobs:
+
+ contrib-delta:
+ name: Delta contrib (Spark ${{ matrix.profile.spark }})
+ runs-on: ubuntu-24.04
+ container:
Review Comment:
> Could the suite take an env var, something like `COMET_DELTA_S3_REQUIRED=1`, that this job sets to turn the cancel into a hard failure?
Done in 17546fd. The MinIO step sets `COMET_DELTA_S3_REQUIRED=1`, and with it armed the suite fails in `beforeAll` when Docker is unavailable or MinIO does not start, after tearing down what it started, instead of cancelling. Only `1` or `true` arm it; the parsing and the failure message are unit-tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4169096373
##########
native/core/src/parquet/datetime_rebase.rs:
##########
@@ -0,0 +1,3407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Per-file datetime calendar-rebase handling for the parquet scan.
+//!
+//! Spark 2.4 and earlier wrote dates and timestamps in the hybrid Julian + Gregorian calendar;
+//! Spark 3.0+ uses the proleptic Gregorian calendar and records the calendar policy of every
+//! file it writes in the parquet footer's key-value metadata (`org.apache.spark.version`,
+//! `org.apache.spark.legacyDateTime`, `org.apache.spark.legacyINT96`,
+//! `org.apache.spark.timeZone`). Spark's reader resolves the rebase policy from EACH FILE's
+//! writer metadata (`DataSourceUtils.datetimeRebaseSpec` / `int96RebaseSpec`) -- the session's
+//! `spark.sql.parquet.datetimeRebaseModeInRead` conf only applies to files whose metadata does
+//! not decide the policy on its own -- so a reader that ignores the metadata silently returns
+//! values shifted by up to ten days for dates before 1582-10-15 (e.g. `1500-01-01` reads as
+//! `1500-01-10`).
+//!
+//! This module mirrors that per-file resolution: [`resolve_file_rebase_policies`] computes the
+//! date / INT64-timestamp / INT96-timestamp policies from a file's arrow schema metadata (the
+//! parquet key-value pairs survive the parquet -> arrow schema conversion), and
+//! [`wrap_datetime_rebase`] wraps the per-file rewritten expressions' column references in a
+//! [`SparkDatetimeRebaseExpr`] that rebases values exactly where that is possible without the
+//! JVM's historical timezone tables (dates always; timestamps for a fixed UTC writer zone) and
+//! refuses -- rather than silently corrupting -- ancient values it cannot rebase. Nested
+//! columns are rebuilt leaf by leaf (struct / list / map / fixed-size list / dictionary), each
+//! leaf under its own policy, with nulls and offsets preserved. Modern values are always the
+//! identity under every policy: from 1582-10-15 onward for dates, and from
+//! [`LAST_SWITCH_JULIAN_TS_SECONDS`] (1900-01-01T00:00:00Z, Spark's
+//! `RebaseDateTime.lastSwitchJulianTs`) onward for timestamps.
+//!
+//! Spark applies `datetimeRebaseSpec` to INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns
+//! and `int96RebaseSpec` to INT96 columns. The two physical types are indistinguishable in the
+//! arrow schema DataFusion hands the expression adapter (both surface as `Timestamp(us, "UTC")`
+//! after INT96 coercion), so Comet's parquet reader factory stamps the file's INT96 leaf
+//! ordinals -- taken from the parquet footer's own `SchemaDescriptor` -- into the key-value
+//! metadata under [`INT96_LEAVES_METADATA_KEY`] before the arrow schema is derived (see
+//! [`stamp_int96_leaves`] and `eager_page_index_reader_factory.rs`), and the adapter attributes
+//! every timestamp leaf to its spec from that stamp. Without a stamp, the two specs are merged:
+//! agreement decides, disagreement degrades to [`RebasePolicy::CheckAncient`].
+//!
+//! The wrapper sits BENEATH the schema adapter's nested narrowing (the struct -> struct convert
+//! that keeps only the requested children), which is what keeps those ordinals physical -- but
+//! it means the wrapper sees every physical child, requested or not. Spark only ever decodes
+//! the requested nested schema, so [`FileRebasePolicies::restrict_to_requested`] marks the
+//! physical leaves the narrowing drops as the identity: an unrequested ancient `s.ts` never
+//! blocks `select s.d`, exactly as in Spark.
+//!
+//! The same pairing decides a timestamp leaf's policy by the type the query READS it as, since
+//! `ParquetVectorUpdaterFactory.getUpdater` keys on the requested Spark type, not the parquet
+//! annotation: a leaf read as `TIMESTAMP_NTZ` never rebases (INT96 or INT64; Spark 4.x's
+//! `BinaryToSQLTimestampUpdater` / `LongUpdater` consult no mode, and Spark 3.x refuses the
+//! INT96 and adjusted-INT64 pairings outright, which Comet's `allow_timestamp_ltz_to_ntz` gate
+//! reproduces); a leaf read as `TIMESTAMP` rebases under the datetime spec even when the file
+//! declares it `isAdjustedToUTC=false` (`isTimestampTypeMatched` checks the unit only); and a
+//! `DATE` leaf keeps the date policy whether read as `DATE` or, on Spark 4.x, as
+//! `TIMESTAMP_NTZ` (`DateToTimestampNTZWithRebaseUpdater`). A date or timestamp leaf read as a
+//! number (a `DATE` as `INT`, a `TIMESTAMP` of any unit as `BIGINT`, e.g. NANOS under
+//! `spark.sql.legacy.parquet.nanosAsLong`) never rebases: the numeric updaters come first in
+//! `getUpdater` and return the stored value.
+//!
+//! Currently only enabled by the Delta scan arms via
+//! `SparkParquetOptions::rebase_from_file_metadata`, which also carries the session read modes
+//! ([`SessionRebaseModes`], forwarded from the JVM) that decide the policy for files without
+//! Spark writer metadata; the plain NativeScan keeps its documented no-rebase behavior (see
+//! the compatibility guide and issue #5010).
+
+use std::collections::HashMap;
+use std::fmt::{self, Display};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Date32Array, FixedSizeListArray, GenericListArray, MapArray,
+ OffsetSizeTrait, PrimitiveArray, RecordBatch, StructArray,
+};
+use arrow::datatypes::{
+ ArrowPrimitiveType, ArrowTimestampType, DataType, Date32Type, FieldRef, Schema, SchemaRef,
+ TimeUnit, TimestampMicrosecondType, TimestampMillisecondType, TimestampNanosecondType,
+ TimestampSecondType,
+};
+use arrow::error::ArrowError;
+use datafusion::common::tree_node::{Transformed, TreeNode};
+use datafusion::common::{DataFusionError, Result as DataFusionResult};
+use datafusion::physical_expr::expressions::Column;
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::ColumnarValue;
+use parquet::basic::Type as ParquetPhysicalType;
+use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaData};
+use parquet::schema::types::SchemaDescriptor;
+
+use super::name_fold::fold_names;
+use super::parquet_support::field_id;
+
+/// Footer key naming the Spark release that wrote the file; absent for non-Spark writers.
+const SPARK_VERSION_METADATA_KEY: &str = "org.apache.spark.version";
+/// Present (empty value) when the file's dates and INT64 timestamps were written with
+/// `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_DATETIME_KEY: &str = "org.apache.spark.legacyDateTime";
+/// Present (empty value) when the file's INT96 timestamps were written with
+/// `spark.sql.parquet.int96RebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_INT96_KEY: &str = "org.apache.spark.legacyINT96";
+/// The writer session's time zone, stamped alongside either legacy flag.
+const SPARK_TIMEZONE_KEY: &str = "org.apache.spark.timeZone";
+
+/// Key-value metadata entry Comet's parquet reader factory adds to a file's footer metadata
+/// (in memory only, never written back) so the expression adapter can tell INT96 timestamp
+/// columns from INT64 ones after both have been coerced to the same arrow type. Value:
+/// `"<leaf count>:<comma-separated INT96 leaf ordinals>"`, where leaves are the file's
+/// primitive columns in `SchemaDescriptor::columns()` order -- the same depth-first order
+/// parquet-rs assigns arrow leaves, so an arrow-side depth-first walk lines up with it. The
+/// leaf count lets the reader detect a stamp that does not describe the schema it is paired
+/// with (see [`Int96Attribution::from_schema`]).
+pub(crate) const INT96_LEAVES_METADATA_KEY: &str = "comet.int96_leaf_columns";
+
+/// Day of the Gregorian cutover (1582-10-15) as days since the epoch; rebasing is the identity
+/// from this day onward. Same value as Spark's `RebaseDateTime.lastSwitchJulianDay`.
+const LAST_SWITCH_JULIAN_DAY: i32 = -141427;
+
+/// Spark's `RebaseDateTime.lastSwitchJulianTs` (and `lastSwitchGregorianTs`) in seconds since
+/// the epoch: 1900-01-01T00:00:00Z. Spark derives it as the latest switch instant across every
+/// zone in its `julian-gregorian-rebase-micros.json` table (`getLastSwitchTs`, which also
+/// asserts the calendars' difference is zero for every zone from then on): most zones ran on
+/// local mean time before 1900, so the last instant at which rebasing changes a value in ANY
+/// zone is 1900-01-01T00:00:00Z, not the 1582 cutover. `createTimestampRebaseFuncInRead`
+/// under `EXCEPTION` throws exactly for `micros < lastSwitchJulianTs` (after converting
+/// `TIMESTAMP_MILLIS` to micros), and `rebaseJulianToGregorianMicros` is the identity from it
+/// onward in every zone. The value is in seconds so it scales exactly to any timestamp unit.
+pub(crate) const LAST_SWITCH_JULIAN_TS_SECONDS: i64 = -2_208_988_800;
+
+/// The per-century differences between the Julian and proleptic Gregorian calendars, and the
+/// Julian-calendar switch days at which each difference starts to apply. Copied verbatim from
+/// Spark's `RebaseDateTime.julianGregDiffs` / `julianGregDiffSwitchDay` (which Spark generated
+/// from `localRebaseJulianToGregorianDays`); `rebase_julian_to_gregorian_days` must stay
+/// value-for-value equal to Spark's `rebaseJulianToGregorianDays`.
+const JULIAN_GREG_DIFFS: [i32; 14] = [2, 1, 0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, 0];
+const JULIAN_GREG_DIFF_SWITCH_DAY: [i32; 14] = [
+ -719164, -682945, -646420, -609895, -536845, -500320, -463795, -390745, -354220, -317695,
+ -244645, -208120, -171595, -141427,
+];
+
+/// Proleptic-Gregorian days since 1970-01-01 for a nominal civil date, via Howard Hinnant's
+/// `days_from_civil`. `d` may exceed the month's length; the excess rolls into the following
+/// month exactly like `LocalDate.of(y, m, 1).plusDays(d - 1)` in Spark's
+/// `localRebaseJulianToGregorianDays` (how the non-existent proleptic date `1000-02-29`,
+/// valid in the Julian calendar, lands on `1000-03-01`).
+fn days_from_civil(y: i64, m: i64, d: i64) -> i64 {
+ let y = if m <= 2 { y - 1 } else { y };
+ let era = y.div_euclid(400);
+ let yoe = y - era * 400; // [0, 399]
+ let mp = (m + 9) % 12; // [0, 11], March = 0
+ let doy = (153 * mp + 2) / 5 + d - 1;
+ let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
+ era * 146097 + doe - 719468
+}
+
+/// Julian-calendar civil date `(year, month, day)` for a day count since 1970-01-01 that labels
+/// days in the Julian calendar (astronomical year numbering: 1 BCE is year 0). Standard
+/// Julian-day-number conversion (E.G. Richards' algorithm), exact for any day.
+fn julian_day_to_civil(days: i64) -> (i64, i64, i64) {
+ // Integer (noon) Julian Day Number of this civil day: 1970-01-01 is JDN 2440588.
+ let jdn = days + 2_440_588;
+ let f = jdn + 1401;
+ let e = 4 * f + 3;
+ let g = e.rem_euclid(1461) / 4;
+ let h = 5 * g + 2;
+ let day = h.rem_euclid(153) / 5 + 1;
+ let month = (h / 153 + 2).rem_euclid(12) + 1;
+ let year = e.div_euclid(1461) - 4716 + (14 - month) / 12;
+ (year, month, day)
+}
+
+/// Exact port of Spark's `RebaseDateTime.rebaseJulianToGregorianDays`: reinterprets a day count
+/// written in the hybrid Julian + Gregorian calendar as the proleptic Gregorian day count of the
+/// same nominal civil date. Identity for days from 1582-10-15 onward. Days before the tables'
+/// range (before Julian `0001-01-01`) take the calendar-arithmetic path, mirroring Spark's
+/// `localRebaseJulianToGregorianDays` fallback.
+pub(crate) fn rebase_julian_to_gregorian_days(days: i32) -> i32 {
+ if days < JULIAN_GREG_DIFF_SWITCH_DAY[0] {
+ let (y, m, d) = julian_day_to_civil(days as i64);
+ (days_from_civil(y, m, 1) + (d - 1)) as i32
+ } else {
+ // Spark's rebaseDays: linear search from the most recent switch day.
+ let mut i = JULIAN_GREG_DIFF_SWITCH_DAY.len();
+ loop {
+ i -= 1;
+ if i == 0 || days >= JULIAN_GREG_DIFF_SWITCH_DAY[i] {
+ break;
+ }
+ }
+ days + JULIAN_GREG_DIFFS[i]
+ }
+}
+
+/// Timezone strings from `org.apache.spark.timeZone` that denote a fixed zero-offset zone in
+/// both `java.util.TimeZone` and `java.time`. Only for these is timestamp rebasing the pure
+/// nominal-date shift [`SparkDatetimeRebaseExpr::rebase_timestamp_utc`] computes; any other (or
+/// absent) zone needs the JVM's historical timezone tables and stays on the
+/// refuse-ancient-values path.
+const UTC_EQUIVALENT_TIMEZONES: [&str; 6] = ["UTC", "Etc/UTC", "GMT", "Etc/GMT", "Z", "+00:00"];
+
+/// How the writer's session time zone (if recorded) affects timestamp rebasing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum WriterTimeZone {
+ /// A fixed zero-offset zone: rebasing reduces to the exact nominal-date shift.
+ Utc,
+ /// Any other zone, or none recorded (pre-3.0 files): ancient values cannot be rebased
+ /// without the JVM's historical timezone data.
+ OtherOrUnknown,
+}
+
+/// One session-level datetime rebase read mode (a `LegacyBehaviorPolicy` value of
+/// `spark.sql.parquet.datetimeRebaseModeInRead` / `int96RebaseModeInRead`), consulted by
+/// [`resolve_file_rebase_policies`] ONLY for files whose footer metadata does not decide the
+/// policy on its own -- exactly the `getOrElse` fallback in Spark's
+/// `DataSourceUtils.getRebaseSpec`. Files that carry `org.apache.spark.version` ignore these
+/// modes entirely, on every Spark version.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) enum RebaseReadMode {
+ /// Refuse ancient values (Spark raises `SparkUpgradeException`); maps to
+ /// [`RebasePolicy::CheckAncient`]. The default mirrors the conservative posture used
+ /// before the conf was plumbed through (and Spark 3.x's own conf default).
+ #[default]
+ Exception,
+ /// Read values as proleptic Gregorian without rebasing.
+ Corrected,
+ /// Rebase from the hybrid Julian + Gregorian calendar.
+ Legacy,
+}
+
+impl RebaseReadMode {
+ /// Parses a `LegacyBehaviorPolicy` conf value. `SQLConf` validates and upper-cases the
+ /// session conf, but a per-relation `datetimeRebaseMode` option arrives verbatim, so the
+ /// match is case-insensitive. Anything unrecognized -- including the empty string a proto
+ /// producer that predates the field sends -- falls back to [`RebaseReadMode::Exception`],
+ /// which refuses ancient values rather than silently corrupting them.
+ pub(crate) fn from_conf_value(value: &str) -> Self {
+ match value.to_ascii_uppercase().as_str() {
+ "CORRECTED" => RebaseReadMode::Corrected,
+ "LEGACY" => RebaseReadMode::Legacy,
+ _ => RebaseReadMode::Exception,
+ }
+ }
+}
+
+/// The session's effective datetime rebase read modes, one per spec class (INT64
+/// dates/timestamps vs INT96 timestamps), forwarded from the JVM at planning time.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) struct SessionRebaseModes {
+ /// `spark.sql.parquet.datetimeRebaseModeInRead` (or the relation's `datetimeRebaseMode`).
+ pub datetime: RebaseReadMode,
+ /// `spark.sql.parquet.int96RebaseModeInRead` (or the relation's `int96RebaseMode`).
+ pub int96: RebaseReadMode,
+}
+
+/// Calendar policy of one file's date or timestamp columns, resolved from writer metadata the
+/// same way Spark's `DataSourceUtils.getRebaseSpec` resolves it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum RebasePolicy {
+ /// Written in the proleptic Gregorian calendar; values pass through untouched.
+ Corrected,
+ /// Written in the hybrid Julian + Gregorian calendar; values must be rebased.
+ Legacy(WriterTimeZone),
+ /// Policy could not be pinned down (contradictory flags, or a non-Spark writer under the
+ /// `EXCEPTION` read mode): modern values -- identical under either calendar -- pass,
+ /// ancient values raise. Mirrors Spark's `EXCEPTION` behavior (`SparkUpgradeException`).
+ CheckAncient,
+}
+
+/// Which of a file's leaf columns are physically INT96, from the stamp the parquet reader
+/// factory adds under [`INT96_LEAVES_METADATA_KEY`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) enum Int96Attribution {
+ /// No stamp, or a stamp whose leaf count does not match the schema it arrived with: the
+ /// INT64 and INT96 timestamp specs cannot be told apart per column and are merged.
+ Unknown,
+ /// Sorted leaf ordinals (depth-first over the file schema's primitive columns) that are
+ /// INT96; every other timestamp leaf is INT64.
+ Known(Vec<usize>),
+}
+
+impl Int96Attribution {
+ /// Parses the stamp out of `schema`'s metadata and validates its leaf count against the
+ /// schema's own depth-first leaf count, so a stamp that does not describe this schema (a
+ /// crafted footer key, or a cached-metadata mismatch) degrades to [`Self::Unknown`].
+ fn from_schema(schema: &Schema) -> Self {
+ let Some(stamp) = schema.metadata().get(INT96_LEAVES_METADATA_KEY) else {
+ return Int96Attribution::Unknown;
+ };
+ let Some((count, ordinals)) = stamp.split_once(':') else {
+ return Int96Attribution::Unknown;
+ };
+ let schema_leaves: usize = schema
+ .fields()
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ if count.parse::<usize>().ok() != Some(schema_leaves) {
+ return Int96Attribution::Unknown;
+ }
+ let parsed: Option<Vec<usize>> = if ordinals.is_empty() {
+ Some(Vec::new())
+ } else {
+ ordinals
+ .split(',')
+ .map(|o| o.parse::<usize>().ok().filter(|o| *o < schema_leaves))
+ .collect()
+ };
+ match parsed {
+ Some(mut leaves) => {
+ leaves.sort_unstable();
+ Int96Attribution::Known(leaves)
+ }
+ None => Int96Attribution::Unknown,
+ }
+ }
+
+ /// `Some(true)` / `Some(false)` when the leaf is known to be INT96 / INT64, `None` when
+ /// the attribution is unknown.
+ fn is_int96(&self, leaf: usize) -> Option<bool> {
+ match self {
+ Int96Attribution::Unknown => None,
+ Int96Attribution::Known(leaves) => Some(leaves.binary_search(&leaf).is_ok()),
+ }
+ }
+}
+
+/// The [`INT96_LEAVES_METADATA_KEY`] value describing `schema`: its leaf count and the
+/// ordinals of its INT96 primitive columns.
+pub(crate) fn int96_leaf_stamp(schema: &SchemaDescriptor) -> String {
+ let ordinals: Vec<String> = schema
+ .columns()
+ .iter()
+ .enumerate()
+ .filter(|(_, column)| column.physical_type() == ParquetPhysicalType::INT96)
+ .map(|(ordinal, _)| ordinal.to_string())
+ .collect();
+ format!("{}:{}", schema.num_columns(), ordinals.join(","))
+}
+
+/// Returns a copy of `metadata` whose key-value metadata carries the [`int96_leaf_stamp`] of
+/// its own schema, or `None` when it already does (the common case after the first open of a
+/// file, since the caller caches the stamped copy). Any pre-existing entry under the key --
+/// a file cannot legitimately carry one -- is replaced, never trusted. Only the file-level
+/// key-value list changes; row groups and page indexes are carried over as-is. The parquet
+/// API cannot carry a file decryptor, nor `FileMetaData`'s crate-private encryption fields
+/// (encryption algorithm, footer signing key metadata), across this rebuild, so callers must
+/// not stamp opens that supply decryption properties -- and the only consumer, the Delta
+/// scan, declines every encrypted-parquet configuration before planning, so a parquet
+/// modular encryption file never reaches this path with or without those properties.
+pub(crate) fn stamp_int96_leaves(metadata: &ParquetMetaData) -> Option<ParquetMetaData> {
+ let file_metadata = metadata.file_metadata();
+ let stamp = int96_leaf_stamp(file_metadata.schema_descr());
+ let existing = file_metadata
+ .key_value_metadata()
+ .and_then(|kvs| kvs.iter().find(|kv| kv.key == INT96_LEAVES_METADATA_KEY))
+ .and_then(|kv| kv.value.as_deref());
+ if existing == Some(stamp.as_str()) {
+ return None;
+ }
+ let mut key_values: Vec<KeyValue> = file_metadata
+ .key_value_metadata()
+ .map(|kvs| {
+ kvs.iter()
+ .filter(|kv| kv.key != INT96_LEAVES_METADATA_KEY)
+ .cloned()
+ .collect()
+ })
+ .unwrap_or_default();
+ key_values.push(KeyValue::new(INT96_LEAVES_METADATA_KEY.to_string(), stamp));
+ let stamped_file_metadata = FileMetaData::new(
+ file_metadata.version(),
+ file_metadata.num_rows(),
+ file_metadata.created_by().map(str::to_string),
+ Some(key_values),
+ file_metadata.schema_descr_ptr(),
+ file_metadata.column_orders().cloned(),
+ );
+ Some(
+ ParquetMetaData::new(stamped_file_metadata, metadata.row_groups().to_vec())
+ .into_builder()
+ .set_column_index(metadata.column_index().cloned())
+ .set_offset_index(metadata.offset_index().cloned())
+ .build(),
+ )
+}
+
+/// Per-file rebase policies for the three affected column classes, plus the INT96
+/// attribution that selects between the two timestamp specs per leaf.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct FileRebasePolicies {
+ /// `DATE` columns, governed by `org.apache.spark.legacyDateTime` alone.
+ pub date: RebasePolicy,
+ /// INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns, adjusted to UTC or not: the
+ /// datetime spec (same resolution as `date`), as Spark's `ParquetVectorUpdaterFactory`
+ /// selects for INT64 read as `TIMESTAMP`.
+ pub int64_timestamp: RebasePolicy,
+ /// INT96 columns: the INT96 spec (`org.apache.spark.legacyINT96`, min version 3.1.0).
+ pub int96_timestamp: RebasePolicy,
+ /// Which timestamp leaves are INT96. See [`Int96Attribution`].
+ pub int96_leaves: Int96Attribution,
+ /// Sorted depth-first leaf ordinals -- over the physical file schema, the same ordinals
+ /// `int96_leaves` uses -- that the query does not read: nested children the schema
+ /// adapter's struct narrowing drops before any value leaves the scan. Spark never decodes
+ /// them either, so their policy is the identity whatever the file's calendar. Empty until
+ /// [`Self::restrict_to_requested`] runs (every leaf requested).
+ pub unrequested_leaves: Vec<usize>,
+ /// Sorted physical leaf ordinals of timezone-carrying timestamps (INT96, or INT64 with
+ /// `isAdjustedToUTC=true`) the query reads as `TIMESTAMP_NTZ`. Spark decodes those with
+ /// `BinaryToSQLTimestampUpdater` / `LongUpdater`, which never rebase, so their policy is
+ /// the identity whatever the file's calendar. Filled by [`Self::restrict_to_requested`].
+ pub ntz_requested_leaves: Vec<usize>,
+ /// Sorted physical leaf ordinals of timezone-free INT64 timestamps
+ /// (`isAdjustedToUTC=false`) the query reads as `TIMESTAMP`. Spark's INT64 branch checks
+ /// only the unit (`isTimestampTypeMatched`) and hands a `TimestampType` request to
+ /// `LongWithRebaseUpdater` under the datetime spec, adjusted or not, so these leaves take
+ /// the same policy as adjusted INT64 leaves. Filled by [`Self::restrict_to_requested`].
+ pub ltz_requested_leaves: Vec<usize>,
+ /// Sorted physical leaf ordinals of dates and timestamps the query reads as a number
+ /// (`INT`, `BIGINT`, ...). Spark's `ParquetVectorUpdaterFactory` picks the numeric updaters
+ /// before any date or timestamp arm, and none of them rebase, so their policy is the
+ /// identity whatever the file's calendar. Filled by [`Self::restrict_to_requested`].
+ pub numeric_requested_leaves: Vec<usize>,
+}
+
+/// The leaf ordinals [`push_unrequested_leaves`] records while pairing a physical type with
+/// the type the query reads it as. Each list is emitted in depth-first order, so it is already
+/// sorted for the binary searches in [`FileRebasePolicies`].
+#[derive(Debug, Default)]
+struct LeafPairing {
+ unrequested: Vec<usize>,
+ ntz_requested: Vec<usize>,
+ ltz_requested: Vec<usize>,
+ numeric_requested: Vec<usize>,
+}
+
+impl FileRebasePolicies {
+ /// True when some policy is not the plain proleptic-Gregorian pass-through, i.e. when the
+ /// per-column wrap in [`wrap_datetime_rebase`] can install anything at all.
+ pub(crate) fn any_rebase_needed(&self) -> bool {
+ self.date != RebasePolicy::Corrected
+ || self.int64_timestamp != RebasePolicy::Corrected
+ || self.int96_timestamp != RebasePolicy::Corrected
+ }
+
+ /// True when the query reads the leaf as a date or timestamp: it is requested, and not
+ /// as a number.
+ fn is_requested(&self, leaf: usize) -> bool {
+ self.unrequested_leaves.binary_search(&leaf).is_err()
+ && self.numeric_requested_leaves.binary_search(&leaf).is_err()
+ }
+
+ /// The policy of the `Date32` leaf at depth-first ordinal `leaf`: the file's date policy,
+ /// or the identity when the query does not read that leaf.
+ fn date_policy(&self, leaf: usize) -> RebasePolicy {
+ if self.is_requested(leaf) {
+ self.date
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+
+ /// The policy of the timezone-carrying timestamp leaf at depth-first ordinal `leaf`: the
+ /// identity when the query does not read it or reads it as `TIMESTAMP_NTZ` (Spark's NTZ
+ /// updaters never rebase; on Spark 3.x the pairing is refused before any rebase decision,
+ /// which Comet's `allow_timestamp_ltz_to_ntz` gate reproduces); otherwise its physical
+ /// type's spec when the attribution is known, or else the two specs merged -- agreement
+ /// decides, disagreement degrades to [`RebasePolicy::CheckAncient`], which still passes
+ /// every modern value and refuses only ancient ones.
+ fn timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) || self.ntz_requested_leaves.binary_search(&leaf).is_ok() {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ Some(false) => self.int64_timestamp,
+ None if self.int64_timestamp == self.int96_timestamp => self.int64_timestamp,
+ None => RebasePolicy::CheckAncient,
+ }
+ }
+
+ /// The policy of the timezone-free timestamp leaf at depth-first ordinal `leaf` (INT64 with
+ /// `isAdjustedToUTC=false`): the identity unless the query reads it as `TIMESTAMP`, which
+ /// Spark decodes with `LongWithRebaseUpdater` under the datetime spec exactly like an
+ /// adjusted INT64 leaf. The stamp is still consulted so a leaf it names INT96 follows the
+ /// INT96 spec; without a stamp the physical type itself proves INT64, so the two specs are
+ /// not merged.
+ fn tz_free_timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) || self.ltz_requested_leaves.binary_search(&leaf).is_err() {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ _ => self.int64_timestamp,
+ }
+ }
+
+ /// These policies with every physical leaf the query does not read marked the identity,
+ /// and every timestamp leaf whose requested type differs from its physical one in timezone
+ /// presence recorded, so [`leaf_policies`] can pick the policy Spark's
+ /// `ParquetVectorUpdaterFactory.getUpdater` picks for the REQUESTED type.
+ /// `requested` pairs each top-level field of `physical_schema` (by position) with the type
+ /// of the logical field the schema adapter narrows it to -- `None` for a column without a
+ /// logical counterpart, whose leaves are left as they are (no expression reads it anyway).
+ /// Nested children pair the way the adapter's struct convert selects them (see
+ /// [`push_unrequested_leaves`]); the INT96 attribution is untouched, since the ordinals
+ /// stay physical. `requested` is parallel to the schema's fields; should a caller pass a
+ /// shorter slice, the trailing columns simply keep every leaf (the safe direction).
+ pub(crate) fn restrict_to_requested(
+ mut self,
+ physical_schema: &Schema,
+ requested: &[Option<&DataType>],
+ case_sensitive: bool,
+ use_field_id: bool,
+ ) -> Self {
+ debug_assert_eq!(requested.len(), physical_schema.fields().len());
+ let matching = FieldMatching {
+ case_sensitive,
+ use_field_id,
+ };
+ let mut next_leaf = 0;
+ let mut pairing = LeafPairing::default();
+ for (field, requested) in physical_schema.fields().iter().zip(requested) {
+ match requested {
+ Some(logical) => push_unrequested_leaves(
+ field.data_type(),
+ logical,
+ &mut next_leaf,
+ matching,
+ &mut pairing,
+ ),
+ None => next_leaf += leaf_count(field.data_type()),
+ }
+ }
+ // Emitted in depth-first order, so already sorted for the binary searches.
+ self.unrequested_leaves = pairing.unrequested;
+ self.ntz_requested_leaves = pairing.ntz_requested;
+ self.ltz_requested_leaves = pairing.ltz_requested;
+ self.numeric_requested_leaves = pairing.numeric_requested;
+ self
+ }
+}
+
+/// The field-matching rules of the schema adapter's nested narrowing
+/// (`parquet_convert_struct_to_struct`): names fold per `case_sensitive`, and Parquet field ids
+/// select fields when `use_field_id` is set.
+#[derive(Debug, Clone, Copy)]
+struct FieldMatching {
+ case_sensitive: bool,
+ use_field_id: bool,
+}
+
+/// Appends to `out.unrequested` the depth-first leaf ordinals of `physical` (counting from
+/// `next_leaf`, which advances past every leaf of `physical`) that reading it as `requested`
+/// drops, and records in `out.ntz_requested` / `out.ltz_requested` the timestamp leaves whose
+/// requested type has the opposite timezone presence (a timezone-carrying leaf read as
+/// `TIMESTAMP_NTZ`, a timezone-free leaf read as `TIMESTAMP`); the unit is irrelevant to
+/// either, as it is to Spark's `isTimestampTypeMatched`. A date leaf read as `TINYINT`,
+/// `SMALLINT`, `INT`, `BIGINT` or `DOUBLE`, and a timestamp leaf read as `BIGINT`, go to
+/// `out.numeric_requested`: Spark's INT32 and INT64 numeric updaters for those pairings never
+/// rebase. Spark accepts a date as `BIGINT` or `DOUBLE` only on 4.x (3.x refuses the read);
+/// the identity is still the right policy for it.
+///
+/// Recurses through exactly the pairings `parquet_convert_array` narrows, and no others: a
+/// struct child is dropped only when NO requested child selects it by either rule the struct
+/// convert uses -- folded name, or Parquet field id when ids are in play -- and an ambiguous
+/// child (several requested children select it) is kept; `List` pairs with `List` by element
+/// type, and `Map` with a `Map` of the same key ordering by its entries, positionally. Any
+/// other pairing -- a `LargeList` / `FixedSizeList` / dictionary, a map whose ordering
+/// differs, or a shape mismatch -- is handed to arrow's cast or passed through whole by the
+/// convert, so it keeps every leaf under its physical type's policy. Keeping a superset of
+/// what the narrowing reads is always safe (a spurious check at worst); dropping a leaf the
+/// narrowing reads would skip its rebase, so every doubt resolves to "requested". Timestamp
+/// leaves inside those pass-through shapes are never recorded either, so they keep the
+/// physical rule (a spurious check for an NTZ request, no rebase for a `TIMESTAMP` request of
+/// a timezone-free leaf); Spark's requested schemas never take those arrow shapes.
+fn push_unrequested_leaves(
+ physical: &DataType,
+ requested: &DataType,
+ next_leaf: &mut usize,
+ matching: FieldMatching,
+ out: &mut LeafPairing,
+) {
+ match (physical, requested) {
+ (DataType::Timestamp(_, Some(_)), DataType::Timestamp(_, None)) => {
+ out.ntz_requested.push(*next_leaf);
+ *next_leaf += 1;
+ }
+ (DataType::Timestamp(_, None), DataType::Timestamp(_, Some(_))) => {
+ out.ltz_requested.push(*next_leaf);
+ *next_leaf += 1;
+ }
+ (
+ DataType::Date32,
+ DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Int64
+ | DataType::Float64,
+ )
+ | (DataType::Timestamp(_, _), DataType::Int64) => {
+ out.numeric_requested.push(*next_leaf);
+ *next_leaf += 1;
+ }
+ (DataType::Struct(physical_fields), DataType::Struct(requested_fields)) => {
+ let names: Vec<&str> = physical_fields
+ .iter()
+ .chain(requested_fields.iter())
+ .map(|f| f.name().as_str())
+ .collect();
+ // A fold failure means the names could not be compared at all; keeping every leaf
+ // requested is the safe superset, the same as the pass-through pairings below.
+ let Ok(folded) = fold_names(&names, matching.case_sensitive) else {
+ *next_leaf += leaf_count(physical);
+ return;
+ };
+ let (physical_folded, requested_folded) = folded.split_at(physical_fields.len());
+ for (i, child) in physical_fields.iter().enumerate() {
+ let child_id = if matching.use_field_id {
+ field_id(child)
+ } else {
+ None
+ };
+ let mut selectors = requested_fields.iter().enumerate().filter(|(j, r)| {
+ requested_folded[*j] == physical_folded[i]
+ || (child_id.is_some() && field_id(r) == child_id)
+ });
+ match (selectors.next(), selectors.next()) {
+ (None, _) => {
+ let n = leaf_count(child.data_type());
+ out.unrequested.extend(*next_leaf..*next_leaf + n);
+ *next_leaf += n;
+ }
+ (Some((_, requested_child)), None) => push_unrequested_leaves(
+ child.data_type(),
+ requested_child.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (Some(_), Some(_)) => *next_leaf += leaf_count(child.data_type()),
+ }
+ }
+ }
+ (DataType::List(physical_item), DataType::List(requested_item)) => push_unrequested_leaves(
+ physical_item.data_type(),
+ requested_item.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (
+ DataType::Map(physical_entries, physical_sorted),
+ DataType::Map(requested_entries, requested_sorted),
+ ) if physical_sorted == requested_sorted => {
+ match (physical_entries.data_type(), requested_entries.data_type()) {
+ (DataType::Struct(physical_kv), DataType::Struct(requested_kv))
+ if physical_kv.len() == requested_kv.len() =>
+ {
+ for (p, r) in physical_kv.iter().zip(requested_kv.iter()) {
+ push_unrequested_leaves(
+ p.data_type(),
+ r.data_type(),
+ next_leaf,
+ matching,
+ out,
+ );
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+}
+
+/// The writer time zone recorded in `metadata`, classified for timestamp rebasing. Mirrors the
+/// `Option(lookupFileMeta(SPARK_TIMEZONE_METADATA_KEY))` lookup Spark's `getRebaseSpec` performs
+/// for every LEGACY resolution, conf-fallback included; Spark substitutes the JVM default zone
+/// when the key is absent (`RebaseSpec.timeZone`), which is unavailable natively, so an absent or
+/// non-UTC zone classifies as [`WriterTimeZone::OtherOrUnknown`] (dates still rebase fully --
+/// the day rebase is zone-free -- while ancient timestamps refuse rather than guess).
+fn writer_time_zone(metadata: &HashMap<String, String>) -> WriterTimeZone {
+ match metadata.get(SPARK_TIMEZONE_KEY) {
+ Some(tz) if UTC_EQUIVALENT_TIMEZONES.contains(&tz.as_str()) => WriterTimeZone::Utc,
+ _ => WriterTimeZone::OtherOrUnknown,
+ }
+}
+
+/// One spec resolution, mirroring Spark's `DataSourceUtils.getRebaseSpec` exactly: a Spark
+/// version below `min_version` (lexicographic comparison, same as the Scala `String.<`) or a
+/// present legacy flag means LEGACY; a Spark version at/after `min_version` without the flag
+/// means CORRECTED; no Spark version at all falls back to `conf_mode`, the session read conf
+/// forwarded from the JVM (`getRebaseSpec`'s `modeByConfig` fallback, its ONLY use of the
+/// conf): CORRECTED passes values through, LEGACY rebases (with the writer zone from the
+/// file's `org.apache.spark.timeZone` key, same lookup as the metadata-driven LEGACY path),
+/// and EXCEPTION refuses ancient values as [`RebasePolicy::CheckAncient`].
+fn resolve_spec(
+ metadata: &HashMap<String, String>,
+ min_version: &str,
+ legacy_key: &str,
+ conf_mode: RebaseReadMode,
+) -> RebasePolicy {
+ match metadata.get(SPARK_VERSION_METADATA_KEY) {
+ None => match conf_mode {
+ RebaseReadMode::Corrected => RebasePolicy::Corrected,
+ RebaseReadMode::Legacy => RebasePolicy::Legacy(writer_time_zone(metadata)),
+ RebaseReadMode::Exception => RebasePolicy::CheckAncient,
+ },
+ Some(version) => {
+ if version.as_str() < min_version || metadata.contains_key(legacy_key) {
+ RebasePolicy::Legacy(writer_time_zone(metadata))
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+ }
+}
+
+/// Resolves the per-file rebase policies from a file's arrow schema: the parquet footer's
+/// key-value pairs in its metadata decide the specs (the datetime spec uses min version
+/// `3.0.0` and the INT96 spec `3.1.0`, matching `DataSourceUtils.datetimeRebaseSpec` /
+/// `int96RebaseSpec`; `session_modes` supplies the per-spec conf fallback for files without
+/// Spark writer metadata), and the reader factory's INT96 stamp -- validated against the
+/// schema's leaf structure -- attributes each timestamp leaf to its spec.
+pub(crate) fn resolve_file_rebase_policies(
+ physical_file_schema: &Schema,
+ session_modes: SessionRebaseModes,
+) -> FileRebasePolicies {
+ let metadata = physical_file_schema.metadata();
+ let datetime_spec = resolve_spec(
+ metadata,
+ "3.0.0",
+ SPARK_LEGACY_DATETIME_KEY,
+ session_modes.datetime,
+ );
+ let int96_spec = resolve_spec(
+ metadata,
+ "3.1.0",
+ SPARK_LEGACY_INT96_KEY,
+ session_modes.int96,
+ );
+ FileRebasePolicies {
+ date: datetime_spec,
+ int64_timestamp: datetime_spec,
+ int96_timestamp: int96_spec,
+ int96_leaves: Int96Attribution::from_schema(physical_file_schema),
+ unrequested_leaves: Vec::new(),
+ ntz_requested_leaves: Vec::new(),
+ ltz_requested_leaves: Vec::new(),
+ numeric_requested_leaves: Vec::new(),
+ }
+}
+
+/// Number of primitive leaves `dt` contains in a depth-first walk -- the same count and order
+/// parquet-rs uses when it maps the file's `SchemaDescriptor` columns onto the arrow schema, so
+/// arrow-side leaf ordinals line up with [`int96_leaf_stamp`]'s.
+fn leaf_count(dt: &DataType) -> usize {
+ match dt {
+ DataType::Struct(fields) => fields.iter().map(|f| leaf_count(f.data_type())).sum(),
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_count(f.data_type()),
+ DataType::Dictionary(_, value) => leaf_count(value),
+ DataType::RunEndEncoded(_, value) => leaf_count(value.data_type()),
+ DataType::Union(fields, _) => fields.iter().map(|(_, f)| leaf_count(f.data_type())).sum(),
+ _ => 1,
+ }
+}
+
+/// Appends the policy of every leaf of `dt`, in depth-first order, to `out`, consuming leaf
+/// ordinals from `next_leaf` (exactly [`leaf_count`] of them). Only `Date32` and timestamps
+/// have a policy to apply, and only when the query reads the leaf; a timestamp leaf's policy
+/// follows the type the query reads it as (see [`FileRebasePolicies::timestamp_policy`] and
+/// [`FileRebasePolicies::tz_free_timestamp_policy`]), and every other leaf is the identity
+/// ([`RebasePolicy::Corrected`]).
+fn leaf_policies(
+ dt: &DataType,
+ next_leaf: &mut usize,
+ policies: &FileRebasePolicies,
+ out: &mut Vec<RebasePolicy>,
+) {
+ match dt {
+ DataType::Date32 => {
+ out.push(policies.date_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Timestamp(_, Some(_)) => {
+ out.push(policies.timestamp_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Timestamp(_, None) => {
+ out.push(policies.tz_free_timestamp_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Struct(fields) => {
+ for f in fields {
+ leaf_policies(f.data_type(), next_leaf, policies, out);
+ }
+ }
+ // Mirrors `leaf_count` variant for variant, so a rebase-affected leaf inside a nested
+ // type `rebase_array` cannot rebuild (views, run-end, union -- never produced from a
+ // parquet schema) still gets its real policy and makes `rebase_array` refuse loudly
+ // instead of being stamped the identity.
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_policies(f.data_type(), next_leaf, policies, out),
+ DataType::Dictionary(_, value) => leaf_policies(value, next_leaf, policies, out),
+ DataType::RunEndEncoded(_, value) => {
+ leaf_policies(value.data_type(), next_leaf, policies, out)
+ }
+ DataType::Union(fields, _) => {
+ for (_, f) in fields.iter() {
+ leaf_policies(f.data_type(), next_leaf, policies, out);
+ }
+ }
+ _ => {
+ *next_leaf += 1;
+ out.push(RebasePolicy::Corrected);
Review Comment:
[P2] Apply the requested DATE policy to supported INT32-backed columns. This catch-all marks unannotated `INT32` as `Corrected`, even when the requested type is `Date32`, so `wrap_datetime_rebase` skips it before the schema adapter casts it to DATE. A Parquet catalog table `(v DATE)` backed by metadata-free `message m { required int32 v; }` containing `-171655`, converted using `CONVERT TO DELTA ... NO STATISTICS`, reaches the new native path. With `datetimeRebaseModeInRead=LEGACY`, Spark returns `1500-01-01`, but native returns `1500-01-10`. Under `EXCEPTION`, Spark raises `SparkUpgradeException`, while native returns the row. This silently changes dates in newly admitted Delta reads. Please carry the requested-DATE policy through this pairing and handle the integer storage, or conservatively decline it, with a regression covering both modes.
Evidence: Fresh reproduction used an otherwise unchanged `git archive` of the reviewed head at `/tmp/recheck5365-41e-native-dzu0d4hq`. The disposable `review_int32_as_date_calendar` test writes unannotated INT32 value `-171655` and calls current `init_datasource_exec` with requested `Date32` and metadata rebasing enabled. CORRECTED, LEGACY and EXCEPTION all returned `-171655` (1500-01-10). Log: `/tmp/recheck5365-41e-raw-int32-native.log`. Stock Spark 3.5.9/Delta 3.3.2 converted the same file through a catalog table declared `(v DATE)`: CORRECTED returned 1500-01-10, LEGACY returned 1500-01-01, and EXCEPTION raised SparkUpgradeException. Assertions passed in `/tmp/recheck5365-41e-raw-int32-spark.py`; output is `/tmp/recheck5365-41e-raw-int32-spark.log`. Official Spark v3.5.9/v4.0.4/v4.1.3 `ParquetVectorUpdaterFactory` sources select `IntegerWithRebaseUpdater` from requested `DateType` without requiring a DATE annotation. Current Delta admission accepts the declared DATE schema, and
`delta_spark_scan.rs` enables the exercised rebase path. Full Comet/JNI execution was not performed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3905341388
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1627 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Zero-I/O in the common case, like the encryption gate above: native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The three keys where Hadoop's OWN resolver (`S3AUtils#lookupPassword`) is long-bucket-aware,
+ * verified via `javap` against `hadoop-aws` 3.3.4: `S3AUtils#lookupPassword(bucket, conf,
+ * baseKey)` builds `longBucketKey = "fs.s3a.bucket." + bucket + "." + baseKey` (note: the FULL
+ * `baseKey`, already `fs.s3a`-prefixed, appended after the bucket segment) and reads it via
+ * `getPassword` BEFORE the short-bucket key, keeping the long value whenever `getPassword`
+ * returns non-empty for it and only falling through to short-then-global otherwise. Kept as a
+ * named subset purely for documentation/discovery purposes (see
+ * [[hadoopLookupPasswordEffective]]'s doc); every key in [[AllS3ConfigKeys]] below is resolved
+ * through the SAME long-then-short-then-global function regardless of membership here -- see
+ * that function's doc for why extending long-bucket-awareness to keys where it does not
+ * strictly apply is safe.
+ */
+ private val CredentialValueKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ /**
+ * Every `fs.s3a.*` base key that governs whether a claimed native scan actually behaves like
+ * Hadoop's own reader would, checked by [[s3ConfigDivergenceReason]] via
+ * [[hadoopLookupPasswordEffective]] (see that function's doc for the resolution order and the
+ * deliberate over-decline rationale for the non-credential entries below). Two groups, kept
+ * merged into one list and one resolution path -- NOT two lists with two different resolution
+ * mechanisms, which is what let `fs.s3a.encryption.algorithm` silently fall through the
+ * short-then-global-only path while Hadoop's real resolver for it was long-bucket-aware (the
+ * SSE-C long-bucket-alias gap): the ORIGINAL [[CredentialValueKeys]] three, plus every OTHER
+ * per-bucket `fs.s3a.*` base key native's S3 client's `get_config` (s3.rs) resolves, verified
+ * directly against its call sites: `extract_s3_config_options` (endpoint.region,
+ * path.style.access, endpoint, requester.pays.enabled), `lookup_provider_class` (the
+ * Comet-specific credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name).
+ *
+ * SYNC NOTE: this list's non-credential entries must stay a superset of native's
+ * `NATIVE_S3A_CONFIG_PROPERTIES` constant (`native/core/src/parquet/objectstore/s3.rs`,
+ * property suffixes without the `fs.s3a.` prefix) -- `DeltaScanContribSuite`'s
+ * discovery-harness test asserts this mechanically. Literal strings, not the
+ * [[AwsCredentialsProviderKey]] / [[AssumedRoleCredentialsProviderKey]] vals declared below,
+ * purely to avoid a forward reference inside this `object` body; kept textually identical to
+ * those two constants.
+ */
+ private[delta] val AllS3ConfigKeys: Seq[String] = Seq(
+ "fs.s3a.access.key",
+ "fs.s3a.secret.key",
+ "fs.s3a.session.token",
+ "fs.s3a.aws.credentials.provider",
+ "fs.s3a.assumed.role.arn",
+ "fs.s3a.assumed.role.session.name",
+ "fs.s3a.assumed.role.credentials.provider",
+ "fs.s3a.endpoint",
+ "fs.s3a.endpoint.region",
+ "fs.s3a.path.style.access",
+ "fs.s3a.requester.pays.enabled",
+ "fs.s3a.comet.credential.provider.class")
+
+ /**
+ * The short-bucket-then-global value resolved for `baseKey` under `bucket` from `hadoopConf`,
+ * skipping an empty value at either alias exactly like [[plainValue]]. NOT used by
+ * [[s3ConfigDivergenceReason]]/[[s3KeyDivergenceReason]] any more -- every key checked there
+ * resolves through [[hadoopLookupPasswordEffective]] uniformly (long-then-short-then-global),
+ * which is a strict superset of what this function reads. This function's one remaining caller
+ * is [[shortThenGlobalOrReason]], which reads provider-CLASS strings (from the ORIGINAL,
+ * unpropagated conf) for name-support validation in [[providerClassReason]]/
+ * [[assumedRoleProviderClassReason]] -- by the time those run, [[s3ConfigDivergenceReason]] has
+ * already proven Hadoop's and native's effective values agree for the same key, so whichever of
+ * the two (equal) values this narrower read returns does not affect correctness there. NEVER
+ * used to compute native's own effective value -- see [[nativeShortThenGlobal]] for that.
+ */
+ private def shortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket` from the ORIGINAL, unpropagated `hadoopConf` -- `NativeConfig
+ * .extractObjectStoreOptions` forwards `Configuration#get`'s substituted value for every
+ * `fs.s3a.*` entry with no bucket-option propagation step of its own, so the original conf is
+ * the right input here. Unlike [[shortThenGlobal]]/[[plainValue]], this mirrors `get_config`
+ * faithfully: PRESENCE of the short-bucket key alone -- never its emptiness -- decides whether
+ * native falls back to the global key (`get_config` is a plain `HashMap::get`, which returns
+ * `Some` for a key explicitly set to `""`), so an explicitly empty or whitespace-only
+ * short-bucket value resolves to `Some("")` here and never falls through to global -- the
+ * OPPOSITE of Hadoop's own `getPassword`/`lookupPassword` semantics (see
+ * [[hadoopLookupPasswordEffective]]), which treat empty as absent and keep trying the next
+ * alias. The ONLY function used to compute native's effective value in
+ * [[s3KeyDivergenceReason]].
+ *
+ * Deliberately does NOT apply `get_config_trimmed`'s `.trim()` here: [[s3KeyDivergenceReason]]
+ * trims both this value and Hadoop's effective value together, symmetrically, at the point they
+ * are compared, rather than one-sidedly here -- trimming only the native side would flag a
+ * spurious divergence for a value neither side's whitespace actually changes the behavior of
+ * once each side's own downstream parsing normalizes it (e.g. Hadoop's own multi-line
+ * `fs.s3a.aws.credentials.provider` default, which both Hadoop and native additionally trim per
+ * comma-separated entry after splitting), while a one-sided trim would make an
+ * otherwise-identical default value look diverged for every bucket, never claiming natively at
+ * all.
+ */
+ private def nativeShortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ Option(hadoopConf.get(shortKey)).orElse(Option(hadoopConf.get(baseKey)))
+ }
+
+ /**
+ * Faithful in-memory replica of `S3AUtils#propagateBucketOptions` (`hadoop-aws`), which
+ * `S3AFileSystem#initialize` calls FIRST, before any option or credential is read:
+ * `Configuration conf = propagateBucketOptions(originalConf, bucket); ...; setConf(conf);` --
+ * every subsequent `conf.get`/`getPassword` call in that filesystem instance, including
+ * `${...}` variable substitution, resolves against this propagated view, not the original conf.
+ * `hadoop-aws` is not on this module's runtime classpath (see the string-literal-keys note
+ * above), so `S3AUtils#propagateBucketOptions` cannot be called directly; this reproduces its
+ * logic verbatim using only `hadoop-common`'s `Configuration`:
+ * {{{
+ * public static Configuration propagateBucketOptions(Configuration source, String bucket) {
+ * final String bucketPrefix = FS_S3A_BUCKET_PREFIX + bucket + '.';
+ * final Configuration dest = new Configuration(source);
+ * for (Map.Entry<String, String> entry : source) {
+ * final String key = entry.getKey();
+ * final String value = entry.getValue(); // the (unexpanded) value
+ * if (!key.startsWith(bucketPrefix) || bucketPrefix.equals(key)) continue;
+ * final String stripped = key.substring(bucketPrefix.length());
+ * if (stripped.startsWith("bucket.") || "impl".equals(stripped)) {
+ * // ignored
+ * } else {
+ * final String generic = FS_S3A_PREFIX + stripped;
+ * dest.set(generic, value, ...); // overwrites any existing global value
+ * }
+ * }
+ * return dest;
+ * }
+ * }}}
+ * Note the LONG bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) folds to an unread
+ * `fs.s3a.fs.s3a.<key>` key here too, exactly like the real method -- `stripped` already starts
+ * with `fs.s3a.` in that case, so prepending `fs.s3a.` again produces a key nothing ever reads.
+ */
+ private def propagateBucketOptions(hadoopConf: Configuration, bucket: String): Configuration = {
+ val bucketPrefix = s"fs.s3a.bucket.$bucket."
+ val dest = new Configuration(hadoopConf)
+ hadoopConf.iterator().asScala.foreach { entry =>
+ val key = entry.getKey
+ if (key.startsWith(bucketPrefix) && key != bucketPrefix) {
+ val stripped = key.substring(bucketPrefix.length)
+ if (!stripped.startsWith("bucket.") && stripped != "impl") {
+ dest.set(s"fs.s3a.$stripped", entry.getValue)
+ }
+ }
+ }
+ dest
+ }
+
+ /**
+ * Canonical and deprecated Hadoop S3A encryption-algorithm config keys, verified via `javap`
+ * against `hadoop-aws` 3.3.4's `org.apache.hadoop.fs.s3a.Constants`: `S3_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.encryption.algorithm"` (canonical) and `SERVER_SIDE_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.server-side-encryption-algorithm"` (DEPRECATED -- note the hyphen before "algorithm",
+ * unlike the corresponding `*.key` constants below, which both use a `.key` suffix).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, same
+ * rationale as [[HadoopCredentialProviderPathKey]].
+ */
+ private val S3EncryptionAlgorithmKey = "fs.s3a.encryption.algorithm"
+ private val DeprecatedS3EncryptionAlgorithmKey = "fs.s3a.server-side-encryption-algorithm"
+
+ /**
+ * The exact strings `S3AEncryptionMethods#getMethod` accepts, verified via `javap`/CFR against
+ * `hadoop-aws` 3.3.4's `S3AEncryptionMethods` enum: `NONE("")`, `SSE_S3("AES256", serverSide =
+ * true, requiresSecret = false)`, `SSE_KMS("SSE-KMS", serverSide = true, requiresSecret =
+ * false)`, `SSE_C("SSE-C", serverSide = true, requiresSecret = true)`, `CSE_KMS("CSE-KMS",
+ * serverSide = false, requiresSecret = true)`, `CSE_CUSTOM("CSE-CUSTOM", serverSide = false,
+ * requiresSecret = true)`. `getMethod` parses case-insensitively
+ * (`values().find(_.getMethod.equalsIgnoreCase(algorithm))`), matched below the same way.
+ *
+ * ALLOWLIST, not a blocklist (replaces the former SSE-C-only blocklist): only the algorithms S3
+ * decrypts transparently on GET/HEAD given read permission alone, with NO extra request header
+ * and NO client-side step, are safe for a native scan that forwards none of Hadoop's
+ * `fs.s3a.encryption.*`/`fs.s3a.server-side-encryption*` options --
+ * - `AES256` (SSE_S3, `serverSide = true`): plain server-side encryption, transparent on GET.
+ * - `SSE-KMS` (SSE_KMS, `serverSide = true`): server-side, KMS-managed key, transparent on
+ * GET given KMS decrypt permission (no header).
+ * - `DSSE-KMS`: NOT present in this enum on `hadoop-aws` 3.3.4 (confirmed by the six values
+ * listed above) -- `S3AEncryptionMethods.getMethod("DSSE-KMS")` throws
+ * `IOException("Unknown encryption algorithm DSSE-KMS")` on this version, so
+ * `S3AUtils#buildEncryptionSecrets` (and therefore Hadoop's own reader) already fails
+ * before ever reading such a table under 3.3.4, meaning this string can never actually be
+ * the resolved value on the declared target version -- admitting it here is inert there.
+ * Included anyway, forward-compatible, for a newer `hadoop-aws` on the runtime classpath (a
+ * later Hadoop release; this module has no compile-time `hadoop-aws` dependency, see the
+ * string-literal-keys note above) where DSSE-KMS is a real, dual-layer, server-side
+ * algorithm decrypted transparently on GET the same way SSE-KMS is. Every other value
+ * declines: `SSE-C` (SSE_C is `serverSide = true` in Hadoop's own enum, but `requiresSecret
+ * \= true` -- S3 rejects a GET/HEAD for an SSE-C object outright (400 Bad Request) unless
+ * the customer key is resent as a request header on every call, so a native scan that never
+ * learns the key cannot succeed at all, where Hadoop's own reader -- whose request factory
+ * attaches the key -- would), `CSE-KMS`/`CSE-CUSTOM` (`serverSide = false`: client-side
+ * encryption decrypts object bytes locally in the SDK layer, which the native Parquet
+ * reader has no equivalent of -- it would read raw ciphertext), and any future/unknown
+ * value (a value `S3AEncryptionMethods.getMethod` itself would reject is certainly not one
+ * of the three confirmed-transparent algorithms above; declining is the only safe default
+ * for anything this gate cannot positively confirm).
+ */
+ private val AllowedEncryptionAlgorithms: Set[String] = Set("AES256", "SSE-KMS", "DSSE-KMS")
+
+ /**
+ * `bucket`'s effective encryption-algorithm key and value under `hadoopConf`, or `None` when
+ * neither the canonical nor deprecated key is set anywhere consulted. Mirrors
+ * `S3AUtils#buildEncryptionSecrets`'s real resolution order, verified via `javap`/CFR
+ * decompilation of `hadoop-aws` 3.3.4's `S3AUtils.class`:
+ * {{{
+ * String algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.server-side-encryption-algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.server-side-encryption-algorithm");
+ * }}}
+ * i.e. bucket-tier (canonical, then deprecated), THEN global-tier (canonical, then deprecated)
+ * -- the two tiers are never interleaved key-by-key, so this must stay two explicit bucket-tier
+ * lookups followed by two explicit global-tier lookups, not a single
+ * [[hadoopLookupPasswordEffective]] call per key (which would let an unset canonical bucket key
+ * fall through straight to the canonical GLOBAL value ahead of a SET deprecated bucket key, the
+ * wrong answer).
+ *
+ * THE FIX for the SSE-C long-bucket-alias gap is entirely inside the bucket tier:
+ * `lookupBucketSecret` itself is long-then-short, decompiled from `hadoop-aws` 3.3.4's
+ * `S3AUtils.class`:
+ * {{{
+ * // longBucketKey = fs.s3a.bucket.B.fs.s3a.<key>
+ * String longBucketKey = String.format(BUCKET_PATTERN, bucket, baseKey);
+ * String initialVal = getPassword(conf, longBucketKey, null, null);
+ * // shortBucketKey = fs.s3a.bucket.B.<key>
+ * String shortBucketKey = String.format(BUCKET_PATTERN, bucket, subkey);
+ * // keeps initialVal (the LONG value) if non-empty
+ * return getPassword(conf, shortBucketKey, initialVal, null);
+ * }}}
+ * i.e. the SAME long-bucket-key construction and long-wins-if-nonempty semantics as
+ * `S3AUtils#lookupPassword` (see [[CredentialValueKeys]]/[[hadoopLookupPasswordEffective]]) --
+ * the encryption algorithm is NOT one of the keys that flows through
+ * `S3AUtils#propagateBucketOptions` (which folds an unrelated per-bucket LONG form into an
+ * unread key). An earlier version of this function modeled the bucket tier as SHORT-only,
+ * documented as "the LONG bucket form is genuinely never consulted for this key" -- that
+ * documentation was wrong (this decompilation supersedes it): a bucket configured only via
+ * `fs.s3a.bucket.B.fs.s3a.encryption.algorithm=SSE-C` bypassed the SSE-C gate entirely, because
+ * Hadoop's own reader DOES read that long form (and picks SSE-C), while this function reported
+ * `None` (nothing set) and the allowlist check below never even ran.
+ *
+ * The canonical-vs-deprecated distinction below is frequently moot in practice: `hadoop-aws`'s
+ * `S3AFileSystem.addDeprecatedKeys()` statically registers `fs.s3a.server-side-encryption-*` as
+ * `Configuration`-level deprecated aliases of `fs.s3a.encryption.*` (verified via `javap`), a
+ * registration that lives in a static field on Hadoop's `Configuration` class -- process-wide
+ * once `S3AFileSystem`'s class has loaded anywhere in the JVM, which a real scan has always
+ * already done by the time this gate runs, since reading the S3 table at all requires loading
+ * that class. Once active, `Configuration#get` resolves either literal key to the identical
+ * value transparently, making the two-key cascade below redundant (but harmless) for that case;
+ * it remains the operative path only when nothing else in the process has loaded
+ * `S3AFileSystem` yet.
+ *
+ * ALSO walks the Hadoop-credential-provider (JCEKS) path via [[resolveViaCredentialAliases]]
+ * for each of the four lookups below, matching `lookupBucketSecret`/`lookupPassword`'s real
+ * per-alias `getPassword` calls (quoted above) exactly: both are `getPassword`, not plain
+ * `Configuration#get`, so a bucket storing the algorithm name ONLY in a JCEKS keystore is
+ * exactly as real a Hadoop deployment shape for this key as it is for the credential keys
+ * [[hadoopLookupPasswordEffective]] already covers -- there is nothing algorithm-specific that
+ * makes JCEKS storage implausible here, so an earlier version of this function skipping it
+ * (documented at the time as "the algorithm NAME is not credential-sensitive data, so storing
+ * it in a keystore is not a realistic Hadoop deployment pattern") was an unjustified, narrower
+ * read than Hadoop's own resolver actually performs, under-declining a bucket whose algorithm
+ * is keystore-only. [[resolveViaCredentialAliases]]'s Arm B/C split still means this is zero
+ * extra I/O for the common case: keystore I/O only happens when a Hadoop credential-provider
+ * path is actually configured for the bucket, contained in that function's own try/catch.
+ * `bucketTier`/`globalTier` return `Left` (propagated straight through by [[orElseTier]]) when
+ * [[resolveViaCredentialAliases]] cannot safely verify a tier at all (an S3A-scoped provider
+ * path, or a corrupt/unreadable global keystore) -- correctly short-circuiting the whole
+ * cascade with a decline rather than silently falling through to a later tier that might look
+ * unset only because the true value was unverifiable.
+ */
+ private def effectiveEncryptionAlgorithm(
+ hadoopConf: Configuration,
+ bucket: String): Either[String, Option[(String, String)]] = {
+ def bucketTier(baseKey: String): Either[String, Option[(String, String)]] = {
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(longKey, shortKey))
+ .map(_.map(baseKey -> _))
+ }
+ def globalTier(baseKey: String): Either[String, Option[(String, String)]] =
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(baseKey)).map(_.map(baseKey -> _))
+
+ // Short-circuits on Left (unverifiable tier) or Right(Some(_)) (resolved); only Right(None)
+ // (tier definitively unset) falls through to `next`, mirroring buildEncryptionSecrets's
+ // sequential `if (algorithm == null) algorithm = ...` cascade exactly.
+ def orElseTier(
+ current: Either[String, Option[(String, String)]],
+ next: => Either[String, Option[(String, String)]])
+ : Either[String, Option[(String, String)]] =
+ current match {
+ case Left(reason) => Left(reason)
+ case Right(Some(value)) => Right(Some(value))
+ case Right(None) => next
+ }
+
+ orElseTier(
+ bucketTier(S3EncryptionAlgorithmKey),
+ orElseTier(
+ bucketTier(DeprecatedS3EncryptionAlgorithmKey),
+ orElseTier(
+ globalTier(S3EncryptionAlgorithmKey),
+ globalTier(DeprecatedS3EncryptionAlgorithmKey))))
+ }
+
+ private def unsupportedEncryptionAlgorithmDeclineReason(
+ bucket: String,
+ algorithmKey: String,
+ algorithm: String): String =
+ s"Native Delta scan does not support $algorithmKey=$algorithm for $bucket " +
+ "(the native S3 client only supports unencrypted objects and S3's transparent " +
+ "server-side algorithms -- AES256/SSE-S3, SSE-KMS, and DSSE-KMS decrypt on GET/HEAD given " +
+ "read permission alone, with no extra request header; SSE-C additionally requires the " +
+ "customer-provided key resent as a header on every GET/HEAD request, which the native S3 " +
+ "client's extract_s3_config_options never forwards, and CSE-KMS/CSE-CUSTOM decrypt object " +
+ "bytes client-side, a layer the native Parquet reader does not have -- any of these would " +
+ "fail outright or silently read ciphertext where Hadoop's own reader succeeds)"
+
+ /**
+ * First reason any bucket among `uris` is configured for an encryption algorithm the native S3
+ * client cannot safely read, or `None` when claimable. Allowlist-based (see
+ * [[AllowedEncryptionAlgorithms]]): only `AES256`/`SSE-KMS`/`DSSE-KMS` (and unset/empty) pass;
+ * every other resolved value -- `SSE-C`, `CSE-KMS`, `CSE-CUSTOM`, or any unrecognized future
+ * algorithm string -- declines. Deliberately NOT a blocklist keyed on `SSE-C` alone: an
+ * allowlist is safe by construction against a Hadoop release adding a new encryption method
+ * this gate has never heard of, where a blocklist would silently admit it. Never interpolates a
+ * resolved key value, only key names, the bucket, and the (non-secret) algorithm name. Declines
+ * on a `Left` from [[effectiveEncryptionAlgorithm]] too (an unverifiable credential-provider
+ * arm, e.g. an S3A-scoped provider path or a corrupt/unreadable global keystore) -- the
+ * algorithm cannot be ruled safe when it cannot be read at all.
+ */
+ private[delta] def unsupportedEncryptionAlgorithmReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ effectiveEncryptionAlgorithm(hadoopConf, bucket) match {
+ case Left(reason) => Some(reason)
+ case Right(None) => None
+ case Right(Some((key, value))) =>
+ if (!AllowedEncryptionAlgorithms.exists(_.equalsIgnoreCase(value))) {
+ Some(unsupportedEncryptionAlgorithmDeclineReason(bucket, key, value))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Canonical Hadoop S3A HTTP-proxy host config key, verified via CFR decompilation of
+ * `hadoop-aws` 3.3.4's `S3AUtils.class` (`initProxySupport`):
+ * {{{
+ * String proxyHost = conf.getTrimmed("fs.s3a.proxy.host", "");
+ * int proxyPort = conf.getInt("fs.s3a.proxy.port", -1);
+ * if (!proxyHost.isEmpty()) {
+ * ...
+ * String proxyUsername =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.username", null, null);
+ * String proxyPassword =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.password", null, null);
+ * ...
+ * }
+ * }}}
+ * `fs.s3a.proxy.host`/`fs.s3a.proxy.port` resolve via a PLAIN, non-bucket-scoped, non-JCEKS
+ * `Configuration#getTrimmed`/`getInt` call -- NOT `lookupPassword` -- against whatever conf
+ * `S3AFileSystem#initialize` already ran through `propagateBucketOptions` before
+ * `createAwsConf`/`initProxySupport` ever runs; only the SIBLING `fs.s3a.proxy.username`/
+ * `fs.s3a.proxy.password` keys go through `lookupPassword` (bucket long/short/global,
+ * JCEKS-aware). So the host is bucket-aware only via `propagateBucketOptions`'s short-bucket-
+ * form folding, never the long-bucket form, and never a credential-provider read -- the SAME
+ * shape as `endpoint`/`path.style.access` (the "propagated option" family, see
+ * [[AllS3ConfigKeys]]'s doc), not the credential family.
+ *
+ * Resolved here through [[hadoopLookupPasswordEffective]] anyway (long-then-short-then-global,
+ * credential-provider-aware) for the identical reason [[AllS3ConfigKeys]]'s non-credential
+ * entries are: checking MORE than Hadoop's real, narrower resolution can only ever find a value
+ * Hadoop itself ignores, which can only turn an already-necessary decline into one found
+ * slightly more eagerly -- never a missed one. A missed decline is exactly the failure mode
+ * this gate exists to close (native has NO HTTP-proxy support of any kind -- no
+ * `fs.s3a.proxy.*` key is read anywhere in `s3.rs`), so over-approximating is the conservative
+ * direction here, unlike a value-EQUALITY comparator (e.g. [[s3KeyDivergenceReason]]) where
+ * over-approximating could manufacture a false divergence between two sides that would
+ * otherwise agree.
+ */
+ private val S3ProxyHostKey = "fs.s3a.proxy.host"
+
+ private def unsupportedProxyReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key configured for $bucket (the native S3 client has " +
+ "no HTTP proxy support at all -- no fs.s3a.proxy.* key is read anywhere in its object " +
+ "store layer -- so a claimed scan would connect to S3 directly instead of routing through " +
+ "the configured proxy, either bypassing an egress/network-segmentation policy or simply " +
+ "failing to reach the endpoint)"
+
+ /**
+ * First reason any bucket among `uris` has an HTTP proxy configured via [[S3ProxyHostKey]], or
+ * `None` when claimable. Ordered alongside [[unsupportedEncryptionAlgorithmReason]] among the
+ * other conf-only gates that are zero-I/O in the common case, ahead of
+ * [[s3ConfigDivergenceReason]]: [[hadoopLookupPasswordEffective]] only performs real keystore
+ * I/O, inside its own try/catch, when a Hadoop credential-provider path is actually configured
+ * for this bucket (see that function's doc for the Arm A/B/C dispatch). Never interpolates a
+ * resolved value: the proxy HOST is not secret, but naming it here would be a strange place to
+ * first surface it, and proxy CREDENTIALS (`fs.s3a.proxy.username`/`fs.s3a.proxy.password`, not
+ * read by this gate at all -- the whole point of gating on the host is that a non-empty host
+ * declines before any proxy credential would ever need to be forwarded) must never appear in a
+ * decline reason regardless.
+ */
+ private[delta] def proxyGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI],
+ propagatedConfCache: MutableMap[String, Configuration] = MutableMap.empty)
+ : Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ try {
+ val propagatedConf =
+ propagatedConfCache.getOrElseUpdate(
+ bucket,
+ propagateBucketOptions(hadoopConf, bucket))
+ hadoopLookupPasswordEffective(propagatedConf, bucket, S3ProxyHostKey) match {
+ case Left(reason) => Some(reason)
+ case Right(Some(_)) => Some(unsupportedProxyReason(bucket, S3ProxyHostKey))
+ case Right(None) => None
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableValueReason(bucket, S3ProxyHostKey, e))
+ }
+ }
+ }
+ }
+
+ /**
+ * `baseKey`'s long, short, then global per-bucket aliases, in Hadoop's own resolution order.
+ */
+ private def longThenShortThenGlobalAliases(bucket: String, baseKey: String): Seq[String] = {
+ val suffix = baseKey.stripPrefix("fs.s3a.")
+ Seq(s"fs.s3a.bucket.$bucket.fs.s3a.$suffix", s"fs.s3a.bucket.$bucket.$suffix", baseKey)
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * The three-way Hadoop credential-provider-path precheck shared by every `getPassword`-based
+ * resolution below, factored out of what used to be [[hadoopLookupPasswordEffective]]'s own
+ * body so [[effectiveEncryptionAlgorithm]] can reuse it for its own bucket-tier-only and
+ * global-tier-only lookups (see that function's doc for why the algorithm needs its resolution
+ * split into separate tiers rather than one combined long-then-short-then-global list). This
+ * precheck is a property of the BUCKET alone, independent of which base key or alias list a
+ * caller goes on to resolve: Arm A -- an S3A-scoped or bucket-scoped Hadoop credential-provider
+ * path configures a provider `Configuration#getPassword` does not consult
+ * (`S3AUtils#lookupPassword`/`#lookupBucketSecret` build their own S3A-only provider list per
+ * bucket, which this plain-conf-reading gate cannot mirror) -- returns
+ * [[UnverifiableProvider]], NO keystore I/O. Arm B -- only the GLOBAL Hadoop provider path is
+ * set (which `getPassword` DOES consult) -- returns [[GlobalProviderOnly]]; the caller must
+ * wrap its own `getPassword` calls in try/catch, since real keystore I/O only happens once this
+ * arm is reached. Arm C -- no provider path is set anywhere -- returns [[NoProvider]], zero-I/O
+ * plain conf only.
+ */
+ private sealed trait CredentialProviderArm
+ private case class UnverifiableProvider(offendingKey: String) extends CredentialProviderArm
+ private case object GlobalProviderOnly extends CredentialProviderArm
+ private case object NoProvider extends CredentialProviderArm
+
+ private def credentialProviderArm(
+ hadoopConf: Configuration,
+ bucket: String): CredentialProviderArm = {
+ val bucketPathKey = s3aBucketProviderPathKey(bucket)
+ val bucketLongPathKey = s3aBucketLongProviderPathKey(bucket)
+ val s3aPathSet = nonEmptyConf(hadoopConf, S3aCredentialProviderPathKey)
+ val bucketPathSet = nonEmptyConf(hadoopConf, bucketPathKey)
+ val bucketLongPathSet = nonEmptyConf(hadoopConf, bucketLongPathKey)
+ if (s3aPathSet || bucketPathSet || bucketLongPathSet) {
+ val offendingKey =
+ if (s3aPathSet) S3aCredentialProviderPathKey
+ else if (bucketPathSet) bucketPathKey
+ else bucketLongPathKey
+ UnverifiableProvider(offendingKey)
+ } else if (nonEmptyConf(hadoopConf, HadoopCredentialProviderPathKey)) {
+ GlobalProviderOnly
+ } else {
+ NoProvider
+ }
+ }
+
+ /**
+ * `aliases`' effective value under `hadoopConf`/`bucket`, tried in the given order and keeping
+ * the first non-empty result -- or `Left(reason)` when the value cannot be safely verified.
+ * Dispatches on [[credentialProviderArm]]: [[UnverifiableProvider]] declines outright (zero
+ * I/O); [[GlobalProviderOnly]] resolves every alias via `Configuration#getPassword` (real
+ * keystore I/O, contained in try/catch: a corrupt/unreadable store must decline this bucket,
+ * not abort planning for the whole session); [[NoProvider]] resolves every alias via plain
+ * [[plainValue]] reads (zero I/O). Generalizes what used to be
+ * [[hadoopLookupPasswordEffective]]'s entire body to an explicit, caller-supplied alias list --
+ * [[hadoopLookupPasswordEffective]] is now the long-then-short-then-global special case of
+ * this; [[effectiveEncryptionAlgorithm]] is the bucket-tier-only (two aliases) and
+ * global-tier-only (one alias) special cases, matching
+ * `S3AUtils#lookupBucketSecret`/`#lookupPassword`'s real per-tier `getPassword` calls exactly
+ * instead of [[plainValue]]'s JCEKS-blind read that under-declined a keystore-only algorithm.
+ */
+ private def resolveViaCredentialAliases(
+ hadoopConf: Configuration,
+ bucket: String,
+ aliases: Seq[String]): Either[String, Option[String]] =
+ credentialProviderArm(hadoopConf, bucket) match {
+ case UnverifiableProvider(offendingKey) =>
+ Left(s3aScopedProviderPathReason(bucket, offendingKey))
+ case GlobalProviderOnly =>
+ try {
+ Right(
+ aliases.iterator
+ .map(alias =>
+ Option(hadoopConf.getPassword(alias)).map(new String(_)).filter(_.nonEmpty))
+ .collectFirst { case Some(v) => v })
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Left(unverifiableCredentialProviderReason(bucket, e))
+ }
+ case NoProvider =>
+ Right(aliases.flatMap(plainValue(hadoopConf, _)).headOption)
+ }
+
+ /**
+ * `bucket`'s effective value for `baseKey` in Hadoop's own `S3AUtils#lookupPassword` resolution
+ * order -- long bucket alias, then short bucket alias, then global, each tried through a Hadoop
+ * credential provider before falling back to plain conf (see [[resolveViaCredentialAliases]]
+ * for the Arm A/B/C dispatch this delegates to) -- or `Left(reason)` when the value cannot be
+ * safely verified.
+ *
+ * USED FOR EVERY KEY IN [[AllS3ConfigKeys]], not just [[CredentialValueKeys]] -- see that val's
+ * doc for why the structural bug behind the SSE-C long-bucket-alias gap was exactly this split
+ * (some keys resolved long-bucket-aware, others short-only) and why resolving every key through
+ * this SAME long-then-short-then-global function is fail-safe even for the keys (`endpoint`,
+ * `path.style.access`, `requester.pays.enabled`, `comet.credential.provider.class`) whose real
+ * Hadoop resolver is `S3AUtils#propagateBucketOptions` + plain `Configuration#get`, which never
+ * consults the long bucket form or a credential provider at all: checking the long alias (or a
+ * credential-provider path) for one of THOSE keys can only ever find a value real Hadoop itself
+ * ignores. That can make this function return a value differing from Hadoop's TRUE effective
+ * value only in the direction of an EXTRA divergence (a long-form/keystore value set
+ * differently from short/global for a key Hadoop never reads that way) -- which
+ * [[s3KeyDivergenceReason]] turns into an extra DECLINE, never a wrongly-claimed scan, since a
+ * value Hadoop never reads cannot be the reason a wrongly-claimed native scan would misbehave.
+ * A key added to [[AllS3ConfigKeys]] in the future inherits this safety automatically; no
+ * per-key classification decision (and no repeat of the SSE-C bug) is needed.
+ *
+ * `hadoopConf` must be a [[propagateBucketOptions]] result (the caller,
+ * [[s3KeyDivergenceReason]], always passes one) so that `${...}` references embedded in any
+ * alias resolve exactly like `S3AFileSystem#initialize`'s real propagate-then-resolve order.
+ */
+ private def hadoopLookupPasswordEffective(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Either[String, Option[String]] =
+ resolveViaCredentialAliases(
+ hadoopConf,
+ bucket,
+ longThenShortThenGlobalAliases(bucket, baseKey))
+
+ private def effectiveValueDivergenceReason(bucket: String, key: String): String =
+ s"Native Delta scan cannot forward $key for $bucket (Hadoop's effective value for this key " +
+ "differs from what the native S3 client resolves, so its credentials or configuration " +
+ "would differ from Hadoop's)"
+
+ private def unverifiableValueReason(bucket: String, key: String, error: Throwable): String =
+ s"Native Delta scan cannot verify $key for $bucket (Configuration#get raised " +
+ s"${error.getClass.getName}), declining rather than risk forwarding a stale or " +
+ "diverging value"
+
+ /**
+ * `None` when `baseKey`'s Hadoop-effective and native-effective values under `bucket` agree, or
+ * a decline reason naming `baseKey` and `bucket` (never a value) when they diverge or either
+ * side cannot be safely computed.
+ *
+ * Hadoop's effective value is computed against [[propagateBucketOptions]]'s result, mirroring
+ * `S3AFileSystem#initialize`'s actual order (propagate bucket options into the conf FIRST, only
+ * THEN read/substitute options against it), via Hadoop's `lookupPassword`-order read
+ * ([[hadoopLookupPasswordEffective]]) for EVERY key in [[AllS3ConfigKeys]] -- see that
+ * function's doc for why using the same long-then-short-then-global resolution uniformly,
+ * rather than a short-then-global-only read for the keys Hadoop's real resolver does not make
+ * long-bucket-aware, is a deliberate, fail-safe over-approximation. Native's effective value is
+ * always `nativeShortThenGlobal(hadoopConf, ...)` on the ORIGINAL, unpropagated conf, matching
+ * `NativeConfig.extractObjectStoreOptions`'s actual forwarding semantics (no propagation step)
+ * AND native's `get_config` presence-based (not emptiness-based) short-vs-global fallback --
+ * see [[nativeShortThenGlobal]]. Both values are trimmed together, symmetrically, right before
+ * the equality check below (mirroring `get_config_trimmed`'s `.trim()`, which native applies
+ * regardless of which alias it read) rather than trimming [[nativeShortThenGlobal]]'s result on
+ * its own -- see [[nativeShortThenGlobal]]'s doc for why a one-sided trim there would flag a
+ * spurious divergence. Comparing against the propagated view (rather than the original conf, as
+ * an earlier version of this check did) matters because propagation can change what a `${...}`
+ * reference inside one bucket-scoped value resolves to: e.g.
+ * `fs.s3a.bucket.B.access.key=${fs.s3a.custom.ref}` with `fs.s3a.bucket.B.custom.ref=X` and
+ * global `fs.s3a.custom.ref=Y` propagates to `fs.s3a.custom.ref=X` (overwriting the global `Y`)
+ * before the access key's `${...}` reference is ever substituted, so Hadoop resolves `X` while
+ * a check against the unpropagated conf would (wrongly) also see `Y`, the same value native
+ * forwards -- masking a real divergence. Wrapped in try/catch: `Configuration#get` raises
+ * `IllegalStateException` once `${...}` substitution recurses past Hadoop's `MAX_SUBST` bound
+ * (e.g. a two-key mutual reference cycle); declining is safer than crashing planning or
+ * comparing a partially-substituted value.
+ */
+ private def s3KeyDivergenceReason(
+ hadoopConf: Configuration,
+ propagatedConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ try {
+ val hadoopEffective: Either[String, Option[String]] =
+ hadoopLookupPasswordEffective(propagatedConf, bucket, baseKey)
Review Comment:
Good catch, and it broke more than this one key: the blanket unification I landed earlier reads wider than the real consumer for plain options, and a comparator that reads wider can manufacture equality, which is the dangerous direction. The comparator now carries a per-key consumer tier: the credential trio keeps lookupPassword semantics (the SSE-C and JCEKS tests all still pass untouched), everything else resolves the way its actual consumer does, plain get on the propagated conf, with the ${} reference expanding under the propagated view. Your four-setting shape now declines naming the endpoint key, the companion without the per-bucket referent override still admits, and a lone long-form endpoint alias now admits too, since propagateBucketOptions folds it into the unread fs.s3a.fs.s3a.endpoint key. Adding a key without a tier no longer compiles, a harness test pins the password tier to exactly the trio, and each consumer assignment was checked against the 3.3.4 bytecode rather
than the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3904468093
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1627 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Zero-I/O in the common case, like the encryption gate above: native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The three keys where Hadoop's OWN resolver (`S3AUtils#lookupPassword`) is long-bucket-aware,
+ * verified via `javap` against `hadoop-aws` 3.3.4: `S3AUtils#lookupPassword(bucket, conf,
+ * baseKey)` builds `longBucketKey = "fs.s3a.bucket." + bucket + "." + baseKey` (note: the FULL
+ * `baseKey`, already `fs.s3a`-prefixed, appended after the bucket segment) and reads it via
+ * `getPassword` BEFORE the short-bucket key, keeping the long value whenever `getPassword`
+ * returns non-empty for it and only falling through to short-then-global otherwise. Kept as a
+ * named subset purely for documentation/discovery purposes (see
+ * [[hadoopLookupPasswordEffective]]'s doc); every key in [[AllS3ConfigKeys]] below is resolved
+ * through the SAME long-then-short-then-global function regardless of membership here -- see
+ * that function's doc for why extending long-bucket-awareness to keys where it does not
+ * strictly apply is safe.
+ */
+ private val CredentialValueKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ /**
+ * Every `fs.s3a.*` base key that governs whether a claimed native scan actually behaves like
+ * Hadoop's own reader would, checked by [[s3ConfigDivergenceReason]] via
+ * [[hadoopLookupPasswordEffective]] (see that function's doc for the resolution order and the
+ * deliberate over-decline rationale for the non-credential entries below). Two groups, kept
+ * merged into one list and one resolution path -- NOT two lists with two different resolution
+ * mechanisms, which is what let `fs.s3a.encryption.algorithm` silently fall through the
+ * short-then-global-only path while Hadoop's real resolver for it was long-bucket-aware (the
+ * SSE-C long-bucket-alias gap): the ORIGINAL [[CredentialValueKeys]] three, plus every OTHER
+ * per-bucket `fs.s3a.*` base key native's S3 client's `get_config` (s3.rs) resolves, verified
+ * directly against its call sites: `extract_s3_config_options` (endpoint.region,
+ * path.style.access, endpoint, requester.pays.enabled), `lookup_provider_class` (the
+ * Comet-specific credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name).
+ *
+ * SYNC NOTE: this list's non-credential entries must stay a superset of native's
+ * `NATIVE_S3A_CONFIG_PROPERTIES` constant (`native/core/src/parquet/objectstore/s3.rs`,
+ * property suffixes without the `fs.s3a.` prefix) -- `DeltaScanContribSuite`'s
+ * discovery-harness test asserts this mechanically. Literal strings, not the
+ * [[AwsCredentialsProviderKey]] / [[AssumedRoleCredentialsProviderKey]] vals declared below,
+ * purely to avoid a forward reference inside this `object` body; kept textually identical to
+ * those two constants.
+ */
+ private[delta] val AllS3ConfigKeys: Seq[String] = Seq(
+ "fs.s3a.access.key",
+ "fs.s3a.secret.key",
+ "fs.s3a.session.token",
+ "fs.s3a.aws.credentials.provider",
+ "fs.s3a.assumed.role.arn",
+ "fs.s3a.assumed.role.session.name",
+ "fs.s3a.assumed.role.credentials.provider",
+ "fs.s3a.endpoint",
+ "fs.s3a.endpoint.region",
+ "fs.s3a.path.style.access",
+ "fs.s3a.requester.pays.enabled",
+ "fs.s3a.comet.credential.provider.class")
+
+ /**
+ * The short-bucket-then-global value resolved for `baseKey` under `bucket` from `hadoopConf`,
+ * skipping an empty value at either alias exactly like [[plainValue]]. NOT used by
+ * [[s3ConfigDivergenceReason]]/[[s3KeyDivergenceReason]] any more -- every key checked there
+ * resolves through [[hadoopLookupPasswordEffective]] uniformly (long-then-short-then-global),
+ * which is a strict superset of what this function reads. This function's one remaining caller
+ * is [[shortThenGlobalOrReason]], which reads provider-CLASS strings (from the ORIGINAL,
+ * unpropagated conf) for name-support validation in [[providerClassReason]]/
+ * [[assumedRoleProviderClassReason]] -- by the time those run, [[s3ConfigDivergenceReason]] has
+ * already proven Hadoop's and native's effective values agree for the same key, so whichever of
+ * the two (equal) values this narrower read returns does not affect correctness there. NEVER
+ * used to compute native's own effective value -- see [[nativeShortThenGlobal]] for that.
+ */
+ private def shortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket` from the ORIGINAL, unpropagated `hadoopConf` -- `NativeConfig
+ * .extractObjectStoreOptions` forwards `Configuration#get`'s substituted value for every
+ * `fs.s3a.*` entry with no bucket-option propagation step of its own, so the original conf is
+ * the right input here. Unlike [[shortThenGlobal]]/[[plainValue]], this mirrors `get_config`
+ * faithfully: PRESENCE of the short-bucket key alone -- never its emptiness -- decides whether
+ * native falls back to the global key (`get_config` is a plain `HashMap::get`, which returns
+ * `Some` for a key explicitly set to `""`), so an explicitly empty or whitespace-only
+ * short-bucket value resolves to `Some("")` here and never falls through to global -- the
+ * OPPOSITE of Hadoop's own `getPassword`/`lookupPassword` semantics (see
+ * [[hadoopLookupPasswordEffective]]), which treat empty as absent and keep trying the next
+ * alias. The ONLY function used to compute native's effective value in
+ * [[s3KeyDivergenceReason]].
+ *
+ * Deliberately does NOT apply `get_config_trimmed`'s `.trim()` here: [[s3KeyDivergenceReason]]
+ * trims both this value and Hadoop's effective value together, symmetrically, at the point they
+ * are compared, rather than one-sidedly here -- trimming only the native side would flag a
+ * spurious divergence for a value neither side's whitespace actually changes the behavior of
+ * once each side's own downstream parsing normalizes it (e.g. Hadoop's own multi-line
+ * `fs.s3a.aws.credentials.provider` default, which both Hadoop and native additionally trim per
+ * comma-separated entry after splitting), while a one-sided trim would make an
+ * otherwise-identical default value look diverged for every bucket, never claiming natively at
+ * all.
+ */
+ private def nativeShortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ Option(hadoopConf.get(shortKey)).orElse(Option(hadoopConf.get(baseKey)))
+ }
+
+ /**
+ * Faithful in-memory replica of `S3AUtils#propagateBucketOptions` (`hadoop-aws`), which
+ * `S3AFileSystem#initialize` calls FIRST, before any option or credential is read:
+ * `Configuration conf = propagateBucketOptions(originalConf, bucket); ...; setConf(conf);` --
+ * every subsequent `conf.get`/`getPassword` call in that filesystem instance, including
+ * `${...}` variable substitution, resolves against this propagated view, not the original conf.
+ * `hadoop-aws` is not on this module's runtime classpath (see the string-literal-keys note
+ * above), so `S3AUtils#propagateBucketOptions` cannot be called directly; this reproduces its
+ * logic verbatim using only `hadoop-common`'s `Configuration`:
+ * {{{
+ * public static Configuration propagateBucketOptions(Configuration source, String bucket) {
+ * final String bucketPrefix = FS_S3A_BUCKET_PREFIX + bucket + '.';
+ * final Configuration dest = new Configuration(source);
+ * for (Map.Entry<String, String> entry : source) {
+ * final String key = entry.getKey();
+ * final String value = entry.getValue(); // the (unexpanded) value
+ * if (!key.startsWith(bucketPrefix) || bucketPrefix.equals(key)) continue;
+ * final String stripped = key.substring(bucketPrefix.length());
+ * if (stripped.startsWith("bucket.") || "impl".equals(stripped)) {
+ * // ignored
+ * } else {
+ * final String generic = FS_S3A_PREFIX + stripped;
+ * dest.set(generic, value, ...); // overwrites any existing global value
+ * }
+ * }
+ * return dest;
+ * }
+ * }}}
+ * Note the LONG bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) folds to an unread
+ * `fs.s3a.fs.s3a.<key>` key here too, exactly like the real method -- `stripped` already starts
+ * with `fs.s3a.` in that case, so prepending `fs.s3a.` again produces a key nothing ever reads.
+ */
+ private def propagateBucketOptions(hadoopConf: Configuration, bucket: String): Configuration = {
+ val bucketPrefix = s"fs.s3a.bucket.$bucket."
+ val dest = new Configuration(hadoopConf)
+ hadoopConf.iterator().asScala.foreach { entry =>
+ val key = entry.getKey
+ if (key.startsWith(bucketPrefix) && key != bucketPrefix) {
+ val stripped = key.substring(bucketPrefix.length)
+ if (!stripped.startsWith("bucket.") && stripped != "impl") {
+ dest.set(s"fs.s3a.$stripped", entry.getValue)
+ }
+ }
+ }
+ dest
+ }
+
+ /**
+ * Canonical and deprecated Hadoop S3A encryption-algorithm config keys, verified via `javap`
+ * against `hadoop-aws` 3.3.4's `org.apache.hadoop.fs.s3a.Constants`: `S3_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.encryption.algorithm"` (canonical) and `SERVER_SIDE_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.server-side-encryption-algorithm"` (DEPRECATED -- note the hyphen before "algorithm",
+ * unlike the corresponding `*.key` constants below, which both use a `.key` suffix).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, same
+ * rationale as [[HadoopCredentialProviderPathKey]].
+ */
+ private val S3EncryptionAlgorithmKey = "fs.s3a.encryption.algorithm"
+ private val DeprecatedS3EncryptionAlgorithmKey = "fs.s3a.server-side-encryption-algorithm"
+
+ /**
+ * The exact strings `S3AEncryptionMethods#getMethod` accepts, verified via `javap`/CFR against
+ * `hadoop-aws` 3.3.4's `S3AEncryptionMethods` enum: `NONE("")`, `SSE_S3("AES256", serverSide =
+ * true, requiresSecret = false)`, `SSE_KMS("SSE-KMS", serverSide = true, requiresSecret =
+ * false)`, `SSE_C("SSE-C", serverSide = true, requiresSecret = true)`, `CSE_KMS("CSE-KMS",
+ * serverSide = false, requiresSecret = true)`, `CSE_CUSTOM("CSE-CUSTOM", serverSide = false,
+ * requiresSecret = true)`. `getMethod` parses case-insensitively
+ * (`values().find(_.getMethod.equalsIgnoreCase(algorithm))`), matched below the same way.
+ *
+ * ALLOWLIST, not a blocklist (replaces the former SSE-C-only blocklist): only the algorithms S3
+ * decrypts transparently on GET/HEAD given read permission alone, with NO extra request header
+ * and NO client-side step, are safe for a native scan that forwards none of Hadoop's
+ * `fs.s3a.encryption.*`/`fs.s3a.server-side-encryption*` options --
+ * - `AES256` (SSE_S3, `serverSide = true`): plain server-side encryption, transparent on GET.
+ * - `SSE-KMS` (SSE_KMS, `serverSide = true`): server-side, KMS-managed key, transparent on
+ * GET given KMS decrypt permission (no header).
+ * - `DSSE-KMS`: NOT present in this enum on `hadoop-aws` 3.3.4 (confirmed by the six values
+ * listed above) -- `S3AEncryptionMethods.getMethod("DSSE-KMS")` throws
+ * `IOException("Unknown encryption algorithm DSSE-KMS")` on this version, so
+ * `S3AUtils#buildEncryptionSecrets` (and therefore Hadoop's own reader) already fails
+ * before ever reading such a table under 3.3.4, meaning this string can never actually be
+ * the resolved value on the declared target version -- admitting it here is inert there.
+ * Included anyway, forward-compatible, for a newer `hadoop-aws` on the runtime classpath (a
+ * later Hadoop release; this module has no compile-time `hadoop-aws` dependency, see the
+ * string-literal-keys note above) where DSSE-KMS is a real, dual-layer, server-side
+ * algorithm decrypted transparently on GET the same way SSE-KMS is. Every other value
+ * declines: `SSE-C` (SSE_C is `serverSide = true` in Hadoop's own enum, but `requiresSecret
+ * \= true` -- S3 rejects a GET/HEAD for an SSE-C object outright (400 Bad Request) unless
+ * the customer key is resent as a request header on every call, so a native scan that never
+ * learns the key cannot succeed at all, where Hadoop's own reader -- whose request factory
+ * attaches the key -- would), `CSE-KMS`/`CSE-CUSTOM` (`serverSide = false`: client-side
+ * encryption decrypts object bytes locally in the SDK layer, which the native Parquet
+ * reader has no equivalent of -- it would read raw ciphertext), and any future/unknown
+ * value (a value `S3AEncryptionMethods.getMethod` itself would reject is certainly not one
+ * of the three confirmed-transparent algorithms above; declining is the only safe default
+ * for anything this gate cannot positively confirm).
+ */
+ private val AllowedEncryptionAlgorithms: Set[String] = Set("AES256", "SSE-KMS", "DSSE-KMS")
+
+ /**
+ * `bucket`'s effective encryption-algorithm key and value under `hadoopConf`, or `None` when
+ * neither the canonical nor deprecated key is set anywhere consulted. Mirrors
+ * `S3AUtils#buildEncryptionSecrets`'s real resolution order, verified via `javap`/CFR
+ * decompilation of `hadoop-aws` 3.3.4's `S3AUtils.class`:
+ * {{{
+ * String algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.server-side-encryption-algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.server-side-encryption-algorithm");
+ * }}}
+ * i.e. bucket-tier (canonical, then deprecated), THEN global-tier (canonical, then deprecated)
+ * -- the two tiers are never interleaved key-by-key, so this must stay two explicit bucket-tier
+ * lookups followed by two explicit global-tier lookups, not a single
+ * [[hadoopLookupPasswordEffective]] call per key (which would let an unset canonical bucket key
+ * fall through straight to the canonical GLOBAL value ahead of a SET deprecated bucket key, the
+ * wrong answer).
+ *
+ * THE FIX for the SSE-C long-bucket-alias gap is entirely inside the bucket tier:
+ * `lookupBucketSecret` itself is long-then-short, decompiled from `hadoop-aws` 3.3.4's
+ * `S3AUtils.class`:
+ * {{{
+ * // longBucketKey = fs.s3a.bucket.B.fs.s3a.<key>
+ * String longBucketKey = String.format(BUCKET_PATTERN, bucket, baseKey);
+ * String initialVal = getPassword(conf, longBucketKey, null, null);
+ * // shortBucketKey = fs.s3a.bucket.B.<key>
+ * String shortBucketKey = String.format(BUCKET_PATTERN, bucket, subkey);
+ * // keeps initialVal (the LONG value) if non-empty
+ * return getPassword(conf, shortBucketKey, initialVal, null);
+ * }}}
+ * i.e. the SAME long-bucket-key construction and long-wins-if-nonempty semantics as
+ * `S3AUtils#lookupPassword` (see [[CredentialValueKeys]]/[[hadoopLookupPasswordEffective]]) --
+ * the encryption algorithm is NOT one of the keys that flows through
+ * `S3AUtils#propagateBucketOptions` (which folds an unrelated per-bucket LONG form into an
+ * unread key). An earlier version of this function modeled the bucket tier as SHORT-only,
+ * documented as "the LONG bucket form is genuinely never consulted for this key" -- that
+ * documentation was wrong (this decompilation supersedes it): a bucket configured only via
+ * `fs.s3a.bucket.B.fs.s3a.encryption.algorithm=SSE-C` bypassed the SSE-C gate entirely, because
+ * Hadoop's own reader DOES read that long form (and picks SSE-C), while this function reported
+ * `None` (nothing set) and the allowlist check below never even ran.
+ *
+ * The canonical-vs-deprecated distinction below is frequently moot in practice: `hadoop-aws`'s
+ * `S3AFileSystem.addDeprecatedKeys()` statically registers `fs.s3a.server-side-encryption-*` as
+ * `Configuration`-level deprecated aliases of `fs.s3a.encryption.*` (verified via `javap`), a
+ * registration that lives in a static field on Hadoop's `Configuration` class -- process-wide
+ * once `S3AFileSystem`'s class has loaded anywhere in the JVM, which a real scan has always
+ * already done by the time this gate runs, since reading the S3 table at all requires loading
+ * that class. Once active, `Configuration#get` resolves either literal key to the identical
+ * value transparently, making the two-key cascade below redundant (but harmless) for that case;
+ * it remains the operative path only when nothing else in the process has loaded
+ * `S3AFileSystem` yet.
+ *
+ * ALSO walks the Hadoop-credential-provider (JCEKS) path via [[resolveViaCredentialAliases]]
+ * for each of the four lookups below, matching `lookupBucketSecret`/`lookupPassword`'s real
+ * per-alias `getPassword` calls (quoted above) exactly: both are `getPassword`, not plain
+ * `Configuration#get`, so a bucket storing the algorithm name ONLY in a JCEKS keystore is
+ * exactly as real a Hadoop deployment shape for this key as it is for the credential keys
+ * [[hadoopLookupPasswordEffective]] already covers -- there is nothing algorithm-specific that
+ * makes JCEKS storage implausible here, so an earlier version of this function skipping it
+ * (documented at the time as "the algorithm NAME is not credential-sensitive data, so storing
+ * it in a keystore is not a realistic Hadoop deployment pattern") was an unjustified, narrower
+ * read than Hadoop's own resolver actually performs, under-declining a bucket whose algorithm
+ * is keystore-only. [[resolveViaCredentialAliases]]'s Arm B/C split still means this is zero
+ * extra I/O for the common case: keystore I/O only happens when a Hadoop credential-provider
+ * path is actually configured for the bucket, contained in that function's own try/catch.
+ * `bucketTier`/`globalTier` return `Left` (propagated straight through by [[orElseTier]]) when
+ * [[resolveViaCredentialAliases]] cannot safely verify a tier at all (an S3A-scoped provider
+ * path, or a corrupt/unreadable global keystore) -- correctly short-circuiting the whole
+ * cascade with a decline rather than silently falling through to a later tier that might look
+ * unset only because the true value was unverifiable.
+ */
+ private def effectiveEncryptionAlgorithm(
+ hadoopConf: Configuration,
+ bucket: String): Either[String, Option[(String, String)]] = {
+ def bucketTier(baseKey: String): Either[String, Option[(String, String)]] = {
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(longKey, shortKey))
+ .map(_.map(baseKey -> _))
+ }
+ def globalTier(baseKey: String): Either[String, Option[(String, String)]] =
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(baseKey)).map(_.map(baseKey -> _))
+
+ // Short-circuits on Left (unverifiable tier) or Right(Some(_)) (resolved); only Right(None)
+ // (tier definitively unset) falls through to `next`, mirroring buildEncryptionSecrets's
+ // sequential `if (algorithm == null) algorithm = ...` cascade exactly.
+ def orElseTier(
+ current: Either[String, Option[(String, String)]],
+ next: => Either[String, Option[(String, String)]])
+ : Either[String, Option[(String, String)]] =
+ current match {
+ case Left(reason) => Left(reason)
+ case Right(Some(value)) => Right(Some(value))
+ case Right(None) => next
+ }
+
+ orElseTier(
+ bucketTier(S3EncryptionAlgorithmKey),
+ orElseTier(
+ bucketTier(DeprecatedS3EncryptionAlgorithmKey),
+ orElseTier(
+ globalTier(S3EncryptionAlgorithmKey),
+ globalTier(DeprecatedS3EncryptionAlgorithmKey))))
+ }
+
+ private def unsupportedEncryptionAlgorithmDeclineReason(
+ bucket: String,
+ algorithmKey: String,
+ algorithm: String): String =
+ s"Native Delta scan does not support $algorithmKey=$algorithm for $bucket " +
+ "(the native S3 client only supports unencrypted objects and S3's transparent " +
+ "server-side algorithms -- AES256/SSE-S3, SSE-KMS, and DSSE-KMS decrypt on GET/HEAD given " +
+ "read permission alone, with no extra request header; SSE-C additionally requires the " +
+ "customer-provided key resent as a header on every GET/HEAD request, which the native S3 " +
+ "client's extract_s3_config_options never forwards, and CSE-KMS/CSE-CUSTOM decrypt object " +
+ "bytes client-side, a layer the native Parquet reader does not have -- any of these would " +
+ "fail outright or silently read ciphertext where Hadoop's own reader succeeds)"
+
+ /**
+ * First reason any bucket among `uris` is configured for an encryption algorithm the native S3
+ * client cannot safely read, or `None` when claimable. Allowlist-based (see
+ * [[AllowedEncryptionAlgorithms]]): only `AES256`/`SSE-KMS`/`DSSE-KMS` (and unset/empty) pass;
+ * every other resolved value -- `SSE-C`, `CSE-KMS`, `CSE-CUSTOM`, or any unrecognized future
+ * algorithm string -- declines. Deliberately NOT a blocklist keyed on `SSE-C` alone: an
+ * allowlist is safe by construction against a Hadoop release adding a new encryption method
+ * this gate has never heard of, where a blocklist would silently admit it. Never interpolates a
+ * resolved key value, only key names, the bucket, and the (non-secret) algorithm name. Declines
+ * on a `Left` from [[effectiveEncryptionAlgorithm]] too (an unverifiable credential-provider
+ * arm, e.g. an S3A-scoped provider path or a corrupt/unreadable global keystore) -- the
+ * algorithm cannot be ruled safe when it cannot be read at all.
+ */
+ private[delta] def unsupportedEncryptionAlgorithmReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ effectiveEncryptionAlgorithm(hadoopConf, bucket) match {
+ case Left(reason) => Some(reason)
+ case Right(None) => None
+ case Right(Some((key, value))) =>
+ if (!AllowedEncryptionAlgorithms.exists(_.equalsIgnoreCase(value))) {
+ Some(unsupportedEncryptionAlgorithmDeclineReason(bucket, key, value))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Canonical Hadoop S3A HTTP-proxy host config key, verified via CFR decompilation of
+ * `hadoop-aws` 3.3.4's `S3AUtils.class` (`initProxySupport`):
+ * {{{
+ * String proxyHost = conf.getTrimmed("fs.s3a.proxy.host", "");
+ * int proxyPort = conf.getInt("fs.s3a.proxy.port", -1);
+ * if (!proxyHost.isEmpty()) {
+ * ...
+ * String proxyUsername =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.username", null, null);
+ * String proxyPassword =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.password", null, null);
+ * ...
+ * }
+ * }}}
+ * `fs.s3a.proxy.host`/`fs.s3a.proxy.port` resolve via a PLAIN, non-bucket-scoped, non-JCEKS
+ * `Configuration#getTrimmed`/`getInt` call -- NOT `lookupPassword` -- against whatever conf
+ * `S3AFileSystem#initialize` already ran through `propagateBucketOptions` before
+ * `createAwsConf`/`initProxySupport` ever runs; only the SIBLING `fs.s3a.proxy.username`/
+ * `fs.s3a.proxy.password` keys go through `lookupPassword` (bucket long/short/global,
+ * JCEKS-aware). So the host is bucket-aware only via `propagateBucketOptions`'s short-bucket-
+ * form folding, never the long-bucket form, and never a credential-provider read -- the SAME
+ * shape as `endpoint`/`path.style.access` (the "propagated option" family, see
+ * [[AllS3ConfigKeys]]'s doc), not the credential family.
+ *
+ * Resolved here through [[hadoopLookupPasswordEffective]] anyway (long-then-short-then-global,
+ * credential-provider-aware) for the identical reason [[AllS3ConfigKeys]]'s non-credential
+ * entries are: checking MORE than Hadoop's real, narrower resolution can only ever find a value
+ * Hadoop itself ignores, which can only turn an already-necessary decline into one found
+ * slightly more eagerly -- never a missed one. A missed decline is exactly the failure mode
+ * this gate exists to close (native has NO HTTP-proxy support of any kind -- no
+ * `fs.s3a.proxy.*` key is read anywhere in `s3.rs`), so over-approximating is the conservative
+ * direction here, unlike a value-EQUALITY comparator (e.g. [[s3KeyDivergenceReason]]) where
+ * over-approximating could manufacture a false divergence between two sides that would
+ * otherwise agree.
+ */
+ private val S3ProxyHostKey = "fs.s3a.proxy.host"
+
+ private def unsupportedProxyReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key configured for $bucket (the native S3 client has " +
+ "no HTTP proxy support at all -- no fs.s3a.proxy.* key is read anywhere in its object " +
+ "store layer -- so a claimed scan would connect to S3 directly instead of routing through " +
+ "the configured proxy, either bypassing an egress/network-segmentation policy or simply " +
+ "failing to reach the endpoint)"
+
+ /**
+ * First reason any bucket among `uris` has an HTTP proxy configured via [[S3ProxyHostKey]], or
+ * `None` when claimable. Ordered alongside [[unsupportedEncryptionAlgorithmReason]] among the
+ * other conf-only gates that are zero-I/O in the common case, ahead of
+ * [[s3ConfigDivergenceReason]]: [[hadoopLookupPasswordEffective]] only performs real keystore
+ * I/O, inside its own try/catch, when a Hadoop credential-provider path is actually configured
+ * for this bucket (see that function's doc for the Arm A/B/C dispatch). Never interpolates a
+ * resolved value: the proxy HOST is not secret, but naming it here would be a strange place to
+ * first surface it, and proxy CREDENTIALS (`fs.s3a.proxy.username`/`fs.s3a.proxy.password`, not
+ * read by this gate at all -- the whole point of gating on the host is that a non-empty host
+ * declines before any proxy credential would ever need to be forwarded) must never appear in a
+ * decline reason regardless.
+ */
+ private[delta] def proxyGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI],
+ propagatedConfCache: MutableMap[String, Configuration] = MutableMap.empty)
+ : Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ try {
+ val propagatedConf =
+ propagatedConfCache.getOrElseUpdate(
+ bucket,
+ propagateBucketOptions(hadoopConf, bucket))
+ hadoopLookupPasswordEffective(propagatedConf, bucket, S3ProxyHostKey) match {
+ case Left(reason) => Some(reason)
+ case Right(Some(_)) => Some(unsupportedProxyReason(bucket, S3ProxyHostKey))
+ case Right(None) => None
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableValueReason(bucket, S3ProxyHostKey, e))
+ }
+ }
+ }
+ }
+
+ /**
+ * `baseKey`'s long, short, then global per-bucket aliases, in Hadoop's own resolution order.
+ */
+ private def longThenShortThenGlobalAliases(bucket: String, baseKey: String): Seq[String] = {
+ val suffix = baseKey.stripPrefix("fs.s3a.")
+ Seq(s"fs.s3a.bucket.$bucket.fs.s3a.$suffix", s"fs.s3a.bucket.$bucket.$suffix", baseKey)
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * The three-way Hadoop credential-provider-path precheck shared by every `getPassword`-based
+ * resolution below, factored out of what used to be [[hadoopLookupPasswordEffective]]'s own
+ * body so [[effectiveEncryptionAlgorithm]] can reuse it for its own bucket-tier-only and
+ * global-tier-only lookups (see that function's doc for why the algorithm needs its resolution
+ * split into separate tiers rather than one combined long-then-short-then-global list). This
+ * precheck is a property of the BUCKET alone, independent of which base key or alias list a
+ * caller goes on to resolve: Arm A -- an S3A-scoped or bucket-scoped Hadoop credential-provider
+ * path configures a provider `Configuration#getPassword` does not consult
+ * (`S3AUtils#lookupPassword`/`#lookupBucketSecret` build their own S3A-only provider list per
+ * bucket, which this plain-conf-reading gate cannot mirror) -- returns
+ * [[UnverifiableProvider]], NO keystore I/O. Arm B -- only the GLOBAL Hadoop provider path is
+ * set (which `getPassword` DOES consult) -- returns [[GlobalProviderOnly]]; the caller must
+ * wrap its own `getPassword` calls in try/catch, since real keystore I/O only happens once this
+ * arm is reached. Arm C -- no provider path is set anywhere -- returns [[NoProvider]], zero-I/O
+ * plain conf only.
+ */
+ private sealed trait CredentialProviderArm
+ private case class UnverifiableProvider(offendingKey: String) extends CredentialProviderArm
+ private case object GlobalProviderOnly extends CredentialProviderArm
+ private case object NoProvider extends CredentialProviderArm
+
+ private def credentialProviderArm(
+ hadoopConf: Configuration,
+ bucket: String): CredentialProviderArm = {
+ val bucketPathKey = s3aBucketProviderPathKey(bucket)
+ val bucketLongPathKey = s3aBucketLongProviderPathKey(bucket)
+ val s3aPathSet = nonEmptyConf(hadoopConf, S3aCredentialProviderPathKey)
+ val bucketPathSet = nonEmptyConf(hadoopConf, bucketPathKey)
+ val bucketLongPathSet = nonEmptyConf(hadoopConf, bucketLongPathKey)
+ if (s3aPathSet || bucketPathSet || bucketLongPathSet) {
+ val offendingKey =
+ if (s3aPathSet) S3aCredentialProviderPathKey
+ else if (bucketPathSet) bucketPathKey
+ else bucketLongPathKey
+ UnverifiableProvider(offendingKey)
+ } else if (nonEmptyConf(hadoopConf, HadoopCredentialProviderPathKey)) {
+ GlobalProviderOnly
+ } else {
+ NoProvider
+ }
+ }
+
+ /**
+ * `aliases`' effective value under `hadoopConf`/`bucket`, tried in the given order and keeping
+ * the first non-empty result -- or `Left(reason)` when the value cannot be safely verified.
+ * Dispatches on [[credentialProviderArm]]: [[UnverifiableProvider]] declines outright (zero
+ * I/O); [[GlobalProviderOnly]] resolves every alias via `Configuration#getPassword` (real
+ * keystore I/O, contained in try/catch: a corrupt/unreadable store must decline this bucket,
+ * not abort planning for the whole session); [[NoProvider]] resolves every alias via plain
+ * [[plainValue]] reads (zero I/O). Generalizes what used to be
+ * [[hadoopLookupPasswordEffective]]'s entire body to an explicit, caller-supplied alias list --
+ * [[hadoopLookupPasswordEffective]] is now the long-then-short-then-global special case of
+ * this; [[effectiveEncryptionAlgorithm]] is the bucket-tier-only (two aliases) and
+ * global-tier-only (one alias) special cases, matching
+ * `S3AUtils#lookupBucketSecret`/`#lookupPassword`'s real per-tier `getPassword` calls exactly
+ * instead of [[plainValue]]'s JCEKS-blind read that under-declined a keystore-only algorithm.
+ */
+ private def resolveViaCredentialAliases(
+ hadoopConf: Configuration,
+ bucket: String,
+ aliases: Seq[String]): Either[String, Option[String]] =
+ credentialProviderArm(hadoopConf, bucket) match {
+ case UnverifiableProvider(offendingKey) =>
+ Left(s3aScopedProviderPathReason(bucket, offendingKey))
+ case GlobalProviderOnly =>
+ try {
+ Right(
+ aliases.iterator
+ .map(alias =>
+ Option(hadoopConf.getPassword(alias)).map(new String(_)).filter(_.nonEmpty))
+ .collectFirst { case Some(v) => v })
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Left(unverifiableCredentialProviderReason(bucket, e))
+ }
+ case NoProvider =>
+ Right(aliases.flatMap(plainValue(hadoopConf, _)).headOption)
Review Comment:
[P2] Honor disabled clear-text credential fallback without a store
Could the no-provider arm honor `hadoop.security.credential.clear-text-fallback=false`? With no credential-store paths, an obsolete nonempty `fs.s3a.access.key`/`fs.s3a.secret.key` pair, and `fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider`, Hadoop 3.3.4's `getPassword` ignores those plaintext keys. Simple reports no credentials and the ordered chain proceeds to the later provider.
Here, `plainValue` reads the obsolete pair with `get`, reports equality with native, and admits the otherwise eligible Delta scan. Native treats the present static pair as valid, and its credential chain returns it before trying Environment. Given working environment credentials on the driver/executors and a configured pair that cannot access the table, this changes a Hadoop-readable scan into a failing native read.
Please preserve `getPassword` semantics in this arm or decline this setting. Removing the obsolete keys removes the mismatch. Enabling clear-text fallback alone does not make an unauthorized pair valid. This is a source-derived, unexecuted case, and the shortcut was already present at the previous reviewed head.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1627 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Zero-I/O in the common case, like the encryption gate above: native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The three keys where Hadoop's OWN resolver (`S3AUtils#lookupPassword`) is long-bucket-aware,
+ * verified via `javap` against `hadoop-aws` 3.3.4: `S3AUtils#lookupPassword(bucket, conf,
+ * baseKey)` builds `longBucketKey = "fs.s3a.bucket." + bucket + "." + baseKey` (note: the FULL
+ * `baseKey`, already `fs.s3a`-prefixed, appended after the bucket segment) and reads it via
+ * `getPassword` BEFORE the short-bucket key, keeping the long value whenever `getPassword`
+ * returns non-empty for it and only falling through to short-then-global otherwise. Kept as a
+ * named subset purely for documentation/discovery purposes (see
+ * [[hadoopLookupPasswordEffective]]'s doc); every key in [[AllS3ConfigKeys]] below is resolved
+ * through the SAME long-then-short-then-global function regardless of membership here -- see
+ * that function's doc for why extending long-bucket-awareness to keys where it does not
+ * strictly apply is safe.
+ */
+ private val CredentialValueKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ /**
+ * Every `fs.s3a.*` base key that governs whether a claimed native scan actually behaves like
+ * Hadoop's own reader would, checked by [[s3ConfigDivergenceReason]] via
+ * [[hadoopLookupPasswordEffective]] (see that function's doc for the resolution order and the
+ * deliberate over-decline rationale for the non-credential entries below). Two groups, kept
+ * merged into one list and one resolution path -- NOT two lists with two different resolution
+ * mechanisms, which is what let `fs.s3a.encryption.algorithm` silently fall through the
+ * short-then-global-only path while Hadoop's real resolver for it was long-bucket-aware (the
+ * SSE-C long-bucket-alias gap): the ORIGINAL [[CredentialValueKeys]] three, plus every OTHER
+ * per-bucket `fs.s3a.*` base key native's S3 client's `get_config` (s3.rs) resolves, verified
+ * directly against its call sites: `extract_s3_config_options` (endpoint.region,
+ * path.style.access, endpoint, requester.pays.enabled), `lookup_provider_class` (the
+ * Comet-specific credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name).
+ *
+ * SYNC NOTE: this list's non-credential entries must stay a superset of native's
+ * `NATIVE_S3A_CONFIG_PROPERTIES` constant (`native/core/src/parquet/objectstore/s3.rs`,
+ * property suffixes without the `fs.s3a.` prefix) -- `DeltaScanContribSuite`'s
+ * discovery-harness test asserts this mechanically. Literal strings, not the
+ * [[AwsCredentialsProviderKey]] / [[AssumedRoleCredentialsProviderKey]] vals declared below,
+ * purely to avoid a forward reference inside this `object` body; kept textually identical to
+ * those two constants.
+ */
+ private[delta] val AllS3ConfigKeys: Seq[String] = Seq(
+ "fs.s3a.access.key",
+ "fs.s3a.secret.key",
+ "fs.s3a.session.token",
+ "fs.s3a.aws.credentials.provider",
+ "fs.s3a.assumed.role.arn",
+ "fs.s3a.assumed.role.session.name",
+ "fs.s3a.assumed.role.credentials.provider",
+ "fs.s3a.endpoint",
+ "fs.s3a.endpoint.region",
+ "fs.s3a.path.style.access",
+ "fs.s3a.requester.pays.enabled",
+ "fs.s3a.comet.credential.provider.class")
+
+ /**
+ * The short-bucket-then-global value resolved for `baseKey` under `bucket` from `hadoopConf`,
+ * skipping an empty value at either alias exactly like [[plainValue]]. NOT used by
+ * [[s3ConfigDivergenceReason]]/[[s3KeyDivergenceReason]] any more -- every key checked there
+ * resolves through [[hadoopLookupPasswordEffective]] uniformly (long-then-short-then-global),
+ * which is a strict superset of what this function reads. This function's one remaining caller
+ * is [[shortThenGlobalOrReason]], which reads provider-CLASS strings (from the ORIGINAL,
+ * unpropagated conf) for name-support validation in [[providerClassReason]]/
+ * [[assumedRoleProviderClassReason]] -- by the time those run, [[s3ConfigDivergenceReason]] has
+ * already proven Hadoop's and native's effective values agree for the same key, so whichever of
+ * the two (equal) values this narrower read returns does not affect correctness there. NEVER
+ * used to compute native's own effective value -- see [[nativeShortThenGlobal]] for that.
+ */
+ private def shortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket` from the ORIGINAL, unpropagated `hadoopConf` -- `NativeConfig
+ * .extractObjectStoreOptions` forwards `Configuration#get`'s substituted value for every
+ * `fs.s3a.*` entry with no bucket-option propagation step of its own, so the original conf is
+ * the right input here. Unlike [[shortThenGlobal]]/[[plainValue]], this mirrors `get_config`
+ * faithfully: PRESENCE of the short-bucket key alone -- never its emptiness -- decides whether
+ * native falls back to the global key (`get_config` is a plain `HashMap::get`, which returns
+ * `Some` for a key explicitly set to `""`), so an explicitly empty or whitespace-only
+ * short-bucket value resolves to `Some("")` here and never falls through to global -- the
+ * OPPOSITE of Hadoop's own `getPassword`/`lookupPassword` semantics (see
+ * [[hadoopLookupPasswordEffective]]), which treat empty as absent and keep trying the next
+ * alias. The ONLY function used to compute native's effective value in
+ * [[s3KeyDivergenceReason]].
+ *
+ * Deliberately does NOT apply `get_config_trimmed`'s `.trim()` here: [[s3KeyDivergenceReason]]
+ * trims both this value and Hadoop's effective value together, symmetrically, at the point they
+ * are compared, rather than one-sidedly here -- trimming only the native side would flag a
+ * spurious divergence for a value neither side's whitespace actually changes the behavior of
+ * once each side's own downstream parsing normalizes it (e.g. Hadoop's own multi-line
+ * `fs.s3a.aws.credentials.provider` default, which both Hadoop and native additionally trim per
+ * comma-separated entry after splitting), while a one-sided trim would make an
+ * otherwise-identical default value look diverged for every bucket, never claiming natively at
+ * all.
+ */
+ private def nativeShortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ Option(hadoopConf.get(shortKey)).orElse(Option(hadoopConf.get(baseKey)))
+ }
+
+ /**
+ * Faithful in-memory replica of `S3AUtils#propagateBucketOptions` (`hadoop-aws`), which
+ * `S3AFileSystem#initialize` calls FIRST, before any option or credential is read:
+ * `Configuration conf = propagateBucketOptions(originalConf, bucket); ...; setConf(conf);` --
+ * every subsequent `conf.get`/`getPassword` call in that filesystem instance, including
+ * `${...}` variable substitution, resolves against this propagated view, not the original conf.
+ * `hadoop-aws` is not on this module's runtime classpath (see the string-literal-keys note
+ * above), so `S3AUtils#propagateBucketOptions` cannot be called directly; this reproduces its
+ * logic verbatim using only `hadoop-common`'s `Configuration`:
+ * {{{
+ * public static Configuration propagateBucketOptions(Configuration source, String bucket) {
+ * final String bucketPrefix = FS_S3A_BUCKET_PREFIX + bucket + '.';
+ * final Configuration dest = new Configuration(source);
+ * for (Map.Entry<String, String> entry : source) {
+ * final String key = entry.getKey();
+ * final String value = entry.getValue(); // the (unexpanded) value
+ * if (!key.startsWith(bucketPrefix) || bucketPrefix.equals(key)) continue;
+ * final String stripped = key.substring(bucketPrefix.length());
+ * if (stripped.startsWith("bucket.") || "impl".equals(stripped)) {
+ * // ignored
+ * } else {
+ * final String generic = FS_S3A_PREFIX + stripped;
+ * dest.set(generic, value, ...); // overwrites any existing global value
+ * }
+ * }
+ * return dest;
+ * }
+ * }}}
+ * Note the LONG bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) folds to an unread
+ * `fs.s3a.fs.s3a.<key>` key here too, exactly like the real method -- `stripped` already starts
+ * with `fs.s3a.` in that case, so prepending `fs.s3a.` again produces a key nothing ever reads.
+ */
+ private def propagateBucketOptions(hadoopConf: Configuration, bucket: String): Configuration = {
+ val bucketPrefix = s"fs.s3a.bucket.$bucket."
+ val dest = new Configuration(hadoopConf)
+ hadoopConf.iterator().asScala.foreach { entry =>
+ val key = entry.getKey
+ if (key.startsWith(bucketPrefix) && key != bucketPrefix) {
+ val stripped = key.substring(bucketPrefix.length)
+ if (!stripped.startsWith("bucket.") && stripped != "impl") {
+ dest.set(s"fs.s3a.$stripped", entry.getValue)
+ }
+ }
+ }
+ dest
+ }
+
+ /**
+ * Canonical and deprecated Hadoop S3A encryption-algorithm config keys, verified via `javap`
+ * against `hadoop-aws` 3.3.4's `org.apache.hadoop.fs.s3a.Constants`: `S3_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.encryption.algorithm"` (canonical) and `SERVER_SIDE_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.server-side-encryption-algorithm"` (DEPRECATED -- note the hyphen before "algorithm",
+ * unlike the corresponding `*.key` constants below, which both use a `.key` suffix).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, same
+ * rationale as [[HadoopCredentialProviderPathKey]].
+ */
+ private val S3EncryptionAlgorithmKey = "fs.s3a.encryption.algorithm"
+ private val DeprecatedS3EncryptionAlgorithmKey = "fs.s3a.server-side-encryption-algorithm"
+
+ /**
+ * The exact strings `S3AEncryptionMethods#getMethod` accepts, verified via `javap`/CFR against
+ * `hadoop-aws` 3.3.4's `S3AEncryptionMethods` enum: `NONE("")`, `SSE_S3("AES256", serverSide =
+ * true, requiresSecret = false)`, `SSE_KMS("SSE-KMS", serverSide = true, requiresSecret =
+ * false)`, `SSE_C("SSE-C", serverSide = true, requiresSecret = true)`, `CSE_KMS("CSE-KMS",
+ * serverSide = false, requiresSecret = true)`, `CSE_CUSTOM("CSE-CUSTOM", serverSide = false,
+ * requiresSecret = true)`. `getMethod` parses case-insensitively
+ * (`values().find(_.getMethod.equalsIgnoreCase(algorithm))`), matched below the same way.
+ *
+ * ALLOWLIST, not a blocklist (replaces the former SSE-C-only blocklist): only the algorithms S3
+ * decrypts transparently on GET/HEAD given read permission alone, with NO extra request header
+ * and NO client-side step, are safe for a native scan that forwards none of Hadoop's
+ * `fs.s3a.encryption.*`/`fs.s3a.server-side-encryption*` options --
+ * - `AES256` (SSE_S3, `serverSide = true`): plain server-side encryption, transparent on GET.
+ * - `SSE-KMS` (SSE_KMS, `serverSide = true`): server-side, KMS-managed key, transparent on
+ * GET given KMS decrypt permission (no header).
+ * - `DSSE-KMS`: NOT present in this enum on `hadoop-aws` 3.3.4 (confirmed by the six values
+ * listed above) -- `S3AEncryptionMethods.getMethod("DSSE-KMS")` throws
+ * `IOException("Unknown encryption algorithm DSSE-KMS")` on this version, so
+ * `S3AUtils#buildEncryptionSecrets` (and therefore Hadoop's own reader) already fails
+ * before ever reading such a table under 3.3.4, meaning this string can never actually be
+ * the resolved value on the declared target version -- admitting it here is inert there.
+ * Included anyway, forward-compatible, for a newer `hadoop-aws` on the runtime classpath (a
+ * later Hadoop release; this module has no compile-time `hadoop-aws` dependency, see the
+ * string-literal-keys note above) where DSSE-KMS is a real, dual-layer, server-side
+ * algorithm decrypted transparently on GET the same way SSE-KMS is. Every other value
+ * declines: `SSE-C` (SSE_C is `serverSide = true` in Hadoop's own enum, but `requiresSecret
+ * \= true` -- S3 rejects a GET/HEAD for an SSE-C object outright (400 Bad Request) unless
+ * the customer key is resent as a request header on every call, so a native scan that never
+ * learns the key cannot succeed at all, where Hadoop's own reader -- whose request factory
+ * attaches the key -- would), `CSE-KMS`/`CSE-CUSTOM` (`serverSide = false`: client-side
+ * encryption decrypts object bytes locally in the SDK layer, which the native Parquet
+ * reader has no equivalent of -- it would read raw ciphertext), and any future/unknown
+ * value (a value `S3AEncryptionMethods.getMethod` itself would reject is certainly not one
+ * of the three confirmed-transparent algorithms above; declining is the only safe default
+ * for anything this gate cannot positively confirm).
+ */
+ private val AllowedEncryptionAlgorithms: Set[String] = Set("AES256", "SSE-KMS", "DSSE-KMS")
+
+ /**
+ * `bucket`'s effective encryption-algorithm key and value under `hadoopConf`, or `None` when
+ * neither the canonical nor deprecated key is set anywhere consulted. Mirrors
+ * `S3AUtils#buildEncryptionSecrets`'s real resolution order, verified via `javap`/CFR
+ * decompilation of `hadoop-aws` 3.3.4's `S3AUtils.class`:
+ * {{{
+ * String algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.server-side-encryption-algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.server-side-encryption-algorithm");
+ * }}}
+ * i.e. bucket-tier (canonical, then deprecated), THEN global-tier (canonical, then deprecated)
+ * -- the two tiers are never interleaved key-by-key, so this must stay two explicit bucket-tier
+ * lookups followed by two explicit global-tier lookups, not a single
+ * [[hadoopLookupPasswordEffective]] call per key (which would let an unset canonical bucket key
+ * fall through straight to the canonical GLOBAL value ahead of a SET deprecated bucket key, the
+ * wrong answer).
+ *
+ * THE FIX for the SSE-C long-bucket-alias gap is entirely inside the bucket tier:
+ * `lookupBucketSecret` itself is long-then-short, decompiled from `hadoop-aws` 3.3.4's
+ * `S3AUtils.class`:
+ * {{{
+ * // longBucketKey = fs.s3a.bucket.B.fs.s3a.<key>
+ * String longBucketKey = String.format(BUCKET_PATTERN, bucket, baseKey);
+ * String initialVal = getPassword(conf, longBucketKey, null, null);
+ * // shortBucketKey = fs.s3a.bucket.B.<key>
+ * String shortBucketKey = String.format(BUCKET_PATTERN, bucket, subkey);
+ * // keeps initialVal (the LONG value) if non-empty
+ * return getPassword(conf, shortBucketKey, initialVal, null);
+ * }}}
+ * i.e. the SAME long-bucket-key construction and long-wins-if-nonempty semantics as
+ * `S3AUtils#lookupPassword` (see [[CredentialValueKeys]]/[[hadoopLookupPasswordEffective]]) --
+ * the encryption algorithm is NOT one of the keys that flows through
+ * `S3AUtils#propagateBucketOptions` (which folds an unrelated per-bucket LONG form into an
+ * unread key). An earlier version of this function modeled the bucket tier as SHORT-only,
+ * documented as "the LONG bucket form is genuinely never consulted for this key" -- that
+ * documentation was wrong (this decompilation supersedes it): a bucket configured only via
+ * `fs.s3a.bucket.B.fs.s3a.encryption.algorithm=SSE-C` bypassed the SSE-C gate entirely, because
+ * Hadoop's own reader DOES read that long form (and picks SSE-C), while this function reported
+ * `None` (nothing set) and the allowlist check below never even ran.
+ *
+ * The canonical-vs-deprecated distinction below is frequently moot in practice: `hadoop-aws`'s
+ * `S3AFileSystem.addDeprecatedKeys()` statically registers `fs.s3a.server-side-encryption-*` as
+ * `Configuration`-level deprecated aliases of `fs.s3a.encryption.*` (verified via `javap`), a
+ * registration that lives in a static field on Hadoop's `Configuration` class -- process-wide
+ * once `S3AFileSystem`'s class has loaded anywhere in the JVM, which a real scan has always
+ * already done by the time this gate runs, since reading the S3 table at all requires loading
+ * that class. Once active, `Configuration#get` resolves either literal key to the identical
+ * value transparently, making the two-key cascade below redundant (but harmless) for that case;
+ * it remains the operative path only when nothing else in the process has loaded
+ * `S3AFileSystem` yet.
+ *
+ * ALSO walks the Hadoop-credential-provider (JCEKS) path via [[resolveViaCredentialAliases]]
+ * for each of the four lookups below, matching `lookupBucketSecret`/`lookupPassword`'s real
+ * per-alias `getPassword` calls (quoted above) exactly: both are `getPassword`, not plain
+ * `Configuration#get`, so a bucket storing the algorithm name ONLY in a JCEKS keystore is
+ * exactly as real a Hadoop deployment shape for this key as it is for the credential keys
+ * [[hadoopLookupPasswordEffective]] already covers -- there is nothing algorithm-specific that
+ * makes JCEKS storage implausible here, so an earlier version of this function skipping it
+ * (documented at the time as "the algorithm NAME is not credential-sensitive data, so storing
+ * it in a keystore is not a realistic Hadoop deployment pattern") was an unjustified, narrower
+ * read than Hadoop's own resolver actually performs, under-declining a bucket whose algorithm
+ * is keystore-only. [[resolveViaCredentialAliases]]'s Arm B/C split still means this is zero
+ * extra I/O for the common case: keystore I/O only happens when a Hadoop credential-provider
+ * path is actually configured for the bucket, contained in that function's own try/catch.
+ * `bucketTier`/`globalTier` return `Left` (propagated straight through by [[orElseTier]]) when
+ * [[resolveViaCredentialAliases]] cannot safely verify a tier at all (an S3A-scoped provider
+ * path, or a corrupt/unreadable global keystore) -- correctly short-circuiting the whole
+ * cascade with a decline rather than silently falling through to a later tier that might look
+ * unset only because the true value was unverifiable.
+ */
+ private def effectiveEncryptionAlgorithm(
+ hadoopConf: Configuration,
+ bucket: String): Either[String, Option[(String, String)]] = {
+ def bucketTier(baseKey: String): Either[String, Option[(String, String)]] = {
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(longKey, shortKey))
+ .map(_.map(baseKey -> _))
+ }
+ def globalTier(baseKey: String): Either[String, Option[(String, String)]] =
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(baseKey)).map(_.map(baseKey -> _))
+
+ // Short-circuits on Left (unverifiable tier) or Right(Some(_)) (resolved); only Right(None)
+ // (tier definitively unset) falls through to `next`, mirroring buildEncryptionSecrets's
+ // sequential `if (algorithm == null) algorithm = ...` cascade exactly.
+ def orElseTier(
+ current: Either[String, Option[(String, String)]],
+ next: => Either[String, Option[(String, String)]])
+ : Either[String, Option[(String, String)]] =
+ current match {
+ case Left(reason) => Left(reason)
+ case Right(Some(value)) => Right(Some(value))
+ case Right(None) => next
+ }
+
+ orElseTier(
+ bucketTier(S3EncryptionAlgorithmKey),
+ orElseTier(
+ bucketTier(DeprecatedS3EncryptionAlgorithmKey),
+ orElseTier(
+ globalTier(S3EncryptionAlgorithmKey),
+ globalTier(DeprecatedS3EncryptionAlgorithmKey))))
+ }
+
+ private def unsupportedEncryptionAlgorithmDeclineReason(
+ bucket: String,
+ algorithmKey: String,
+ algorithm: String): String =
+ s"Native Delta scan does not support $algorithmKey=$algorithm for $bucket " +
+ "(the native S3 client only supports unencrypted objects and S3's transparent " +
+ "server-side algorithms -- AES256/SSE-S3, SSE-KMS, and DSSE-KMS decrypt on GET/HEAD given " +
+ "read permission alone, with no extra request header; SSE-C additionally requires the " +
+ "customer-provided key resent as a header on every GET/HEAD request, which the native S3 " +
+ "client's extract_s3_config_options never forwards, and CSE-KMS/CSE-CUSTOM decrypt object " +
+ "bytes client-side, a layer the native Parquet reader does not have -- any of these would " +
+ "fail outright or silently read ciphertext where Hadoop's own reader succeeds)"
+
+ /**
+ * First reason any bucket among `uris` is configured for an encryption algorithm the native S3
+ * client cannot safely read, or `None` when claimable. Allowlist-based (see
+ * [[AllowedEncryptionAlgorithms]]): only `AES256`/`SSE-KMS`/`DSSE-KMS` (and unset/empty) pass;
+ * every other resolved value -- `SSE-C`, `CSE-KMS`, `CSE-CUSTOM`, or any unrecognized future
+ * algorithm string -- declines. Deliberately NOT a blocklist keyed on `SSE-C` alone: an
+ * allowlist is safe by construction against a Hadoop release adding a new encryption method
+ * this gate has never heard of, where a blocklist would silently admit it. Never interpolates a
+ * resolved key value, only key names, the bucket, and the (non-secret) algorithm name. Declines
+ * on a `Left` from [[effectiveEncryptionAlgorithm]] too (an unverifiable credential-provider
+ * arm, e.g. an S3A-scoped provider path or a corrupt/unreadable global keystore) -- the
+ * algorithm cannot be ruled safe when it cannot be read at all.
+ */
+ private[delta] def unsupportedEncryptionAlgorithmReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ effectiveEncryptionAlgorithm(hadoopConf, bucket) match {
+ case Left(reason) => Some(reason)
+ case Right(None) => None
+ case Right(Some((key, value))) =>
+ if (!AllowedEncryptionAlgorithms.exists(_.equalsIgnoreCase(value))) {
+ Some(unsupportedEncryptionAlgorithmDeclineReason(bucket, key, value))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Canonical Hadoop S3A HTTP-proxy host config key, verified via CFR decompilation of
+ * `hadoop-aws` 3.3.4's `S3AUtils.class` (`initProxySupport`):
+ * {{{
+ * String proxyHost = conf.getTrimmed("fs.s3a.proxy.host", "");
+ * int proxyPort = conf.getInt("fs.s3a.proxy.port", -1);
+ * if (!proxyHost.isEmpty()) {
+ * ...
+ * String proxyUsername =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.username", null, null);
+ * String proxyPassword =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.password", null, null);
+ * ...
+ * }
+ * }}}
+ * `fs.s3a.proxy.host`/`fs.s3a.proxy.port` resolve via a PLAIN, non-bucket-scoped, non-JCEKS
+ * `Configuration#getTrimmed`/`getInt` call -- NOT `lookupPassword` -- against whatever conf
+ * `S3AFileSystem#initialize` already ran through `propagateBucketOptions` before
+ * `createAwsConf`/`initProxySupport` ever runs; only the SIBLING `fs.s3a.proxy.username`/
+ * `fs.s3a.proxy.password` keys go through `lookupPassword` (bucket long/short/global,
+ * JCEKS-aware). So the host is bucket-aware only via `propagateBucketOptions`'s short-bucket-
+ * form folding, never the long-bucket form, and never a credential-provider read -- the SAME
+ * shape as `endpoint`/`path.style.access` (the "propagated option" family, see
+ * [[AllS3ConfigKeys]]'s doc), not the credential family.
+ *
+ * Resolved here through [[hadoopLookupPasswordEffective]] anyway (long-then-short-then-global,
+ * credential-provider-aware) for the identical reason [[AllS3ConfigKeys]]'s non-credential
+ * entries are: checking MORE than Hadoop's real, narrower resolution can only ever find a value
+ * Hadoop itself ignores, which can only turn an already-necessary decline into one found
+ * slightly more eagerly -- never a missed one. A missed decline is exactly the failure mode
+ * this gate exists to close (native has NO HTTP-proxy support of any kind -- no
+ * `fs.s3a.proxy.*` key is read anywhere in `s3.rs`), so over-approximating is the conservative
+ * direction here, unlike a value-EQUALITY comparator (e.g. [[s3KeyDivergenceReason]]) where
+ * over-approximating could manufacture a false divergence between two sides that would
+ * otherwise agree.
+ */
+ private val S3ProxyHostKey = "fs.s3a.proxy.host"
+
+ private def unsupportedProxyReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key configured for $bucket (the native S3 client has " +
+ "no HTTP proxy support at all -- no fs.s3a.proxy.* key is read anywhere in its object " +
+ "store layer -- so a claimed scan would connect to S3 directly instead of routing through " +
+ "the configured proxy, either bypassing an egress/network-segmentation policy or simply " +
+ "failing to reach the endpoint)"
+
+ /**
+ * First reason any bucket among `uris` has an HTTP proxy configured via [[S3ProxyHostKey]], or
+ * `None` when claimable. Ordered alongside [[unsupportedEncryptionAlgorithmReason]] among the
+ * other conf-only gates that are zero-I/O in the common case, ahead of
+ * [[s3ConfigDivergenceReason]]: [[hadoopLookupPasswordEffective]] only performs real keystore
+ * I/O, inside its own try/catch, when a Hadoop credential-provider path is actually configured
+ * for this bucket (see that function's doc for the Arm A/B/C dispatch). Never interpolates a
+ * resolved value: the proxy HOST is not secret, but naming it here would be a strange place to
+ * first surface it, and proxy CREDENTIALS (`fs.s3a.proxy.username`/`fs.s3a.proxy.password`, not
+ * read by this gate at all -- the whole point of gating on the host is that a non-empty host
+ * declines before any proxy credential would ever need to be forwarded) must never appear in a
+ * decline reason regardless.
+ */
+ private[delta] def proxyGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI],
+ propagatedConfCache: MutableMap[String, Configuration] = MutableMap.empty)
+ : Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ try {
+ val propagatedConf =
+ propagatedConfCache.getOrElseUpdate(
+ bucket,
+ propagateBucketOptions(hadoopConf, bucket))
+ hadoopLookupPasswordEffective(propagatedConf, bucket, S3ProxyHostKey) match {
+ case Left(reason) => Some(reason)
+ case Right(Some(_)) => Some(unsupportedProxyReason(bucket, S3ProxyHostKey))
+ case Right(None) => None
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableValueReason(bucket, S3ProxyHostKey, e))
+ }
+ }
+ }
+ }
+
+ /**
+ * `baseKey`'s long, short, then global per-bucket aliases, in Hadoop's own resolution order.
+ */
+ private def longThenShortThenGlobalAliases(bucket: String, baseKey: String): Seq[String] = {
+ val suffix = baseKey.stripPrefix("fs.s3a.")
+ Seq(s"fs.s3a.bucket.$bucket.fs.s3a.$suffix", s"fs.s3a.bucket.$bucket.$suffix", baseKey)
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * The three-way Hadoop credential-provider-path precheck shared by every `getPassword`-based
+ * resolution below, factored out of what used to be [[hadoopLookupPasswordEffective]]'s own
+ * body so [[effectiveEncryptionAlgorithm]] can reuse it for its own bucket-tier-only and
+ * global-tier-only lookups (see that function's doc for why the algorithm needs its resolution
+ * split into separate tiers rather than one combined long-then-short-then-global list). This
+ * precheck is a property of the BUCKET alone, independent of which base key or alias list a
+ * caller goes on to resolve: Arm A -- an S3A-scoped or bucket-scoped Hadoop credential-provider
+ * path configures a provider `Configuration#getPassword` does not consult
+ * (`S3AUtils#lookupPassword`/`#lookupBucketSecret` build their own S3A-only provider list per
+ * bucket, which this plain-conf-reading gate cannot mirror) -- returns
+ * [[UnverifiableProvider]], NO keystore I/O. Arm B -- only the GLOBAL Hadoop provider path is
+ * set (which `getPassword` DOES consult) -- returns [[GlobalProviderOnly]]; the caller must
+ * wrap its own `getPassword` calls in try/catch, since real keystore I/O only happens once this
+ * arm is reached. Arm C -- no provider path is set anywhere -- returns [[NoProvider]], zero-I/O
+ * plain conf only.
+ */
+ private sealed trait CredentialProviderArm
+ private case class UnverifiableProvider(offendingKey: String) extends CredentialProviderArm
+ private case object GlobalProviderOnly extends CredentialProviderArm
+ private case object NoProvider extends CredentialProviderArm
+
+ private def credentialProviderArm(
+ hadoopConf: Configuration,
+ bucket: String): CredentialProviderArm = {
+ val bucketPathKey = s3aBucketProviderPathKey(bucket)
+ val bucketLongPathKey = s3aBucketLongProviderPathKey(bucket)
+ val s3aPathSet = nonEmptyConf(hadoopConf, S3aCredentialProviderPathKey)
+ val bucketPathSet = nonEmptyConf(hadoopConf, bucketPathKey)
+ val bucketLongPathSet = nonEmptyConf(hadoopConf, bucketLongPathKey)
+ if (s3aPathSet || bucketPathSet || bucketLongPathSet) {
+ val offendingKey =
+ if (s3aPathSet) S3aCredentialProviderPathKey
+ else if (bucketPathSet) bucketPathKey
+ else bucketLongPathKey
+ UnverifiableProvider(offendingKey)
+ } else if (nonEmptyConf(hadoopConf, HadoopCredentialProviderPathKey)) {
+ GlobalProviderOnly
+ } else {
+ NoProvider
+ }
+ }
+
+ /**
+ * `aliases`' effective value under `hadoopConf`/`bucket`, tried in the given order and keeping
+ * the first non-empty result -- or `Left(reason)` when the value cannot be safely verified.
+ * Dispatches on [[credentialProviderArm]]: [[UnverifiableProvider]] declines outright (zero
+ * I/O); [[GlobalProviderOnly]] resolves every alias via `Configuration#getPassword` (real
+ * keystore I/O, contained in try/catch: a corrupt/unreadable store must decline this bucket,
+ * not abort planning for the whole session); [[NoProvider]] resolves every alias via plain
+ * [[plainValue]] reads (zero I/O). Generalizes what used to be
+ * [[hadoopLookupPasswordEffective]]'s entire body to an explicit, caller-supplied alias list --
+ * [[hadoopLookupPasswordEffective]] is now the long-then-short-then-global special case of
+ * this; [[effectiveEncryptionAlgorithm]] is the bucket-tier-only (two aliases) and
+ * global-tier-only (one alias) special cases, matching
+ * `S3AUtils#lookupBucketSecret`/`#lookupPassword`'s real per-tier `getPassword` calls exactly
+ * instead of [[plainValue]]'s JCEKS-blind read that under-declined a keystore-only algorithm.
+ */
+ private def resolveViaCredentialAliases(
+ hadoopConf: Configuration,
+ bucket: String,
+ aliases: Seq[String]): Either[String, Option[String]] =
+ credentialProviderArm(hadoopConf, bucket) match {
+ case UnverifiableProvider(offendingKey) =>
+ Left(s3aScopedProviderPathReason(bucket, offendingKey))
+ case GlobalProviderOnly =>
+ try {
+ Right(
+ aliases.iterator
+ .map(alias =>
+ Option(hadoopConf.getPassword(alias)).map(new String(_)).filter(_.nonEmpty))
+ .collectFirst { case Some(v) => v })
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Left(unverifiableCredentialProviderReason(bucket, e))
+ }
+ case NoProvider =>
+ Right(aliases.flatMap(plainValue(hadoopConf, _)).headOption)
+ }
+
+ /**
+ * `bucket`'s effective value for `baseKey` in Hadoop's own `S3AUtils#lookupPassword` resolution
+ * order -- long bucket alias, then short bucket alias, then global, each tried through a Hadoop
+ * credential provider before falling back to plain conf (see [[resolveViaCredentialAliases]]
+ * for the Arm A/B/C dispatch this delegates to) -- or `Left(reason)` when the value cannot be
+ * safely verified.
+ *
+ * USED FOR EVERY KEY IN [[AllS3ConfigKeys]], not just [[CredentialValueKeys]] -- see that val's
+ * doc for why the structural bug behind the SSE-C long-bucket-alias gap was exactly this split
+ * (some keys resolved long-bucket-aware, others short-only) and why resolving every key through
+ * this SAME long-then-short-then-global function is fail-safe even for the keys (`endpoint`,
+ * `path.style.access`, `requester.pays.enabled`, `comet.credential.provider.class`) whose real
+ * Hadoop resolver is `S3AUtils#propagateBucketOptions` + plain `Configuration#get`, which never
+ * consults the long bucket form or a credential provider at all: checking the long alias (or a
+ * credential-provider path) for one of THOSE keys can only ever find a value real Hadoop itself
+ * ignores. That can make this function return a value differing from Hadoop's TRUE effective
+ * value only in the direction of an EXTRA divergence (a long-form/keystore value set
+ * differently from short/global for a key Hadoop never reads that way) -- which
+ * [[s3KeyDivergenceReason]] turns into an extra DECLINE, never a wrongly-claimed scan, since a
+ * value Hadoop never reads cannot be the reason a wrongly-claimed native scan would misbehave.
+ * A key added to [[AllS3ConfigKeys]] in the future inherits this safety automatically; no
+ * per-key classification decision (and no repeat of the SSE-C bug) is needed.
+ *
+ * `hadoopConf` must be a [[propagateBucketOptions]] result (the caller,
+ * [[s3KeyDivergenceReason]], always passes one) so that `${...}` references embedded in any
+ * alias resolve exactly like `S3AFileSystem#initialize`'s real propagate-then-resolve order.
+ */
+ private def hadoopLookupPasswordEffective(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Either[String, Option[String]] =
+ resolveViaCredentialAliases(
+ hadoopConf,
+ bucket,
+ longThenShortThenGlobalAliases(bucket, baseKey))
+
+ private def effectiveValueDivergenceReason(bucket: String, key: String): String =
+ s"Native Delta scan cannot forward $key for $bucket (Hadoop's effective value for this key " +
+ "differs from what the native S3 client resolves, so its credentials or configuration " +
+ "would differ from Hadoop's)"
+
+ private def unverifiableValueReason(bucket: String, key: String, error: Throwable): String =
+ s"Native Delta scan cannot verify $key for $bucket (Configuration#get raised " +
+ s"${error.getClass.getName}), declining rather than risk forwarding a stale or " +
+ "diverging value"
+
+ /**
+ * `None` when `baseKey`'s Hadoop-effective and native-effective values under `bucket` agree, or
+ * a decline reason naming `baseKey` and `bucket` (never a value) when they diverge or either
+ * side cannot be safely computed.
+ *
+ * Hadoop's effective value is computed against [[propagateBucketOptions]]'s result, mirroring
+ * `S3AFileSystem#initialize`'s actual order (propagate bucket options into the conf FIRST, only
+ * THEN read/substitute options against it), via Hadoop's `lookupPassword`-order read
+ * ([[hadoopLookupPasswordEffective]]) for EVERY key in [[AllS3ConfigKeys]] -- see that
+ * function's doc for why using the same long-then-short-then-global resolution uniformly,
+ * rather than a short-then-global-only read for the keys Hadoop's real resolver does not make
+ * long-bucket-aware, is a deliberate, fail-safe over-approximation. Native's effective value is
+ * always `nativeShortThenGlobal(hadoopConf, ...)` on the ORIGINAL, unpropagated conf, matching
+ * `NativeConfig.extractObjectStoreOptions`'s actual forwarding semantics (no propagation step)
+ * AND native's `get_config` presence-based (not emptiness-based) short-vs-global fallback --
+ * see [[nativeShortThenGlobal]]. Both values are trimmed together, symmetrically, right before
+ * the equality check below (mirroring `get_config_trimmed`'s `.trim()`, which native applies
+ * regardless of which alias it read) rather than trimming [[nativeShortThenGlobal]]'s result on
+ * its own -- see [[nativeShortThenGlobal]]'s doc for why a one-sided trim there would flag a
+ * spurious divergence. Comparing against the propagated view (rather than the original conf, as
+ * an earlier version of this check did) matters because propagation can change what a `${...}`
+ * reference inside one bucket-scoped value resolves to: e.g.
+ * `fs.s3a.bucket.B.access.key=${fs.s3a.custom.ref}` with `fs.s3a.bucket.B.custom.ref=X` and
+ * global `fs.s3a.custom.ref=Y` propagates to `fs.s3a.custom.ref=X` (overwriting the global `Y`)
+ * before the access key's `${...}` reference is ever substituted, so Hadoop resolves `X` while
+ * a check against the unpropagated conf would (wrongly) also see `Y`, the same value native
+ * forwards -- masking a real divergence. Wrapped in try/catch: `Configuration#get` raises
+ * `IllegalStateException` once `${...}` substitution recurses past Hadoop's `MAX_SUBST` bound
+ * (e.g. a two-key mutual reference cycle); declining is safer than crashing planning or
+ * comparing a partially-substituted value.
+ */
+ private def s3KeyDivergenceReason(
+ hadoopConf: Configuration,
+ propagatedConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ try {
+ val hadoopEffective: Either[String, Option[String]] =
+ hadoopLookupPasswordEffective(propagatedConf, bucket, baseKey)
Review Comment:
[P2] Compare plain options with the endpoint Hadoop actually uses
Could this comparison retain the propagated plain lookup for non-password options? For an otherwise eligible primitive, NoMapping, no-DV Delta table at `s3a://data-bucket/table`, consider these raw Hadoop settings, with matching valid credentials and an explicit region:
```properties
fs.s3a.custom.ref=https://store-a.example
fs.s3a.bucket.data-bucket.custom.ref=https://store-b.example
fs.s3a.bucket.data-bucket.endpoint=${fs.s3a.custom.ref}
fs.s3a.bucket.data-bucket.fs.s3a.endpoint=https://store-a.example
```
Hadoop propagates the raw bucket options before its plain `getTrimmed(ENDPOINT)`, so it uses store B. The long alias is ignored by that consumer. This resolver instead takes the long alias A, which equals native's resolution of the short expression against the original configuration. The equality check therefore admits a real endpoint mismatch. If the table's objects are readable only through B, the native scan can fail where the Hadoop reader succeeds.
Removing the ignored long alias exposes B versus A and correctly declines. The previous reviewed implementation also detected this case through its propagated non-credential lookup. Please preserve the actual plain-option semantics while keeping password/alias resolution for credential consumers. This counterexample is source-derived and unexecuted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "ErikBPF (via GitHub)" <gi...@apache.org>.
ErikBPF commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5529490551
erikbogado@gmail.com
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3886361807
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,548 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. Strip the parquet.field.id metadata
+ // createPhysicalSchema also stamps: files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations.
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
Review Comment:
The dash plus ZWJ shape came from hand-rolled bridge rules approximating the word-break automaton, so I stopped approximating: the classifier now probes the running JDK's own BreakIterator per codepoint and both sides run the full word-boundary rule set over that data. Your shape matches Java now, and the sweeps grew to triples of exotic characters plus a million-string fuzz, zero mismatches on 17, 21, and 25.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3881798691
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,548 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. Strip the parquet.field.id metadata
+ // createPhysicalSchema also stamps: files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations.
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
Review Comment:
Rebuilt this so it can't be JDK-dependent anymore: the driver now ships the running JVM's own case tables (about 15 KB, only when case insensitive) and native runs Java's algorithm over them, so whatever JDK executes is the JDK that's matched. Your A1Σ pair reproduced here in both wrong directions and both flip correctly now. Interesting find along the way: Java's final sigma is word-boundary based, not the Unicode rule. The parity suite is self-validating now and passes on 17, 21, and 25 locally, including the full-sweep test that failed for you on 21.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "schenksj (via GitHub)" <gi...@apache.org>.
schenksj commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5369724228
Hey @sunchao / @parthchandra — are you looking to move this work over to @dwsmith1983’s series?
We’re 2 PRs into the 10-PR series now that the contrib modules have been merged. The series fully implements all of the Delta protocols, with all 10k+ Delta test cases passing.
I’m fine either way. I won’t have a huge amount of time to work on this over the next couple of months, so it could go faster with a different attendant, but the PRs are ready to roll.
You can see the series here: https://github.com/schenksj/datafusion-comet/pulls (PRs #5–13).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3835742449
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme (finding 8).
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
Review Comment:
Added the admission fallback. An S3A-scoped provider path declines outright; with only the global path set we claim only when the plain conf values match what Configuration.getPassword resolves, so keystore-only or shadowed credentials fall back to Spark. No provider path configured means zero I/O. Tests build a real JCEKS keystore and cover both arms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4242597134
##########
docs/source/user-guide/latest/delta.md:
##########
@@ -0,0 +1,72 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Delta Lake (experimental)
+
+Comet can execute DSv1 Delta Lake table scans natively. Reads planned by
+delta-spark run through Comet's native Parquet scan, inheriting row-group
+pruning, page-index pruning, and filter pushdown, with deletion vectors
+applied inside the scan.
+
+Support is experimental and explicitly opt-in. Two things are required:
+
+1. The `comet-contrib-delta-spark` contrib jar on the classpath, alongside
Review Comment:
Nit, optional: the release build doesn't pass `-Pdelta` yet (#5882), so there is no published `comet-contrib-delta-spark` jar to put on the classpath. Could this line say the jar has to be built from source for now and point at the README's build steps? The `Delta Lake <delta>` entry in `index.rst` also sets its own title, so the sidebar won't show "(experimental)" the way the page heading does.
##########
dev/ci/compute-changes.py:
##########
@@ -577,6 +604,12 @@
"iceberg_1_9": ["nightly", "label:run-iceberg-tests"],
"iceberg_1_10": ["nightly", "label:run-iceberg-tests"],
"iceberg_1_11": ["queue", "label:run-iceberg-tests"],
+ # Same shape for the Delta contrib suites: Spark 3.5 is the profile the
+ # MinIO and feature-off jobs already pin, so it is the one the queue runs;
+ # 4.0 and 4.1 run nightly. One label opts a pull request into all three.
+ "delta_3_5": ["queue", "label:run-delta-tests"],
Review Comment:
This makes `delta_3_5` a required queue job for most PRs. Replaying this filter over the last 40 merges on `main`, it would have run for 30 of them, the same set as `spark_4_1`. Nothing in the PR tier builds with `-Pdelta`, though. A core change that breaks the contrib passes its own PR run and first fails in the queue, where it gets evicted and the PRs behind it are rebuilt. The contrib calls a lot of core internals (`QueryPlanSerde`, `CometNativeScan`, `CometScanRule`, `CometExecRDD`, `NativeConfig` and more), so I'd expect this to come up. The Rust test you added for the S3 keys covers one case of it, not the general one.
I suggested keeping one Delta profile in the queue back in September, but I'd like to revisit that while the contrib is experimental. Could `delta_3_5` move to the nightly tier with 4.0 and 4.1, still behind `run-delta-tests`? And could the Spark 3.5 strict warnings compile in `pr_build_linux.yml` add the `delta` profile, so a change that breaks the contrib build fails on its own PR? I tried `./mvnw test-compile -Pspark-3.5,delta -Pstrict-warnings -DskipTests` on the merge with `main`, and the contrib compiles cleanly, including the new unused-params lint from #6846.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-6124497104
Thanks @dwsmith1983 - I added to merge queue 🎉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4166658471
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.ListBuffer
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable. Only called when
+ * [[isDeltaScan]] is true. `scanHelper` is the [[CometScanExec]] built to drive
+ * [[CometDeltaNativeScan.convert]] on a claim, reused for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`: it runs at most
+ // once per claim attempt, and not at all for the common non-DV-shape case.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
Review Comment:
The algorithm now resolves on the bucket-propagated view as well as the raw conf, and the scan declines unless both land on the allowlist. Your `${fs.s3a.custom.ref}` shape with the `data-bucket` override declines; the same shape without the override still claims with AES256.
Two related changes in 9951923b9. A credential provider path is read only when every entry is a local keystore (`jceks`, `localjceks`, `bcfks` or `localbcfks` over `file`, with the default `fs.file.impl`) or `user:///`, since S3AFileSystem drops providers backed by other filesystems. And decline reasons no longer print a resolved value, which could carry the customer key through a `${...}` reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5520366084
Follow-up review of `512c9521`.
The revised scope is easier to review. Reusing the shared Unicode implementation and separating the independent core fixes leaves a clearer Delta integration. The following items cover correctness, performance, design, and validation. They can be developed as separate changes, with tracking issues linked for follow-up work.
### Calendar handling
Could the reader resolve calendar policy from each file’s writer metadata and apply rebasing or conservative fallback?
Fresh end-to-end tests against this head reproduced a LEGACY-written DATE `1500-01-01` reading as `1500-01-10` through native Delta. An ancient INT96 timestamp shifted by the same nine days. Modern INT96 and corrected-calendar controls passed, with native execution asserted.
The existing INT96 test uses modern timestamps and therefore does not exercise rebasing. Useful regressions would cover ancient values, mixed writer metadata, and temporal predicates. Setting the read mode to `CORRECTED` alone is insufficient because Spark’s file metadata can override that setting.
### Assumed-role session policies
The provider compatibility check could cover configured session restrictions in addition to provider class names.
Fresh component tests show that global and bucket-specific `fs.s3a.assumed.role.policy` settings pass admission. Hadoop 3.4.1 includes this policy in its STS request, while the native provider does not forward it. Switching to native therefore drops the configured session restriction.
Declining these configurations until native support is available would preserve the contract. Global, bucket-specific, and unset-policy controls would pin the behavior. This finding was verified through current-source admission tests and the actual Hadoop implementation, without a live STS request.
### DV file splitting and split-aware preparation
A separate improvement could enable splitting large DV-backed files to increase scan parallelism.
The current admitted DV path carries Spark’s row-index field, which causes the shared planner to disable file splitting. Consequently, repeated whole-file preparation across ordinary DV splits is not an established problem in the current execution path.
If native DV splitting is enabled, access-plan construction should prepare only owned row groups while preserving file-global row positions, whole-file corruption validation, and memory accounting. Dictionary-page ownership, half-open boundaries, and interaction with pruning deserve explicit regressions. Descriptor reuse can then be evaluated where multiple splits actually share a DV.
### Compressed deletion-vector decoding
The native-format decoder currently enumerates each deleted position and reinserts it into a treemap. Preserving compressed bitmap containers during conversion could avoid that additional pass.
Evaluate this with sparse, contiguous, and alternating deletions, measuring decoding time and allocations. Portable-format decoding already uses direct deserialization. This optimization would not eliminate the later iteration needed to construct partial-row-group selections.
### Overlapping bitmap and footer reads
For on-disk DVs, bitmap and Parquet footer reads are independent, but preparation currently waits for bitmap fetching, decoding, and validation before starting the footer read.
Overlapping these operations could reduce remote-storage preparation latency. Retain bounded concurrency across files and preserve cancellation, error handling, and memory reservations. Measure peak memory alongside latency because overlapping operations can increase concurrent resource use. Inline DVs have no bitmap network read to overlap.
### Shared cloud compatibility checks
Cloud compatibility policy could live in a focused helper beside the shared object-store configuration code. Delta admission would then concentrate on protocol features, schemas, and Delta metadata.
Both native Parquet and native Delta use the same underlying client, so shared checks would help compatibility fixes apply consistently. Preserve Hadoop’s distinct rules for plain options, password-backed values, variable substitution, and bucket overrides. This should simplify existing decisions without introducing another provider abstraction.
### Credential lifecycle and scope
Could credential extraction happen during execution-payload construction, after final file selection, and include only the required data-file and DV authorities?
The current planning payload retains options selected by provider-wide prefixes. Deferring and scoping extraction would reduce credential retention and unnecessary payload content.
Tests could verify that planning payloads contain no credentials, execution payloads contain only the selected scopes, and credentials required by external DV locations remain available. No credential leak or performance improvement was demonstrated by this review.
### v2-checkpoint regression coverage
The v2-checkpoint test could verify actual checkpoint artifacts, include a sidecar-backed case, invalidate the Delta log cache, and reopen the snapshot before comparing Spark and native results.
The current fixture performs several commits and then reads immediately. It does not establish that the read reconstructs the snapshot from the checkpoint rather than using cached state.
A post-checkpoint append and time-travel read would further exercise the boundary between Delta’s snapshot reconstruction and native table-data scanning.
### Supported-feature documentation
A capability table would make the experimental reader’s scope easier to understand. It should describe native support and fallback boundaries for ID mapping, mapped nested schemas, CDF, and generated row metadata.
Clarify that constant file metadata and consumed row indexes have different support rules. Link follow-up issues for additional coverage so users can distinguish deliberate fallback from planned extensions.
### Benchmarks and validation
A focused benchmark should cover large files and sparse, contiguous, and alternating deletion patterns. Report preparation time, decoding allocations, reserved and peak memory, total scan time, output equality, and confirmed native execution. Split-file and descriptor-reuse benchmarks should accompany any change enabling native DV splitting.
The previously published benchmark used another head plus optimization commits, so it does not establish this revision’s performance.
Fresh validation on Spark 4.0.4 / Delta 4.0.1 passed 216 existing Delta tests. Three MinIO tests were canceled because Docker was unavailable. Additional checks passed 386 DV component scans and 30 planning probes. The separate historical-calendar regressions reproduced the mismatches described above.
Results for the remaining supported Spark profiles and live object-store tests would complete the validation picture. GitHub reported no CI checks attached to this head at review time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "ErikBPF (via GitHub)" <gi...@apache.org>.
ErikBPF commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5526237994
> @andygrove @sunchao @ErikBPF @comphead do you all have a Slack channel we can use to plan?
Sorry. Posted before from work account. Original message:
No, but I am open to using one if you guys have one. Also open to using discord, since free slack caps message history
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3831221444
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,587 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::ParquetAccessPlan;
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::ObjectStoreExt;
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::PageIndexPolicy;
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::parquet::parquet_support::prepare_object_store_with_configs;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
Review Comment:
Reworked as you suggested. Before build_access_plan runs (row-group count is known from the footer at that point), we reserve a construction bound of 3 x (2*cardinality + row_groups) x size_of::<RowSelector>(); the factor is derived, not guessed: with r selectors retained from finished groups and c in the current group, r + c <= the per-file bound, and the peak is r + 2c (source Vec at doubling capacity) + c (the RowSelection::from copy) <= 3(r + c). After the build we resize down to 2 x the exact retained bytes, the second factor covering create_initial_plan's deep clone alongside the original in the file extensions; since retained <= bound this always shrinks. A rejection now happens before any large allocation, with a distinct construction-phase error, and a test pins that ordering (pool sized to the steady state, below the construction bound). Two things to be upfront about: the bound is worst-case-alternating, so a large contiguous delete now reserves far more than it retains
(your 1M example would reserve ~96MB to keep ~48 bytes) and rejection is an execution-time error rather than a planning fallback; the exact selector count is computable in O(runs) from the already-decoded bitmap, so making the reservation exact is a clean follow-up if you'd like it. The DV blob and decoded bitmap themselves (~250KB at the default cap) remain outside the pool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3831646182
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,613 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.net.URI
+import java.util.Locale
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.CometScanRule
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands. Anything else on the protocol declines the
+ * table. Note `deletionVectors` and `columnMapping` are declined separately (below) so their
+ * fallback reasons are specific.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format (not a further subclass)? Compared by class name,
+ * not `classOf`, deliberately: this is the first gate on every V1 scan, and it must stay inert
+ * when the contrib jar is deployed without delta-spark on the classpath:
+ * `classOf[DeltaParquetFileFormat]` here raises NoClassDefFoundError inside CometScanRule and
+ * takes down every parquet scan in the session. When the name matches, delta-spark is
+ * necessarily present (the instance exists), so the Delta types past this gate are safe.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Returns the first reason this Delta scan cannot go native, or None when it is claimable. Only
+ * called when [[isDeltaScan]] is true. `scanHelper` is the same [[CometScanExec]] the caller
+ * builds to drive [[CometDeltaNativeScan.convert]] on a claim, reused here (rather than listed
+ * separately) to resolve the scan's selected files for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Descriptor deserialization is the expensive part of both consumers below (the DV
+ // cardinality gate and the store-identity collision gate); hoisted once here so it runs at
+ // most once per claim attempt regardless of how many gates end up needing it. `lazy` because
+ // most scans are not DV-shaped and selectedDvDescriptors short-circuits to Seq.empty for
+ // them, but paying even that check is unnecessary work for gates that return earlier.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles with useMetadataRowIndex=false) injects a
+ // generated row-index column directly into the data schema and disables reader
+ // optimizations; its values must come from Spark's reader. Claiming such a scan would
+ // feed NULL row indexes into deletion-vector construction, silently corrupting DML.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ // Name mode is supported by serializing physical-name schemas (the parquet reader then
+ // matches file columns by name natively). Id mode needs the field-id path and stays
+ // declined until validated.
+ val cmMode = metadata.columnMappingMode.name
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping
+ // EXISTS_DEFAULT; decline any column defaults under column mapping rather than
+ // return nulls where a default belongs.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names (not just top-level column
+ // names) to their physical, column-mapped form, and the shared native builder uses the
+ // required schema verbatim as the scan's output schema: struct fields below the top level
+ // would carry physical names. Ordinal access (GetStructField) is unaffected, but
+ // name-sensitive native expressions (e.g. to_json) read the Arrow struct field names
+ // directly and would leak physical names into query results. Restoring logical names
+ // natively needs a rename adapter/proto field for the logical schema (follow-up); decline
+ // until then.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and not
+ // supported, except Delta's DV bookkeeping columns, which the native path emits as
+ // constants (correct by construction once the DV is applied in the reader).
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is not a DV read: it is Delta DML
+ // bookkeeping (findTouchedFiles building deletion bitmaps from REAL row indexes).
+ // Claiming it with constant row indexes would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // The internal columns must form a suffix of the read schema so data-column
+ // positions agree between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // The row-index column's real values are consumed inside the reader when Spark applies
+ // the DV; native applies the DV itself and emits a dead constant instead, so the value
+ // must be provably unused above the scan (beyond the _metadata reassembly that gets
+ // discarded).
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet; decline rather
+ // than silently return nulls for backfilled columns in old files.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bound the memory the native side will retain for expanded DV row selectors before
+ // committing to native execution: applying a deletion vector expands it into per-row
+ // RowSelectors that are reserved against the execution memory pool at scan time (see
+ // delta_dv.rs), and an alternating deleted/retained bitmap produces one non-coalescing
+ // selector per row. A row group's selector count is bounded above by
+ // 2*cardinality + #row-groups (each deleted row splits at most one run into a
+ // select/skip pair, plus one selector per row-group boundary), so the descriptor's
+ // cardinality -- deserialized at planning time via selectedDvDescriptors, no bitmap
+ // decode needed -- is a sound, pessimistic upper bound on the native reservation.
+ // Pessimistic by design: a large but CONTIGUOUS deletion is declined the same as a
+ // large alternating one, even though it would retain far fewer selectors natively; the
+ // conf below makes that recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from InputFileBlockHolder, a thread-local set by Spark's
+ // FileScanRDD; the native scan does not populate it. Delta's own DELETE/UPDATE/MERGE
+ // find-touched-files scans use input_file_name, so this gate is load-bearing for DML
+ // correctness (mirrors core's check in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core).
+ // The DV shape's trailing row-index column is exempt: the gates above already proved its
+ // values are dead and the native path emits a constant for it.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does
+ // not yet wire the decryption-key broadcast to executors, so claiming even a
+ // supported-encryption scan would fail at execution.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults (schema-evolution backfill of map/struct/array columns)
+ // cannot be serialized; a silently-dropped default would misalign the value/index lists
+ // consumed positionally on the native side. Mirrors core's transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read; otherwise a custom Hadoop FileSystem would fail at execution instead of
+ // falling back gracefully. Mirrors core's unsupportedFsSchemes gate.
+ val libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+ val unsupportedFsSchemes = scanExec.relation.location.rootPaths
Review Comment:
**[P2] Check selected-file schemes before claiming a shallow clone**
Could we apply this filesystem gate to the selected data-file URIs, not just the table's `rootPaths`? A valid Delta shallow clone can have a supported `file:` table root while its data files still reference `viewfs://review-mount/source/table/...`. With the default libhdfs scheme set (`hdfs` only), both authority checks accept these same-authority files, and the ordinary `LongType` scan serializes successfully, so the contrib claims it. Native store preparation then fails with `Generic URL error: Unable to recognise URL "viewfs://..."` instead of leaving the scan with Spark.
At `bc98657f`, a local-only Spark 4.0.2 / Delta 4.0.0 probe wrote a Delta table through Hadoop's built-in viewfs mount, shallow-cloned it to a local directory, and successfully read `[0, 1, 2]`. Its actual scan had a `file:` root and `viewfs:` selected files. The exact-current authority helpers accepted those files, while the exact native store-preparation helper rejected their URI. This was a stock-engine/exact-helper probe, not a full Comet/JNI run. Checking the schemes of the files actually selected before claiming would preserve Spark fallback for this valid table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3826293560
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,587 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::ParquetAccessPlan;
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::ObjectStoreExt;
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::PageIndexPolicy;
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::parquet::parquet_support::prepare_object_store_with_configs;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
+ }
+ cursor = row + 1;
+ }
+ if deleted_in_group == num_rows && num_rows > 0 {
+ plan.skip(idx);
+ } else if deleted_in_group > 0 {
+ if group_end > cursor {
+ selectors.push(RowSelector::select((group_end - cursor) as usize));
+ }
+ plan.scan_selection(idx, RowSelection::from(selectors));
+ }
+ group_start = group_end;
+ }
+ // A deleted index beyond the file's total row count means the DV does not
+ // belong to this file (stale or corrupted metadata); silently dropping it
+ // would under-apply deletions.
+ if let Some(&row) = deleted_iter.peek() {
+ return Err(GeneralError(format!(
+ "Deletion vector marks row {row} but the file only has {group_start} rows"
+ )));
+ }
+ Ok(plan)
+}
+
+/// One data file plus everything needed to apply its deletion vector. The
+/// file's size comes from `file.object_meta.size` (built by the planner from
+/// the proto's `file_size`).
+pub struct DvScanFile {
+ pub file: PartitionedFile,
+ /// Full URL of the data file (proto `file_path`).
+ pub file_path: String,
+ pub dv: Option<DeltaSparkDvDescriptor>,
+}
+
+/// Upper bound on concurrent DV-blob and footer fetches per partition. Both
+/// are small ranged reads, so a modest fan-out hides object-store latency
+/// without flooding the store client.
+const DV_FETCH_CONCURRENCY: usize = 8;
+
+/// Called via `block_on` at plan-creation time on the executor task: DV blobs
+/// are small ranged reads and footers are needed to learn row-group
+/// boundaries. Files are fetched concurrently (bounded by
+/// [`DV_FETCH_CONCURRENCY`]) with input order preserved. Footer fetches go
+/// through the scan's shared FileMetadataCache, so the scan's subsequent open
+/// of the same file is served from cache. That reuse relies on each input
+/// [`PartitionedFile`] being returned as-is (only `with_extension` applied),
+/// never rebuilt: the cache entry is keyed by this exact `object_meta` and the
+/// scan later looks it up through the same struct.
+pub async fn attach_access_plans(
+ runtime_env: Arc<RuntimeEnv>,
+ object_store_options: &HashMap<String, String>,
+ files: Vec<DvScanFile>,
+) -> Result<Vec<PartitionedFile>, ExecutionError> {
+ futures::stream::iter(files)
+ .map(|scan_file| {
+ attach_access_plan(Arc::clone(&runtime_env), object_store_options, scan_file)
+ })
+ .buffered(DV_FETCH_CONCURRENCY)
+ .try_collect()
+ .await
+}
+
+/// Resolve one file's deletion vector into an attached [`ParquetAccessPlan`];
+/// files without a DV pass through untouched.
+async fn attach_access_plan(
+ runtime_env: Arc<RuntimeEnv>,
+ object_store_options: &HashMap<String, String>,
+ scan_file: DvScanFile,
+) -> Result<PartitionedFile, ExecutionError> {
+ let DvScanFile {
+ file,
+ file_path,
+ dv,
+ } = scan_file;
+ let dv = match dv {
+ Some(dv) => dv,
+ None => return Ok(file),
+ };
+ if dv.size_in_bytes < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative size {}",
+ dv.size_in_bytes
+ )));
+ }
+
+ let data: Vec<u8> = if let Some(inline) = dv.inline_data {
+ inline
+ } else if let Some(dv_path) = &dv.absolute_path {
+ let offset = dv
+ .offset
+ .ok_or_else(|| GeneralError("On-disk deletion vector missing offset".into()))?;
+ if offset < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative offset {offset}"
+ )));
+ }
+ let offset = offset as u64;
+ // [i32 BE size][data: size_in_bytes][i32 BE crc]
+ let framed_len = 4 + dv.size_in_bytes as u64 + 4;
+ let (dv_url, dv_store_path) = prepare_object_store_with_configs(
+ Arc::clone(&runtime_env),
+ dv_path.clone(),
+ object_store_options,
+ )?;
Review Comment:
Fixed by pre-resolution: all stores (data and DV authorities) are resolved on the JNI thread before entering the runtime, and `attach_access_plans` no longer takes an options map or imports the store builder at all, so the async path structurally can't construct one. Your distinct data/DV bucket scenario is encoded in a new MinIO suite (`CometDeltaS3Suite`), but heads up that it's docker-gated and hasn't run against a live daemon yet, the contrib CI job has no docker socket so those tests cancel. First live signal needs a Docker environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3826283650
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ val common = commonBuilder.build()
+ val tableRoot = relation.location.rootPaths.head.toString
+ val deltaCommon = OperatorOuterClass.DeltaSparkScanCommon
+ .newBuilder()
+ .setTableRoot(tableRoot)
+ .setColumnMappingMode(columnMappingMode(scanExec))
+ .setSourceKey(DeltaPlanDataInjector.sourceKey(tableRoot, common))
+ .build()
+ val deltaScan = OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaCommon)
+ Operator
+ .newBuilder()
+ .setPlanId(scanExec.id)
+ .setContribScan(DeltaSparkScanEnvelope.pack(deltaScan.build()))
+ .build()
+ }
+ }
+
+ /**
+ * Harvest subquery-bearing predicates for this scan from its covering FilterExec. Spark 3.x
+ * strips them from a scan's `dataFilters` at planning (`FileSourceStrategy` routes them to the
+ * post-scan filter only), while Spark 4.x keeps them in `dataFilters`. Collecting them here at
+ * claim time gives the execution-time resolve-and-push path the same inputs on every Spark
+ * version; the dedup keeps Spark 4.x from carrying duplicates.
+ */
+ def subqueryFiltersFromParent(
+ plan: org.apache.spark.sql.execution.SparkPlan,
+ scanExec: FileSourceScanExec): Seq[org.apache.spark.sql.catalyst.expressions.Expression] = {
+ import org.apache.spark.sql.catalyst.expressions.{PlanExpression, SubqueryExpression}
+ // Nearest FilterExec above the scan (the DV shape interposes nodes between them, so do
+ // not require a direct parent-child edge). Safety comes from the reference guard, not the
+ // plan walk: only conjuncts expressed directly over the scan's own output attributes
+ // survive, so filters from other branches or over aliased projections contribute nothing.
+ val filtersAboveScan = plan.collect {
+ case f: org.apache.spark.sql.execution.FilterExec if f.find(_ eq scanExec).isDefined => f
+ }
+ filtersAboveScan.lastOption
+ .map { f =>
+ splitConjunctivePredicates(f.condition)
+ .filter(_.references.subsetOf(scanExec.outputSet))
Review Comment:
Fixed. The harvest now walks the plan from the covering filter down to the scan and only accepts Project and Filter nodes in between, anything else stops it, plus a deterministic check on the conjuncts. Your exact query shape is a regression test now: empty result, scan still native, and an assertion that nothing was pushed. The covering filter is never removed, so this only prevents evaluating the predicate too early.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3835076676
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, SparkPlan}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so
+ * `CometPlanAdaptiveDynamicPruningFilters`'s rewrite (via [[CometScanWithPlanData]]) survives
+ * plan copies, the lesson from CometIcebergNativeScanExec (a transient field is dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node, our
+ // own withDynamicPruningFilters, or a generic Catalyst expression rewrite going through
+ // TreeNode.makeCopy, automatically gets a helper consistent with ITS runtimeFilters. A
+ // stored helper field would desync from rewritten filters (the #3510 class of bug). The
+ // cost is that file listing runs once per executed instance (planning listed separately in
+ // the rule extension); correctness over the duplicate driver-side listing.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ override lazy val outputPartitioning: Partitioning =
+ UnknownPartitioning(perPartitionData.length)
Review Comment:
**[P2] Avoid executing adaptive pruning while inspecting partitioning**
This getter forces `perPartitionData`, which calls `InSubqueryExec.updateResult()`. During AQE, that subquery can still be a non-executable adaptive broadcast placeholder.
A reduced Spark 4.0.2 / Delta 4.0.0 planning harness reproduced this through Spark's normal AQE validation: a DPP join in one `UNION ALL` branch and a coalescible shuffle in another caused validation to inspect this partitioning before the custom DPP rewrite. It then failed with `CometSubqueryAdaptiveBroadcastExec ... does not support the execute() code path`. Other operators remained on Spark, and no native Comet reader executed.
Could we return `UnknownPartitioning(0)` while adaptive placeholders remain and make this a non-lazy `def`, so the temporary value is not cached? A regression with a query-time dimension filter would help. The current DPP test filters the dimension before writing it, so it does not require dynamic pruning.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme (finding 8).
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
Review Comment:
**[P2] Fall back when native S3 cannot preserve Hadoop credential aliases**
Could we check authentication compatibility before claiming these scans? With `SimpleAWSCredentialsProvider` explicitly selected and the access/secret keys stored only in JCEKS, Hadoop resolves the credentials through `Configuration.getPassword`, but this extraction forwards neither the credentials nor the global credential-provider path. The native Simple provider therefore has no credentials.
A local probe verified that Hadoop resolves both aliases while the current extraction omits them. The underlying native limitation predates this PR, but claiming Delta scans exposes it to reads that previously used Hadoop successfully.
A conservative admission fallback, with a focused credential-alias test, would be sufficient here. Full credential-provider support can follow separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3886364795
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.ListBuffer
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable. Only called when
+ * [[isDeltaScan]] is true. `scanHelper` is the [[CometScanExec]] built to drive
+ * [[CometDeltaNativeScan.convert]] on a claim, reused for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`: it runs at most
+ // once per claim attempt, and not at all for the common non-DV-shape case.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
Review Comment:
You were right, encryption settings resolve through lookupPassword and the long form wins, my earlier read of that was wrong. The gate now resolves every key that way (for plain options that direction can only over-decline, which is the safe side) and the algorithm check flipped to an allowlist: unset, AES256, SSE-KMS, DSSE-KMS pass, everything else falls back. Your long-form shape declines now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3886372102
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -193,11 +688,16 @@ fn remap_physical_schema(
}
// Block accidental name match for ID-bearing logical fields whose ID is missing
- // from the file. Mirrors Spark's `generateFakeColumnName` in `matchIdField`.
+ // from the file. Mirrors Spark's `generateFakeColumnName` in `matchIdField`. Runs
+ // regardless of `case_sensitive` (field-ID matching is independent of Spark's
+ // casing config), so `case_tables` may be absent here even when it would be
+ // populated for a genuinely case-insensitive scan -- `names_equal_ignore_case_java`
+ // degrades to `str::to_lowercase` in that case, which is fine for this defensive,
+ // over-approximating check (see `java_lowercase`).
if should_match_by_id
&& unmatched_id_logical_names
.iter()
- .any(|name| name.eq_ignore_ascii_case(field.name()))
+ .any(|name| names_equal_ignore_case_java(name, field.name(), case_tables))
Review Comment:
Fixed, the shield compares exactly under case sensitivity and only fires for fields the name pass didn't claim, matching clipParquetGroupFields. Auditing the neighboring comparisons turned up four more divergences from Spark's semantics (a stray ID-less column could win an exact-name lookup for an ID-matched field, the fake name could collide with a real column, and two duplicate-handling mismatches), all fixed with tests against real files.
One thing from that which was found: validateIcebergFileScanTasks keys on a hardcoded scheme list including gcs, oss, wasb, and wasbs, which the native url parser doesn't recognize, so those Iceberg tables get claimed and then fail at execution. Left untouched here since it's the Iceberg path, but it's the same class of drift this PR kept hitting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3883807851
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -193,11 +688,16 @@ fn remap_physical_schema(
}
// Block accidental name match for ID-bearing logical fields whose ID is missing
- // from the file. Mirrors Spark's `generateFakeColumnName` in `matchIdField`.
+ // from the file. Mirrors Spark's `generateFakeColumnName` in `matchIdField`. Runs
+ // regardless of `case_sensitive` (field-ID matching is independent of Spark's
+ // casing config), so `case_tables` may be absent here even when it would be
+ // populated for a genuinely case-insensitive scan -- `names_equal_ignore_case_java`
+ // degrades to `str::to_lowercase` in that case, which is fine for this defensive,
+ // over-approximating check (see `java_lowercase`).
if should_match_by_id
&& unmatched_id_logical_names
.iter()
- .any(|name| name.eq_ignore_ascii_case(field.name()))
+ .any(|name| names_equal_ignore_case_java(name, field.name(), case_tables))
Review Comment:
[P2] Respect case sensitivity when shielding unmatched field IDs
For an ordinary case-sensitive V1 Parquet scan with field-ID reading enabled, request nullable BIGINT fields `Κ` (U+039A, ID 1) and `κ` (U+03BA, no ID), and read a file containing only `κ` with ID 2 and value 7. Spark null-fills the missing ID 1 field but reads the ID-less field by its exact name. This unconditional case-folded check instead matches physical `κ` to unmatched logical `Κ` and replaces the real column with a fake name; the final nullable projection yields `(NULL, NULL)` instead of `(NULL, 7)`. File ID 2 satisfies the file-ID presence guard, and the names are distinct with case sensitivity enabled. Requested BASE's ASCII comparison does not equate this Greek pair. Please avoid hiding a physical field needed by a legitimate exact-name match. This regression exists at the previous reviewed HEAD too, so it is new to the discussion, not newly introduced by the latest increment.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.ListBuffer
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable. Only called when
+ * [[isDeltaScan]] is true. `scanHelper` is the [[CometScanExec]] built to drive
+ * [[CometDeltaNativeScan.convert]] on a claim, reused for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`: it runs at most
+ // once per claim attempt, and not at all for the common non-DV-shape case.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for SSE-C
+ // before the credential-divergence gates below, which do not otherwise notice this table is
+ // readable through Hadoop only because Hadoop's request factory attaches the customer key
+ // native never learns about.
+ val sseCReason = sseCustomerKeyReason(hadoopConf, dataFileUris ++ dvUris)
+ if (sseCReason.isDefined) {
+ return sseCReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason = s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The three per-bucket base keys where Hadoop's own effective value CAN differ from
+ * short-then-global: `S3AUtils#lookupPassword` additionally honors the long bucket form
+ * (`fs.s3a.bucket.B.fs.s3a.<key>`) and Hadoop credential providers (JCEKS et al.) for exactly
+ * these three, and only these three -- every other `fs.s3a` option instead flows through
+ * `S3AUtils#propagateBucketOptions`, which folds the long form into an unread key
+ * (`fs.s3a.fs.s3a.<key>`) and leaves Hadoop's own effective value at short-then-global already,
+ * identical to native's.
+ */
+ private val CredentialValueKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ /**
+ * Every OTHER per-bucket `fs.s3a.*` base key native's S3 client's `get_config` (s3.rs)
+ * resolves, verified directly against its call sites: `extract_s3_config_options`
+ * (endpoint.region, path.style.access, endpoint, requester.pays.enabled),
+ * `lookup_provider_class` (the Comet-specific credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name). Every one of
+ * these resolves short-bucket-then-global inside `get_config` -- s3.rs never reads the long
+ * bucket form. Literal strings, not the [[AwsCredentialsProviderKey]] /
+ * [[AssumedRoleCredentialsProviderKey]] vals declared below, purely to avoid a forward
+ * reference inside this `object` body; kept textually identical to those two constants.
+ */
+ private val OtherS3ConfigKeys: Seq[String] = Seq(
+ "fs.s3a.aws.credentials.provider",
+ "fs.s3a.assumed.role.arn",
+ "fs.s3a.assumed.role.session.name",
+ "fs.s3a.assumed.role.credentials.provider",
+ "fs.s3a.endpoint",
+ "fs.s3a.endpoint.region",
+ "fs.s3a.path.style.access",
+ "fs.s3a.requester.pays.enabled",
+ "fs.s3a.comet.credential.provider.class")
+
+ private val AllS3ConfigKeys: Seq[String] = CredentialValueKeys ++ OtherS3ConfigKeys
+
+ /**
+ * The short-bucket-then-global value resolved for `baseKey` under `bucket` from `hadoopConf`,
+ * skipping an empty value at either alias exactly like [[plainValue]]. Used on a
+ * [[propagateBucketOptions]] result, this is (for the non-credential keys in
+ * [[OtherS3ConfigKeys]]) Hadoop's own effective value, since real Hadoop reads those options as
+ * plain `Configuration#get(baseKey)` calls against its propagated conf, which already folded
+ * any per-bucket override into `baseKey` itself -- so re-checking the (untouched) short bucket
+ * key here first is a no-op, not a second resolution mechanism, and gives the identical value
+ * `Configuration#get(baseKey)` alone would. Also used by [[shortThenGlobalOrReason]] to read
+ * provider-CLASS strings from the original conf for name-support validation; the earlier
+ * [[s3ConfigDivergenceReason]] gate (which uses [[nativeShortThenGlobal]], not this function,
+ * to model native's actual resolution) already runs ahead of that check and would catch any
+ * divergence this skip-empty read models imprecisely. NEVER used to compute native's own
+ * effective value any more -- see [[nativeShortThenGlobal]] for that.
+ */
+ private def shortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket` from the ORIGINAL, unpropagated `hadoopConf` -- `NativeConfig
+ * .extractObjectStoreOptions` forwards `Configuration#get`'s substituted value for every
+ * `fs.s3a.*` entry with no bucket-option propagation step of its own, so the original conf is
+ * the right input here. Unlike [[shortThenGlobal]]/[[plainValue]], this mirrors `get_config`
+ * faithfully: PRESENCE of the short-bucket key alone -- never its emptiness -- decides whether
+ * native falls back to the global key (`get_config` is a plain `HashMap::get`, which returns
+ * `Some` for a key explicitly set to `""`), so an explicitly empty or whitespace-only
+ * short-bucket value resolves to `Some("")` here and never falls through to global -- the
+ * OPPOSITE of Hadoop's own `getPassword`/`lookupPassword` semantics (see
+ * [[hadoopCredentialEffective]]), which treat empty as absent and keep trying the next alias.
+ * The ONLY function used to compute native's effective value in [[s3KeyDivergenceReason]].
+ *
+ * Deliberately does NOT apply `get_config_trimmed`'s `.trim()` here: [[s3KeyDivergenceReason]]
+ * trims both this value and Hadoop's effective value together, symmetrically, at the point they
+ * are compared, rather than one-sidedly here -- trimming only the native side would flag a
+ * spurious divergence for a value neither side's whitespace actually changes the behavior of
+ * once each side's own downstream parsing normalizes it (e.g. Hadoop's own multi-line
+ * `fs.s3a.aws.credentials.provider` default, which both Hadoop and native additionally trim per
+ * comma-separated entry after splitting), while a one-sided trim would make an
+ * otherwise-identical default value look diverged for every bucket, never claiming natively at
+ * all.
+ */
+ private def nativeShortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ Option(hadoopConf.get(shortKey)).orElse(Option(hadoopConf.get(baseKey)))
+ }
+
+ /**
+ * Faithful in-memory replica of `S3AUtils#propagateBucketOptions` (`hadoop-aws`), which
+ * `S3AFileSystem#initialize` calls FIRST, before any option or credential is read:
+ * `Configuration conf = propagateBucketOptions(originalConf, bucket); ...; setConf(conf);` --
+ * every subsequent `conf.get`/`getPassword` call in that filesystem instance, including
+ * `${...}` variable substitution, resolves against this propagated view, not the original conf.
+ * `hadoop-aws` is not on this module's runtime classpath (see the string-literal-keys note
+ * above), so `S3AUtils#propagateBucketOptions` cannot be called directly; this reproduces its
+ * logic verbatim using only `hadoop-common`'s `Configuration`:
+ * {{{
+ * public static Configuration propagateBucketOptions(Configuration source, String bucket) {
+ * final String bucketPrefix = FS_S3A_BUCKET_PREFIX + bucket + '.';
+ * final Configuration dest = new Configuration(source);
+ * for (Map.Entry<String, String> entry : source) {
+ * final String key = entry.getKey();
+ * final String value = entry.getValue(); // the (unexpanded) value
+ * if (!key.startsWith(bucketPrefix) || bucketPrefix.equals(key)) continue;
+ * final String stripped = key.substring(bucketPrefix.length());
+ * if (stripped.startsWith("bucket.") || "impl".equals(stripped)) {
+ * // ignored
+ * } else {
+ * final String generic = FS_S3A_PREFIX + stripped;
+ * dest.set(generic, value, ...); // overwrites any existing global value
+ * }
+ * }
+ * return dest;
+ * }
+ * }}}
+ * Note the LONG bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) folds to an unread
+ * `fs.s3a.fs.s3a.<key>` key here too, exactly like the real method -- `stripped` already starts
+ * with `fs.s3a.` in that case, so prepending `fs.s3a.` again produces a key nothing ever reads.
+ */
+ private def propagateBucketOptions(hadoopConf: Configuration, bucket: String): Configuration = {
+ val bucketPrefix = s"fs.s3a.bucket.$bucket."
+ val dest = new Configuration(hadoopConf)
+ hadoopConf.iterator().asScala.foreach { entry =>
+ val key = entry.getKey
+ if (key.startsWith(bucketPrefix) && key != bucketPrefix) {
+ val stripped = key.substring(bucketPrefix.length)
+ if (!stripped.startsWith("bucket.") && stripped != "impl") {
+ dest.set(s"fs.s3a.$stripped", entry.getValue)
+ }
+ }
+ }
+ dest
+ }
+
+ /**
+ * Canonical and deprecated Hadoop S3A encryption-algorithm config keys, verified via `javap`
+ * against `hadoop-aws` 3.3.4's `org.apache.hadoop.fs.s3a.Constants`: `S3_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.encryption.algorithm"` (canonical) and `SERVER_SIDE_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.server-side-encryption-algorithm"` (DEPRECATED -- note the hyphen before "algorithm",
+ * unlike the corresponding `*.key` constants below, which both use a `.key` suffix).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, same
+ * rationale as [[HadoopCredentialProviderPathKey]].
+ */
+ private val S3EncryptionAlgorithmKey = "fs.s3a.encryption.algorithm"
+ private val DeprecatedS3EncryptionAlgorithmKey = "fs.s3a.server-side-encryption-algorithm"
+
+ /**
+ * The literal string `S3AEncryptionMethods.SSE_C.getMethod()` returns, verified via `javap`
+ * against `hadoop-aws` 3.3.4: `S3AEncryptionMethods#getMethod(String)` parses the configured
+ * algorithm with `values().find(_.getMethod.equalsIgnoreCase(algorithm))`, so this gate's
+ * comparison is case-insensitive too, matching Hadoop's own parsing rather than being stricter
+ * or looser than it.
+ */
+ private val SseCustomerKeyAlgorithm = "SSE-C"
+
+ /**
+ * `bucket`'s effective encryption-algorithm key and value under `hadoopConf`, or `None` when
+ * neither the canonical nor deprecated key is set anywhere consulted. Mirrors
+ * `S3AUtils#buildEncryptionSecrets`'s real resolution order (verified via `javap`): it first
+ * tries `lookupBucketSecret`, which checks ONLY the bucket-scoped SHORT form of each key
+ * (canonical before deprecated), then falls back to a BUCKET-AGNOSTIC global `lookupPassword`
+ * call for each key (canonical before deprecated) -- confirmed from the bytecode, which passes
+ * a literal `null` bucket argument at that second tier. Unlike [[s3KeyDivergenceReason]]'s
+ * [[CredentialValueKeys]] handling, the LONG bucket form
+ * (`fs.s3a.bucket.B.fs.s3a.encryption.algorithm`) is genuinely never consulted for this key:
+ * running it through [[propagateBucketOptions]] would only fold it into the unread
+ * `fs.s3a.fs.s3a.encryption.algorithm` key documented there, matching Hadoop's own behavior, so
+ * the global tier here reads `hadoopConf` directly rather than a propagated replica.
+ *
+ * The canonical-vs-deprecated distinction below is frequently moot in practice: `hadoop-aws`'s
+ * `S3AFileSystem.addDeprecatedKeys()` statically registers `fs.s3a.server-side-encryption-*` as
+ * `Configuration`-level deprecated aliases of `fs.s3a.encryption.*` (verified via `javap`), a
+ * registration that lives in a static field on Hadoop's `Configuration` class -- process-wide
+ * once `S3AFileSystem`'s class has loaded anywhere in the JVM, which a real scan has always
+ * already done by the time this gate runs, since reading the S3 table at all requires loading
+ * that class. Once active, `Configuration#get` resolves either literal key to the identical
+ * value transparently, making the two `plainValue`/`bucketShort` reads below redundant (but
+ * harmless) for that case; they remain the operative path only when nothing else in the process
+ * has loaded `S3AFileSystem` yet.
+ *
+ * Deliberately does NOT walk the Hadoop-credential-provider (JCEKS) path
+ * `buildEncryptionSecrets` falls back to when neither key is set in plain conf: unlike the
+ * actual key MATERIAL (`fs.s3a.encryption.key`/`fs.s3a.server-side-encryption.key`, never read
+ * here at all -- this gate only needs the ALGORITHM to decide), the algorithm NAME is not
+ * credential-sensitive data, so storing it in a keystore is not a realistic Hadoop deployment
+ * pattern. Documented, narrow limitation: a configuration that stores the algorithm name ONLY
+ * in a credential provider would under-decline here.
+ */
+ private def effectiveEncryptionAlgorithm(
+ hadoopConf: Configuration,
+ bucket: String): Option[(String, String)] = {
+ def bucketShort(baseKey: String): Option[(String, String)] =
+ plainValue(hadoopConf, s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a."))
+ .map(baseKey -> _)
+
+ bucketShort(S3EncryptionAlgorithmKey)
+ .orElse(bucketShort(DeprecatedS3EncryptionAlgorithmKey))
+ .orElse(plainValue(hadoopConf, S3EncryptionAlgorithmKey).map(S3EncryptionAlgorithmKey -> _))
+ .orElse(plainValue(hadoopConf, DeprecatedS3EncryptionAlgorithmKey)
+ .map(DeprecatedS3EncryptionAlgorithmKey -> _))
+ }
+
+ private def sseCustomerKeyDeclineReason(bucket: String, algorithmKey: String): String =
+ s"Native Delta scan does not support $algorithmKey=$SseCustomerKeyAlgorithm for $bucket " +
+ "(SSE-C requires the customer-provided encryption key on every S3 GET/HEAD request; " +
+ "the native S3 client's extract_s3_config_options forwards none of Hadoop's " +
+ "fs.s3a.encryption.*/fs.s3a.server-side-encryption* options, so requests would go out " +
+ "without the required x-amz-server-side-encryption-customer-key* headers and fail, where " +
+ "Hadoop's own reader -- whose request factory attaches the key -- succeeds)"
+
+ /**
+ * First reason any bucket among `uris` is configured for SSE-C (Server-Side Encryption with
+ * Customer-Provided Keys), or `None` when claimable. Scoped to SSE-C only: SSE-S3, SSE-KMS, and
+ * DSSE-KMS are all decrypted transparently by S3 on GET/HEAD once the caller has read/decrypt
+ * permission -- no request header is required, so native's silent failure to forward those
+ * options costs nothing on the read path this gate protects. SSE-C is different in kind, not
+ * degree: S3 rejects a GET/HEAD for an SSE-C object outright (400 Bad Request) unless the
+ * customer key is resent as a request header on every call, so a native scan that never learns
+ * the key cannot succeed at all, where Hadoop's own reader (which does attach it) would.
+ * Client-side encryption (CSE-KMS/CSE-CUSTOM) is a different mechanism entirely -- it
+ * encrypts/decrypts object bytes locally rather than gating the S3 request itself -- and is out
+ * of scope for this HTTP-header gate. Never interpolates a resolved key value, only key names,
+ * the bucket, and the (non-secret) algorithm name.
+ */
+ private[delta] def sseCustomerKeyReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ effectiveEncryptionAlgorithm(hadoopConf, bucket).collect {
+ case (key, value) if value.equalsIgnoreCase(SseCustomerKeyAlgorithm) =>
Review Comment:
[P2] Decline Hadoop CSE-KMS before claiming the native scan
This gate admits `fs.s3a.encryption.algorithm=CSE-KMS`, but the native S3 client has no client-side decryption layer. On the declared Spark 3.5.9/Hadoop 3.3.4 route, Hadoop's default S3 client factory selects an encryption client with KMS materials and range-read decryption. An existing, uniformly CSE-KMS-encrypted Delta table with valid credentials/key permissions, no S3Guard, and ordinary primitive NoMapping/no-DV data is therefore readable through Spark; enabling this contribution sends ciphertext to native Parquet and fails the scan. SSE-KMS is transparent server-side encryption and is not this case. Please retain Spark fallback for CSE-KMS until native decryption is supported. The clean requested BASE has no bundled scan provider and leaves this Delta scan with Spark; this finding predates the latest increment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5488274401
sorry, let me kick off another review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3905355114
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1627 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Zero-I/O in the common case, like the encryption gate above: native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The three keys where Hadoop's OWN resolver (`S3AUtils#lookupPassword`) is long-bucket-aware,
+ * verified via `javap` against `hadoop-aws` 3.3.4: `S3AUtils#lookupPassword(bucket, conf,
+ * baseKey)` builds `longBucketKey = "fs.s3a.bucket." + bucket + "." + baseKey` (note: the FULL
+ * `baseKey`, already `fs.s3a`-prefixed, appended after the bucket segment) and reads it via
+ * `getPassword` BEFORE the short-bucket key, keeping the long value whenever `getPassword`
+ * returns non-empty for it and only falling through to short-then-global otherwise. Kept as a
+ * named subset purely for documentation/discovery purposes (see
+ * [[hadoopLookupPasswordEffective]]'s doc); every key in [[AllS3ConfigKeys]] below is resolved
+ * through the SAME long-then-short-then-global function regardless of membership here -- see
+ * that function's doc for why extending long-bucket-awareness to keys where it does not
+ * strictly apply is safe.
+ */
+ private val CredentialValueKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ /**
+ * Every `fs.s3a.*` base key that governs whether a claimed native scan actually behaves like
+ * Hadoop's own reader would, checked by [[s3ConfigDivergenceReason]] via
+ * [[hadoopLookupPasswordEffective]] (see that function's doc for the resolution order and the
+ * deliberate over-decline rationale for the non-credential entries below). Two groups, kept
+ * merged into one list and one resolution path -- NOT two lists with two different resolution
+ * mechanisms, which is what let `fs.s3a.encryption.algorithm` silently fall through the
+ * short-then-global-only path while Hadoop's real resolver for it was long-bucket-aware (the
+ * SSE-C long-bucket-alias gap): the ORIGINAL [[CredentialValueKeys]] three, plus every OTHER
+ * per-bucket `fs.s3a.*` base key native's S3 client's `get_config` (s3.rs) resolves, verified
+ * directly against its call sites: `extract_s3_config_options` (endpoint.region,
+ * path.style.access, endpoint, requester.pays.enabled), `lookup_provider_class` (the
+ * Comet-specific credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name).
+ *
+ * SYNC NOTE: this list's non-credential entries must stay a superset of native's
+ * `NATIVE_S3A_CONFIG_PROPERTIES` constant (`native/core/src/parquet/objectstore/s3.rs`,
+ * property suffixes without the `fs.s3a.` prefix) -- `DeltaScanContribSuite`'s
+ * discovery-harness test asserts this mechanically. Literal strings, not the
+ * [[AwsCredentialsProviderKey]] / [[AssumedRoleCredentialsProviderKey]] vals declared below,
+ * purely to avoid a forward reference inside this `object` body; kept textually identical to
+ * those two constants.
+ */
+ private[delta] val AllS3ConfigKeys: Seq[String] = Seq(
+ "fs.s3a.access.key",
+ "fs.s3a.secret.key",
+ "fs.s3a.session.token",
+ "fs.s3a.aws.credentials.provider",
+ "fs.s3a.assumed.role.arn",
+ "fs.s3a.assumed.role.session.name",
+ "fs.s3a.assumed.role.credentials.provider",
+ "fs.s3a.endpoint",
+ "fs.s3a.endpoint.region",
+ "fs.s3a.path.style.access",
+ "fs.s3a.requester.pays.enabled",
+ "fs.s3a.comet.credential.provider.class")
+
+ /**
+ * The short-bucket-then-global value resolved for `baseKey` under `bucket` from `hadoopConf`,
+ * skipping an empty value at either alias exactly like [[plainValue]]. NOT used by
+ * [[s3ConfigDivergenceReason]]/[[s3KeyDivergenceReason]] any more -- every key checked there
+ * resolves through [[hadoopLookupPasswordEffective]] uniformly (long-then-short-then-global),
+ * which is a strict superset of what this function reads. This function's one remaining caller
+ * is [[shortThenGlobalOrReason]], which reads provider-CLASS strings (from the ORIGINAL,
+ * unpropagated conf) for name-support validation in [[providerClassReason]]/
+ * [[assumedRoleProviderClassReason]] -- by the time those run, [[s3ConfigDivergenceReason]] has
+ * already proven Hadoop's and native's effective values agree for the same key, so whichever of
+ * the two (equal) values this narrower read returns does not affect correctness there. NEVER
+ * used to compute native's own effective value -- see [[nativeShortThenGlobal]] for that.
+ */
+ private def shortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket` from the ORIGINAL, unpropagated `hadoopConf` -- `NativeConfig
+ * .extractObjectStoreOptions` forwards `Configuration#get`'s substituted value for every
+ * `fs.s3a.*` entry with no bucket-option propagation step of its own, so the original conf is
+ * the right input here. Unlike [[shortThenGlobal]]/[[plainValue]], this mirrors `get_config`
+ * faithfully: PRESENCE of the short-bucket key alone -- never its emptiness -- decides whether
+ * native falls back to the global key (`get_config` is a plain `HashMap::get`, which returns
+ * `Some` for a key explicitly set to `""`), so an explicitly empty or whitespace-only
+ * short-bucket value resolves to `Some("")` here and never falls through to global -- the
+ * OPPOSITE of Hadoop's own `getPassword`/`lookupPassword` semantics (see
+ * [[hadoopLookupPasswordEffective]]), which treat empty as absent and keep trying the next
+ * alias. The ONLY function used to compute native's effective value in
+ * [[s3KeyDivergenceReason]].
+ *
+ * Deliberately does NOT apply `get_config_trimmed`'s `.trim()` here: [[s3KeyDivergenceReason]]
+ * trims both this value and Hadoop's effective value together, symmetrically, at the point they
+ * are compared, rather than one-sidedly here -- trimming only the native side would flag a
+ * spurious divergence for a value neither side's whitespace actually changes the behavior of
+ * once each side's own downstream parsing normalizes it (e.g. Hadoop's own multi-line
+ * `fs.s3a.aws.credentials.provider` default, which both Hadoop and native additionally trim per
+ * comma-separated entry after splitting), while a one-sided trim would make an
+ * otherwise-identical default value look diverged for every bucket, never claiming natively at
+ * all.
+ */
+ private def nativeShortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ Option(hadoopConf.get(shortKey)).orElse(Option(hadoopConf.get(baseKey)))
+ }
+
+ /**
+ * Faithful in-memory replica of `S3AUtils#propagateBucketOptions` (`hadoop-aws`), which
+ * `S3AFileSystem#initialize` calls FIRST, before any option or credential is read:
+ * `Configuration conf = propagateBucketOptions(originalConf, bucket); ...; setConf(conf);` --
+ * every subsequent `conf.get`/`getPassword` call in that filesystem instance, including
+ * `${...}` variable substitution, resolves against this propagated view, not the original conf.
+ * `hadoop-aws` is not on this module's runtime classpath (see the string-literal-keys note
+ * above), so `S3AUtils#propagateBucketOptions` cannot be called directly; this reproduces its
+ * logic verbatim using only `hadoop-common`'s `Configuration`:
+ * {{{
+ * public static Configuration propagateBucketOptions(Configuration source, String bucket) {
+ * final String bucketPrefix = FS_S3A_BUCKET_PREFIX + bucket + '.';
+ * final Configuration dest = new Configuration(source);
+ * for (Map.Entry<String, String> entry : source) {
+ * final String key = entry.getKey();
+ * final String value = entry.getValue(); // the (unexpanded) value
+ * if (!key.startsWith(bucketPrefix) || bucketPrefix.equals(key)) continue;
+ * final String stripped = key.substring(bucketPrefix.length());
+ * if (stripped.startsWith("bucket.") || "impl".equals(stripped)) {
+ * // ignored
+ * } else {
+ * final String generic = FS_S3A_PREFIX + stripped;
+ * dest.set(generic, value, ...); // overwrites any existing global value
+ * }
+ * }
+ * return dest;
+ * }
+ * }}}
+ * Note the LONG bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) folds to an unread
+ * `fs.s3a.fs.s3a.<key>` key here too, exactly like the real method -- `stripped` already starts
+ * with `fs.s3a.` in that case, so prepending `fs.s3a.` again produces a key nothing ever reads.
+ */
+ private def propagateBucketOptions(hadoopConf: Configuration, bucket: String): Configuration = {
+ val bucketPrefix = s"fs.s3a.bucket.$bucket."
+ val dest = new Configuration(hadoopConf)
+ hadoopConf.iterator().asScala.foreach { entry =>
+ val key = entry.getKey
+ if (key.startsWith(bucketPrefix) && key != bucketPrefix) {
+ val stripped = key.substring(bucketPrefix.length)
+ if (!stripped.startsWith("bucket.") && stripped != "impl") {
+ dest.set(s"fs.s3a.$stripped", entry.getValue)
+ }
+ }
+ }
+ dest
+ }
+
+ /**
+ * Canonical and deprecated Hadoop S3A encryption-algorithm config keys, verified via `javap`
+ * against `hadoop-aws` 3.3.4's `org.apache.hadoop.fs.s3a.Constants`: `S3_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.encryption.algorithm"` (canonical) and `SERVER_SIDE_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.server-side-encryption-algorithm"` (DEPRECATED -- note the hyphen before "algorithm",
+ * unlike the corresponding `*.key` constants below, which both use a `.key` suffix).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, same
+ * rationale as [[HadoopCredentialProviderPathKey]].
+ */
+ private val S3EncryptionAlgorithmKey = "fs.s3a.encryption.algorithm"
+ private val DeprecatedS3EncryptionAlgorithmKey = "fs.s3a.server-side-encryption-algorithm"
+
+ /**
+ * The exact strings `S3AEncryptionMethods#getMethod` accepts, verified via `javap`/CFR against
+ * `hadoop-aws` 3.3.4's `S3AEncryptionMethods` enum: `NONE("")`, `SSE_S3("AES256", serverSide =
+ * true, requiresSecret = false)`, `SSE_KMS("SSE-KMS", serverSide = true, requiresSecret =
+ * false)`, `SSE_C("SSE-C", serverSide = true, requiresSecret = true)`, `CSE_KMS("CSE-KMS",
+ * serverSide = false, requiresSecret = true)`, `CSE_CUSTOM("CSE-CUSTOM", serverSide = false,
+ * requiresSecret = true)`. `getMethod` parses case-insensitively
+ * (`values().find(_.getMethod.equalsIgnoreCase(algorithm))`), matched below the same way.
+ *
+ * ALLOWLIST, not a blocklist (replaces the former SSE-C-only blocklist): only the algorithms S3
+ * decrypts transparently on GET/HEAD given read permission alone, with NO extra request header
+ * and NO client-side step, are safe for a native scan that forwards none of Hadoop's
+ * `fs.s3a.encryption.*`/`fs.s3a.server-side-encryption*` options --
+ * - `AES256` (SSE_S3, `serverSide = true`): plain server-side encryption, transparent on GET.
+ * - `SSE-KMS` (SSE_KMS, `serverSide = true`): server-side, KMS-managed key, transparent on
+ * GET given KMS decrypt permission (no header).
+ * - `DSSE-KMS`: NOT present in this enum on `hadoop-aws` 3.3.4 (confirmed by the six values
+ * listed above) -- `S3AEncryptionMethods.getMethod("DSSE-KMS")` throws
+ * `IOException("Unknown encryption algorithm DSSE-KMS")` on this version, so
+ * `S3AUtils#buildEncryptionSecrets` (and therefore Hadoop's own reader) already fails
+ * before ever reading such a table under 3.3.4, meaning this string can never actually be
+ * the resolved value on the declared target version -- admitting it here is inert there.
+ * Included anyway, forward-compatible, for a newer `hadoop-aws` on the runtime classpath (a
+ * later Hadoop release; this module has no compile-time `hadoop-aws` dependency, see the
+ * string-literal-keys note above) where DSSE-KMS is a real, dual-layer, server-side
+ * algorithm decrypted transparently on GET the same way SSE-KMS is. Every other value
+ * declines: `SSE-C` (SSE_C is `serverSide = true` in Hadoop's own enum, but `requiresSecret
+ * \= true` -- S3 rejects a GET/HEAD for an SSE-C object outright (400 Bad Request) unless
+ * the customer key is resent as a request header on every call, so a native scan that never
+ * learns the key cannot succeed at all, where Hadoop's own reader -- whose request factory
+ * attaches the key -- would), `CSE-KMS`/`CSE-CUSTOM` (`serverSide = false`: client-side
+ * encryption decrypts object bytes locally in the SDK layer, which the native Parquet
+ * reader has no equivalent of -- it would read raw ciphertext), and any future/unknown
+ * value (a value `S3AEncryptionMethods.getMethod` itself would reject is certainly not one
+ * of the three confirmed-transparent algorithms above; declining is the only safe default
+ * for anything this gate cannot positively confirm).
+ */
+ private val AllowedEncryptionAlgorithms: Set[String] = Set("AES256", "SSE-KMS", "DSSE-KMS")
+
+ /**
+ * `bucket`'s effective encryption-algorithm key and value under `hadoopConf`, or `None` when
+ * neither the canonical nor deprecated key is set anywhere consulted. Mirrors
+ * `S3AUtils#buildEncryptionSecrets`'s real resolution order, verified via `javap`/CFR
+ * decompilation of `hadoop-aws` 3.3.4's `S3AUtils.class`:
+ * {{{
+ * String algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.server-side-encryption-algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.server-side-encryption-algorithm");
+ * }}}
+ * i.e. bucket-tier (canonical, then deprecated), THEN global-tier (canonical, then deprecated)
+ * -- the two tiers are never interleaved key-by-key, so this must stay two explicit bucket-tier
+ * lookups followed by two explicit global-tier lookups, not a single
+ * [[hadoopLookupPasswordEffective]] call per key (which would let an unset canonical bucket key
+ * fall through straight to the canonical GLOBAL value ahead of a SET deprecated bucket key, the
+ * wrong answer).
+ *
+ * THE FIX for the SSE-C long-bucket-alias gap is entirely inside the bucket tier:
+ * `lookupBucketSecret` itself is long-then-short, decompiled from `hadoop-aws` 3.3.4's
+ * `S3AUtils.class`:
+ * {{{
+ * // longBucketKey = fs.s3a.bucket.B.fs.s3a.<key>
+ * String longBucketKey = String.format(BUCKET_PATTERN, bucket, baseKey);
+ * String initialVal = getPassword(conf, longBucketKey, null, null);
+ * // shortBucketKey = fs.s3a.bucket.B.<key>
+ * String shortBucketKey = String.format(BUCKET_PATTERN, bucket, subkey);
+ * // keeps initialVal (the LONG value) if non-empty
+ * return getPassword(conf, shortBucketKey, initialVal, null);
+ * }}}
+ * i.e. the SAME long-bucket-key construction and long-wins-if-nonempty semantics as
+ * `S3AUtils#lookupPassword` (see [[CredentialValueKeys]]/[[hadoopLookupPasswordEffective]]) --
+ * the encryption algorithm is NOT one of the keys that flows through
+ * `S3AUtils#propagateBucketOptions` (which folds an unrelated per-bucket LONG form into an
+ * unread key). An earlier version of this function modeled the bucket tier as SHORT-only,
+ * documented as "the LONG bucket form is genuinely never consulted for this key" -- that
+ * documentation was wrong (this decompilation supersedes it): a bucket configured only via
+ * `fs.s3a.bucket.B.fs.s3a.encryption.algorithm=SSE-C` bypassed the SSE-C gate entirely, because
+ * Hadoop's own reader DOES read that long form (and picks SSE-C), while this function reported
+ * `None` (nothing set) and the allowlist check below never even ran.
+ *
+ * The canonical-vs-deprecated distinction below is frequently moot in practice: `hadoop-aws`'s
+ * `S3AFileSystem.addDeprecatedKeys()` statically registers `fs.s3a.server-side-encryption-*` as
+ * `Configuration`-level deprecated aliases of `fs.s3a.encryption.*` (verified via `javap`), a
+ * registration that lives in a static field on Hadoop's `Configuration` class -- process-wide
+ * once `S3AFileSystem`'s class has loaded anywhere in the JVM, which a real scan has always
+ * already done by the time this gate runs, since reading the S3 table at all requires loading
+ * that class. Once active, `Configuration#get` resolves either literal key to the identical
+ * value transparently, making the two-key cascade below redundant (but harmless) for that case;
+ * it remains the operative path only when nothing else in the process has loaded
+ * `S3AFileSystem` yet.
+ *
+ * ALSO walks the Hadoop-credential-provider (JCEKS) path via [[resolveViaCredentialAliases]]
+ * for each of the four lookups below, matching `lookupBucketSecret`/`lookupPassword`'s real
+ * per-alias `getPassword` calls (quoted above) exactly: both are `getPassword`, not plain
+ * `Configuration#get`, so a bucket storing the algorithm name ONLY in a JCEKS keystore is
+ * exactly as real a Hadoop deployment shape for this key as it is for the credential keys
+ * [[hadoopLookupPasswordEffective]] already covers -- there is nothing algorithm-specific that
+ * makes JCEKS storage implausible here, so an earlier version of this function skipping it
+ * (documented at the time as "the algorithm NAME is not credential-sensitive data, so storing
+ * it in a keystore is not a realistic Hadoop deployment pattern") was an unjustified, narrower
+ * read than Hadoop's own resolver actually performs, under-declining a bucket whose algorithm
+ * is keystore-only. [[resolveViaCredentialAliases]]'s Arm B/C split still means this is zero
+ * extra I/O for the common case: keystore I/O only happens when a Hadoop credential-provider
+ * path is actually configured for the bucket, contained in that function's own try/catch.
+ * `bucketTier`/`globalTier` return `Left` (propagated straight through by [[orElseTier]]) when
+ * [[resolveViaCredentialAliases]] cannot safely verify a tier at all (an S3A-scoped provider
+ * path, or a corrupt/unreadable global keystore) -- correctly short-circuiting the whole
+ * cascade with a decline rather than silently falling through to a later tier that might look
+ * unset only because the true value was unverifiable.
+ */
+ private def effectiveEncryptionAlgorithm(
+ hadoopConf: Configuration,
+ bucket: String): Either[String, Option[(String, String)]] = {
+ def bucketTier(baseKey: String): Either[String, Option[(String, String)]] = {
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(longKey, shortKey))
+ .map(_.map(baseKey -> _))
+ }
+ def globalTier(baseKey: String): Either[String, Option[(String, String)]] =
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(baseKey)).map(_.map(baseKey -> _))
+
+ // Short-circuits on Left (unverifiable tier) or Right(Some(_)) (resolved); only Right(None)
+ // (tier definitively unset) falls through to `next`, mirroring buildEncryptionSecrets's
+ // sequential `if (algorithm == null) algorithm = ...` cascade exactly.
+ def orElseTier(
+ current: Either[String, Option[(String, String)]],
+ next: => Either[String, Option[(String, String)]])
+ : Either[String, Option[(String, String)]] =
+ current match {
+ case Left(reason) => Left(reason)
+ case Right(Some(value)) => Right(Some(value))
+ case Right(None) => next
+ }
+
+ orElseTier(
+ bucketTier(S3EncryptionAlgorithmKey),
+ orElseTier(
+ bucketTier(DeprecatedS3EncryptionAlgorithmKey),
+ orElseTier(
+ globalTier(S3EncryptionAlgorithmKey),
+ globalTier(DeprecatedS3EncryptionAlgorithmKey))))
+ }
+
+ private def unsupportedEncryptionAlgorithmDeclineReason(
+ bucket: String,
+ algorithmKey: String,
+ algorithm: String): String =
+ s"Native Delta scan does not support $algorithmKey=$algorithm for $bucket " +
+ "(the native S3 client only supports unencrypted objects and S3's transparent " +
+ "server-side algorithms -- AES256/SSE-S3, SSE-KMS, and DSSE-KMS decrypt on GET/HEAD given " +
+ "read permission alone, with no extra request header; SSE-C additionally requires the " +
+ "customer-provided key resent as a header on every GET/HEAD request, which the native S3 " +
+ "client's extract_s3_config_options never forwards, and CSE-KMS/CSE-CUSTOM decrypt object " +
+ "bytes client-side, a layer the native Parquet reader does not have -- any of these would " +
+ "fail outright or silently read ciphertext where Hadoop's own reader succeeds)"
+
+ /**
+ * First reason any bucket among `uris` is configured for an encryption algorithm the native S3
+ * client cannot safely read, or `None` when claimable. Allowlist-based (see
+ * [[AllowedEncryptionAlgorithms]]): only `AES256`/`SSE-KMS`/`DSSE-KMS` (and unset/empty) pass;
+ * every other resolved value -- `SSE-C`, `CSE-KMS`, `CSE-CUSTOM`, or any unrecognized future
+ * algorithm string -- declines. Deliberately NOT a blocklist keyed on `SSE-C` alone: an
+ * allowlist is safe by construction against a Hadoop release adding a new encryption method
+ * this gate has never heard of, where a blocklist would silently admit it. Never interpolates a
+ * resolved key value, only key names, the bucket, and the (non-secret) algorithm name. Declines
+ * on a `Left` from [[effectiveEncryptionAlgorithm]] too (an unverifiable credential-provider
+ * arm, e.g. an S3A-scoped provider path or a corrupt/unreadable global keystore) -- the
+ * algorithm cannot be ruled safe when it cannot be read at all.
+ */
+ private[delta] def unsupportedEncryptionAlgorithmReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ effectiveEncryptionAlgorithm(hadoopConf, bucket) match {
+ case Left(reason) => Some(reason)
+ case Right(None) => None
+ case Right(Some((key, value))) =>
+ if (!AllowedEncryptionAlgorithms.exists(_.equalsIgnoreCase(value))) {
+ Some(unsupportedEncryptionAlgorithmDeclineReason(bucket, key, value))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Canonical Hadoop S3A HTTP-proxy host config key, verified via CFR decompilation of
+ * `hadoop-aws` 3.3.4's `S3AUtils.class` (`initProxySupport`):
+ * {{{
+ * String proxyHost = conf.getTrimmed("fs.s3a.proxy.host", "");
+ * int proxyPort = conf.getInt("fs.s3a.proxy.port", -1);
+ * if (!proxyHost.isEmpty()) {
+ * ...
+ * String proxyUsername =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.username", null, null);
+ * String proxyPassword =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.password", null, null);
+ * ...
+ * }
+ * }}}
+ * `fs.s3a.proxy.host`/`fs.s3a.proxy.port` resolve via a PLAIN, non-bucket-scoped, non-JCEKS
+ * `Configuration#getTrimmed`/`getInt` call -- NOT `lookupPassword` -- against whatever conf
+ * `S3AFileSystem#initialize` already ran through `propagateBucketOptions` before
+ * `createAwsConf`/`initProxySupport` ever runs; only the SIBLING `fs.s3a.proxy.username`/
+ * `fs.s3a.proxy.password` keys go through `lookupPassword` (bucket long/short/global,
+ * JCEKS-aware). So the host is bucket-aware only via `propagateBucketOptions`'s short-bucket-
+ * form folding, never the long-bucket form, and never a credential-provider read -- the SAME
+ * shape as `endpoint`/`path.style.access` (the "propagated option" family, see
+ * [[AllS3ConfigKeys]]'s doc), not the credential family.
+ *
+ * Resolved here through [[hadoopLookupPasswordEffective]] anyway (long-then-short-then-global,
+ * credential-provider-aware) for the identical reason [[AllS3ConfigKeys]]'s non-credential
+ * entries are: checking MORE than Hadoop's real, narrower resolution can only ever find a value
+ * Hadoop itself ignores, which can only turn an already-necessary decline into one found
+ * slightly more eagerly -- never a missed one. A missed decline is exactly the failure mode
+ * this gate exists to close (native has NO HTTP-proxy support of any kind -- no
+ * `fs.s3a.proxy.*` key is read anywhere in `s3.rs`), so over-approximating is the conservative
+ * direction here, unlike a value-EQUALITY comparator (e.g. [[s3KeyDivergenceReason]]) where
+ * over-approximating could manufacture a false divergence between two sides that would
+ * otherwise agree.
+ */
+ private val S3ProxyHostKey = "fs.s3a.proxy.host"
+
+ private def unsupportedProxyReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key configured for $bucket (the native S3 client has " +
+ "no HTTP proxy support at all -- no fs.s3a.proxy.* key is read anywhere in its object " +
+ "store layer -- so a claimed scan would connect to S3 directly instead of routing through " +
+ "the configured proxy, either bypassing an egress/network-segmentation policy or simply " +
+ "failing to reach the endpoint)"
+
+ /**
+ * First reason any bucket among `uris` has an HTTP proxy configured via [[S3ProxyHostKey]], or
+ * `None` when claimable. Ordered alongside [[unsupportedEncryptionAlgorithmReason]] among the
+ * other conf-only gates that are zero-I/O in the common case, ahead of
+ * [[s3ConfigDivergenceReason]]: [[hadoopLookupPasswordEffective]] only performs real keystore
+ * I/O, inside its own try/catch, when a Hadoop credential-provider path is actually configured
+ * for this bucket (see that function's doc for the Arm A/B/C dispatch). Never interpolates a
+ * resolved value: the proxy HOST is not secret, but naming it here would be a strange place to
+ * first surface it, and proxy CREDENTIALS (`fs.s3a.proxy.username`/`fs.s3a.proxy.password`, not
+ * read by this gate at all -- the whole point of gating on the host is that a non-empty host
+ * declines before any proxy credential would ever need to be forwarded) must never appear in a
+ * decline reason regardless.
+ */
+ private[delta] def proxyGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI],
+ propagatedConfCache: MutableMap[String, Configuration] = MutableMap.empty)
+ : Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ try {
+ val propagatedConf =
+ propagatedConfCache.getOrElseUpdate(
+ bucket,
+ propagateBucketOptions(hadoopConf, bucket))
+ hadoopLookupPasswordEffective(propagatedConf, bucket, S3ProxyHostKey) match {
+ case Left(reason) => Some(reason)
+ case Right(Some(_)) => Some(unsupportedProxyReason(bucket, S3ProxyHostKey))
+ case Right(None) => None
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableValueReason(bucket, S3ProxyHostKey, e))
+ }
+ }
+ }
+ }
+
+ /**
+ * `baseKey`'s long, short, then global per-bucket aliases, in Hadoop's own resolution order.
+ */
+ private def longThenShortThenGlobalAliases(bucket: String, baseKey: String): Seq[String] = {
+ val suffix = baseKey.stripPrefix("fs.s3a.")
+ Seq(s"fs.s3a.bucket.$bucket.fs.s3a.$suffix", s"fs.s3a.bucket.$bucket.$suffix", baseKey)
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * The three-way Hadoop credential-provider-path precheck shared by every `getPassword`-based
+ * resolution below, factored out of what used to be [[hadoopLookupPasswordEffective]]'s own
+ * body so [[effectiveEncryptionAlgorithm]] can reuse it for its own bucket-tier-only and
+ * global-tier-only lookups (see that function's doc for why the algorithm needs its resolution
+ * split into separate tiers rather than one combined long-then-short-then-global list). This
+ * precheck is a property of the BUCKET alone, independent of which base key or alias list a
+ * caller goes on to resolve: Arm A -- an S3A-scoped or bucket-scoped Hadoop credential-provider
+ * path configures a provider `Configuration#getPassword` does not consult
+ * (`S3AUtils#lookupPassword`/`#lookupBucketSecret` build their own S3A-only provider list per
+ * bucket, which this plain-conf-reading gate cannot mirror) -- returns
+ * [[UnverifiableProvider]], NO keystore I/O. Arm B -- only the GLOBAL Hadoop provider path is
+ * set (which `getPassword` DOES consult) -- returns [[GlobalProviderOnly]]; the caller must
+ * wrap its own `getPassword` calls in try/catch, since real keystore I/O only happens once this
+ * arm is reached. Arm C -- no provider path is set anywhere -- returns [[NoProvider]], zero-I/O
+ * plain conf only.
+ */
+ private sealed trait CredentialProviderArm
+ private case class UnverifiableProvider(offendingKey: String) extends CredentialProviderArm
+ private case object GlobalProviderOnly extends CredentialProviderArm
+ private case object NoProvider extends CredentialProviderArm
+
+ private def credentialProviderArm(
+ hadoopConf: Configuration,
+ bucket: String): CredentialProviderArm = {
+ val bucketPathKey = s3aBucketProviderPathKey(bucket)
+ val bucketLongPathKey = s3aBucketLongProviderPathKey(bucket)
+ val s3aPathSet = nonEmptyConf(hadoopConf, S3aCredentialProviderPathKey)
+ val bucketPathSet = nonEmptyConf(hadoopConf, bucketPathKey)
+ val bucketLongPathSet = nonEmptyConf(hadoopConf, bucketLongPathKey)
+ if (s3aPathSet || bucketPathSet || bucketLongPathSet) {
+ val offendingKey =
+ if (s3aPathSet) S3aCredentialProviderPathKey
+ else if (bucketPathSet) bucketPathKey
+ else bucketLongPathKey
+ UnverifiableProvider(offendingKey)
+ } else if (nonEmptyConf(hadoopConf, HadoopCredentialProviderPathKey)) {
+ GlobalProviderOnly
+ } else {
+ NoProvider
+ }
+ }
+
+ /**
+ * `aliases`' effective value under `hadoopConf`/`bucket`, tried in the given order and keeping
+ * the first non-empty result -- or `Left(reason)` when the value cannot be safely verified.
+ * Dispatches on [[credentialProviderArm]]: [[UnverifiableProvider]] declines outright (zero
+ * I/O); [[GlobalProviderOnly]] resolves every alias via `Configuration#getPassword` (real
+ * keystore I/O, contained in try/catch: a corrupt/unreadable store must decline this bucket,
+ * not abort planning for the whole session); [[NoProvider]] resolves every alias via plain
+ * [[plainValue]] reads (zero I/O). Generalizes what used to be
+ * [[hadoopLookupPasswordEffective]]'s entire body to an explicit, caller-supplied alias list --
+ * [[hadoopLookupPasswordEffective]] is now the long-then-short-then-global special case of
+ * this; [[effectiveEncryptionAlgorithm]] is the bucket-tier-only (two aliases) and
+ * global-tier-only (one alias) special cases, matching
+ * `S3AUtils#lookupBucketSecret`/`#lookupPassword`'s real per-tier `getPassword` calls exactly
+ * instead of [[plainValue]]'s JCEKS-blind read that under-declined a keystore-only algorithm.
+ */
+ private def resolveViaCredentialAliases(
+ hadoopConf: Configuration,
+ bucket: String,
+ aliases: Seq[String]): Either[String, Option[String]] =
+ credentialProviderArm(hadoopConf, bucket) match {
+ case UnverifiableProvider(offendingKey) =>
+ Left(s3aScopedProviderPathReason(bucket, offendingKey))
+ case GlobalProviderOnly =>
+ try {
+ Right(
+ aliases.iterator
+ .map(alias =>
+ Option(hadoopConf.getPassword(alias)).map(new String(_)).filter(_.nonEmpty))
+ .collectFirst { case Some(v) => v })
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Left(unverifiableCredentialProviderReason(bucket, e))
+ }
+ case NoProvider =>
+ Right(aliases.flatMap(plainValue(hadoopConf, _)).headOption)
Review Comment:
Taken as well: the no-provider arm now honors the flag. With clear-text-fallback=false, password-tier keys resolve to nothing on the Hadoop side, so your stale plaintext pair diverges from native and declines; with the default it compares the pair as before, and flag=false with nothing plaintext still admits (the provider-class key is plain-tier, which the fallback flag never gates — verified in Configuration.getPassword bytecode, including the exact key name and default). One knock-on worth flagging: the encryption gate resolves through the same machinery, so a plaintext-only SSE-C algorithm under flag=false now admits. That is safe because Hadoop reader resolves no algorithm there either and issues a plain GET, and S3 enforces the customer-key header against every reader — an encrypted object fails loudly for both engines, an unencrypted one reads identically. There is a test pinning that reasoning so it cannot regress silently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3869530555
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,548 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. Strip the parquet.field.id metadata
+ // createPhysicalSchema also stamps: files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations.
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
Review Comment:
Added the conservative fallback: under caseSensitive=false, any data-schema name the reader would match against footers (logical names, or physical names under column mapping) containing a non-ASCII character declines. Partition names are exempt since their constants bind by exact name and never touch footer matching. Your CONVERT repro reproduced the (2,NULL) here and now declines with answers matching Spark; ASCII and caseSensitive=true controls still claim natively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3865631945
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1000 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.ListBuffer
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable. Only called when
+ * [[isDeltaScan]] is true. `scanHelper` is the [[CometScanExec]] built to drive
+ * [[CometDeltaNativeScan.convert]] on a claim, reused for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`: it runs at most
+ // once per claim attempt, and not at all for the common non-DV-shape case.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated.
+ val cmMode = metadata.columnMappingMode.name
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Credentials that resolve ONLY through a Hadoop credential provider (JCEKS et al.) are
+ // invisible to the plain-conf extraction forwarded to the native S3 client; reuses hadoopConf
+ // from the encryption gate above.
+ val credentialReason = credentialAliasReason(hadoopConf, dataFileUris ++ dvUris)
+ if (credentialReason.isDefined) {
+ return credentialReason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted by [[credentialAliasReason]] below. `hadoop-aws` is
+ * NOT on this module's runtime classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be
+ * referenced here (would raise `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ /**
+ * The aliases native's S3 credentials provider chain tries per bucket (short, then global),
+ * PLUS the LONG form `S3AUtils#lookupPassword` also consults. Hadoop resolves long before short
+ * before global; native reads only short+global, so a JCEKS entry set ONLY under the long alias
+ * is invisible to native -- a shadowed value under ANY alias here means decline.
+ */
+ private def s3aCredentialAliases(bucket: String): Seq[String] =
+ Seq(
+ s"fs.s3a.bucket.$bucket.fs.s3a.access.key",
+ s"fs.s3a.bucket.$bucket.fs.s3a.secret.key",
+ s"fs.s3a.bucket.$bucket.fs.s3a.session.token",
+ s"fs.s3a.bucket.$bucket.access.key",
+ s"fs.s3a.bucket.$bucket.secret.key",
+ s"fs.s3a.bucket.$bucket.session.token",
+ "fs.s3a.access.key",
+ "fs.s3a.secret.key",
+ "fs.s3a.session.token")
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ /**
+ * The three per-bucket S3A credential base keys the native S3 client's `get_config` (`s3.rs`)
+ * resolves: short bucket key first, then global (native never reads the long bucket key).
+ */
+ private val PlainCredentialBaseKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket`. Used by the credential-provider-class gates below, which read S3A options the same
+ * way native does.
+ */
+ private def effectiveOptionValue(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ private def plainLongFormCredentialDivergenceReason(bucket: String, longKey: String): String =
+ "Native Delta scan cannot forward long-form bucket credentials for " +
+ s"$bucket ($longKey is set but the native S3 client only reads the short bucket and " +
+ "global keys, so its credentials would differ from Hadoop's)"
+
+ /**
+ * Zero-I/O plain-value divergence check: native's `get_config` never reads the long form, so it
+ * can silently diverge from Hadoop's effective value (`getPassword` lets a non-empty long value
+ * WIN over short/global). Declines when the long form is set and the two sides disagree; never
+ * interpolates a resolved value, only the key name.
+ *
+ * Credentials only: every OTHER `fs.s3a` option flows through `propagateBucketOptions`, which
+ * strips only ONE `fs.s3a.bucket.B.` prefix layer, folding a long-form key into
+ * `fs.s3a.fs.s3a.<key>` (inert, unread elsewhere), so Hadoop's effective value already reduces
+ * to short.orElse(global) like native. No gate is needed for that set.
+ */
+ private def plainLongFormCredentialReason(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ PlainCredentialBaseKeys.foldLeft(Option.empty[String]) { (declined, baseKey) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val long = plainValue(hadoopConf, longKey)
+ if (long.isEmpty) {
+ None
+ } else {
+ val short = plainValue(hadoopConf, shortKey)
+ val global = plainValue(hadoopConf, baseKey)
+ val hadoopEffective = long.orElse(short).orElse(global)
+ val nativeEffective = short.orElse(global)
+ if (hadoopEffective != nativeEffective) {
+ Some(plainLongFormCredentialDivergenceReason(bucket, longKey))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * String-literal mirror of every credential-provider class name s3.rs's
+ * `build_aws_credential_provider_metadata` recognizes (Hadoop S3A plus AWS SDK v1/v2 names).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, never
+ * `classOf` references.
+ */
+ private val SupportedCredentialProviderClasses: Set[String] = Set(
+ "org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider",
+ "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider",
+ "com.amazonaws.auth.ContainerCredentialsProvider",
+ "com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper",
+ "software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider",
+ "com.amazonaws.auth.InstanceProfileCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider",
+ "com.amazonaws.auth.EnvironmentVariableCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider",
+ "com.amazonaws.auth.WebIdentityTokenCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider",
+ "com.amazonaws.auth.profile.ProfileCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider",
+ "com.amazonaws.auth.AnonymousAWSCredentials")
+
+ private val AnonymousCredentialProviderClasses: Set[String] = Set(
+ "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider",
+ "com.amazonaws.auth.AnonymousAWSCredentials")
+
+ private val HadoopAssumedRoleProviderClass =
+ "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider"
+
+ private val AwsCredentialsProviderKey = "fs.s3a.aws.credentials.provider"
+ private val AssumedRoleCredentialsProviderKey = "fs.s3a.assumed.role.credentials.provider"
+
+ /** Splits a comma-separated credential-provider-class list the same way s3.rs's parser does. */
+ private def parseProviderClassNames(value: String): Seq[String] =
+ value.split(",").map(_.trim).filter(_.nonEmpty).toSeq
+
+ private def unsupportedProviderReason(bucket: String, key: String, className: String): String =
+ s"Native Delta scan does not support the credential provider class $className " +
+ s"configured via $key for $bucket (the native S3 client only supports a fixed set of " +
+ "provider classes; an unsupported class would fail at scan execution time, after the " +
+ "scan was already claimed, rather than at planning time)"
+
+ private def mixedAnonymousProviderReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key for $bucket naming an anonymous credential " +
+ "provider together with any other provider (the native S3 client rejects this " +
+ "combination at scan execution time)"
+
+ private def anonymousAssumedRoleProviderReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support an anonymous credential provider in $key for " +
+ s"$bucket (the native S3 client does not allow an anonymous provider as the base " +
+ "credentials for an assumed-role chain)"
+
+ private def unsupportedProviderNameReason(
+ bucket: String,
+ key: String,
+ names: Seq[String]): Option[String] =
+ names
+ .find(name => !SupportedCredentialProviderClasses.contains(name))
+ .map(unsupportedProviderReason(bucket, key, _))
+
+ /**
+ * Decline reason when `bucket`'s effective `assumed.role.credentials.provider` names an
+ * unsupported class, or an anonymous one (native rejects ANY anonymous entry here, not just a
+ * mix). Unset defaults to native's own always-supported fallback, so `None` is safe.
+ */
+ private def assumedRoleProviderClassReason(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ effectiveOptionValue(hadoopConf, bucket, AssumedRoleCredentialsProviderKey) match {
+ case None => None
+ case Some(value) =>
+ val names = parseProviderClassNames(value)
+ unsupportedProviderNameReason(bucket, AssumedRoleCredentialsProviderKey, names).orElse {
+ if (names.exists(AnonymousCredentialProviderClasses.contains)) {
+ Some(anonymousAssumedRoleProviderReason(bucket, AssumedRoleCredentialsProviderKey))
+ } else {
+ None
+ }
+ }
+ }
+ }
+
+ /**
+ * Decline reason when `bucket`'s effective `aws.credentials.provider` names an unrecognized
+ * class, mixes an anonymous provider with any other, or a nested
+ * `AssumedRoleCredentialProvider` sub-chain has the same problem. Unset/empty falls back to
+ * native's default chain.
+ */
+ private def providerClassReason(hadoopConf: Configuration, bucket: String): Option[String] = {
+ effectiveOptionValue(hadoopConf, bucket, AwsCredentialsProviderKey).flatMap { value =>
+ val names = parseProviderClassNames(value)
+ unsupportedProviderNameReason(bucket, AwsCredentialsProviderKey, names)
+ .orElse {
+ if (names.length > 1 && names.exists(AnonymousCredentialProviderClasses.contains)) {
+ Some(mixedAnonymousProviderReason(bucket, AwsCredentialsProviderKey))
+ } else {
+ None
+ }
+ }
+ .orElse {
+ if (names.contains(HadoopAssumedRoleProviderClass)) {
+ assumedRoleProviderClassReason(hadoopConf, bucket)
+ } else {
+ None
+ }
+ }
+ }
+ }
+
+ /** First reason any bucket among `uris` names an unsupported credential-provider class. */
+ private[delta] def providerClassGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) declined else providerClassReason(hadoopConf, bucket)
+ }
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def shadowedCredentialAliasReason(bucket: String, alias: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($alias resolves through $HadoopCredentialProviderPathKey but is not present " +
+ "as a plain configuration value, so the native S3 client would have no credentials)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * Compares each [[s3aCredentialAliases]] alias's plain `Configuration#get` against
+ * `Configuration#getPassword` (providers first, plain conf fallback); a difference means
+ * decline. Runs in try/catch: `getPassword` does real keystore I/O, and a corrupt/unreadable
+ * store must decline this bucket, not abort planning for the whole session.
+ */
+ private def verifyGlobalProviderAliases(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ try {
+ s3aCredentialAliases(bucket).foldLeft(Option.empty[String]) { (declined, alias) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ val resolved =
+ Option(hadoopConf.getPassword(alias)).map(new String(_)).filter(_.nonEmpty)
+ resolved match {
+ case Some(value) if !Option(hadoopConf.get(alias)).contains(value) =>
+ Some(shadowedCredentialAliasReason(bucket, alias))
+ case _ => None
+ }
+ }
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableCredentialProviderReason(bucket, e))
+ }
+ }
+
+ /**
+ * The decline reason for `bucket` alone, or `None` when its credentials are safe to forward.
+ * [[plainLongFormCredentialReason]] is checked first (zero-I/O, decisive on its own); then
+ * `None` if no provider-path key is set. Arm A: an S3A-scoped provider-path key points at a
+ * provider `getPassword` does not consult, so any being set declines with no keystore read. Arm
+ * B: only the global path is set (which `getPassword` DOES consult); delegates to
+ * [[verifyGlobalProviderAliases]].
+ */
+ private def bucketCredentialAliasReason(
+ hadoopConf: Configuration,
+ bucket: String,
+ globalPathSet: Boolean,
+ s3aPathSet: Boolean): Option[String] = {
+ plainLongFormCredentialReason(hadoopConf, bucket).orElse {
+ val bucketPathKey = s3aBucketProviderPathKey(bucket)
+ val bucketLongPathKey = s3aBucketLongProviderPathKey(bucket)
+ val bucketPathSet = nonEmptyConf(hadoopConf, bucketPathKey)
+ val bucketLongPathSet = nonEmptyConf(hadoopConf, bucketLongPathKey)
+ if (!globalPathSet && !s3aPathSet && !bucketPathSet && !bucketLongPathSet) {
+ None
+ } else if (s3aPathSet || bucketPathSet || bucketLongPathSet) {
+ val offendingKey =
+ if (s3aPathSet) S3aCredentialProviderPathKey
+ else if (bucketPathSet) bucketPathKey
+ else bucketLongPathKey
+ Some(s3aScopedProviderPathReason(bucket, offendingKey))
+ } else {
+ verifyGlobalProviderAliases(hadoopConf, bucket)
+ }
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf key prefix for GCS authentication options (`fs.gs.auth.*`).
+ * `gcs-connector` is NOT on this module's runtime classpath by default, so referencing an
+ * actual GCS auth class would risk `NoClassDefFoundError`, same rationale as the S3A literals
+ * above.
+ */
+ private val GcsAuthKeyPrefix = "fs.gs.auth."
+
+ /**
+ * True when `uri`'s scheme is `gs` (case-insensitive) -- the ONLY scheme object_store's
+ * `ObjectStoreScheme::parse` (parquet_support.rs) routes to `GoogleCloudStorage`; `gcs` is not
+ * recognized there and is deliberately excluded.
+ */
+ private def isGcsScheme(uri: URI): Boolean =
+ Option(uri.getScheme).exists(_.equalsIgnoreCase("gs"))
+
+ /**
+ * The lowercase-scheme-checked GCS bucket name from `uri`'s authority (host, minus any userinfo
+ * or port), or `None` when `uri`'s scheme is not `gs`. Parses the raw authority manually,
+ * mirroring [[s3Bucket]]'s `URI#getHost`/RFC 3986 `reg-name` reasoning.
+ */
+ private def gcsBucket(uri: URI): Option[String] = {
+ if (!isGcsScheme(uri)) {
+ None
+ } else {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ }
+ }
+
+ /**
+ * The non-empty `fs.gs.auth.*` Hadoop conf keys set on `hadoopConf`, full key names only --
+ * NEVER their values, which are credential material and must never enter a decline reason.
+ * Iterates the conf map directly: no provider resolution, no I/O.
+ */
+ private def gcsAuthKeys(hadoopConf: Configuration): Seq[String] =
+ hadoopConf
+ .iterator()
+ .asScala
+ .collect {
+ case entry
+ if entry.getKey.startsWith(GcsAuthKeyPrefix) &&
+ entry.getValue != null && entry.getValue.nonEmpty =>
+ entry.getKey
+ }
+ .toSeq
+ .distinct
+ .sorted
+
+ /**
+ * Decline reason when any of `uris` resolves to a `gs://` authority AND `hadoopConf` sets any
+ * `fs.gs.auth.*` key, or `None` when claimable. Native forwards none of `fs.gs.*` to the object
+ * store, so a scan relying solely on Hadoop-side GCS credentials would claim here but then fail
+ * authentication natively. Application Default Credentials work identically in both engines and
+ * need no Hadoop conf key, so an ADC-only configuration still claims. Never interpolates a
+ * resolved value, only key names.
+ */
+ private[delta] def gcsHadoopOnlyAuthReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val gcsUris = uris.filter(isGcsScheme)
+ if (gcsUris.isEmpty) {
+ return None
+ }
+ val authKeys = gcsAuthKeys(hadoopConf)
+ if (authKeys.isEmpty) {
+ return None
+ }
+ val buckets = gcsUris.flatMap(gcsBucket).distinct.sorted
+ Some(
+ "Native Delta scan does not support GCS authentication configured only via Hadoop conf " +
+ s"key(s) ${authKeys.mkString(", ")} for gs://${buckets.mkString(", gs://")} " +
+ "(the native GCS client does not forward fs.gs.* options; only Application Default " +
+ "Credentials -- environment or metadata-server -- are available natively)")
+ }
+
+ /**
+ * First reason a native S3 scan cannot faithfully forward this table's Hadoop credentials, or
+ * `None` when claimable. Only `s3`/`s3a` authorities matter here (ABFS/WASB mooted by the
+ * userinfo gate, GCS handled by [[gcsHadoopOnlyAuthReason]]). `Configuration#getPassword`
+ * checks every provider first, falling back to plain conf only when unset, so a keystore-only
+ * or shadowing alias is invisible/wrong to native's plain-conf extraction; when in doubt,
+ * decline. Never interpolates a resolved value, only key names.
+ */
+ private[delta] def credentialAliasReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ if (buckets.isEmpty) {
+ return None
+ }
+ val globalPathSet = nonEmptyConf(hadoopConf, HadoopCredentialProviderPathKey)
+ val s3aPathSet = nonEmptyConf(hadoopConf, S3aCredentialProviderPathKey)
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ bucketCredentialAliasReason(hadoopConf, bucket, globalPathSet, s3aPathSet)
+ }
+ }
+ }
+
+ /**
+ * True when `dataType` is, or structurally contains (through array elements or map keys/
+ * values), a [[StructType]]. Only [[StructType]] fields carry Delta's physical, column-mapped
+ * names; array/map labels themselves are never column-mapped.
+ */
+ private def containsNestedStruct(dataType: DataType): Boolean = dataType match {
+ case _: StructType => true
+ case ArrayType(elementType, _) => containsNestedStruct(elementType)
+ case MapType(keyType, valueType, _) =>
+ containsNestedStruct(keyType) || containsNestedStruct(valueType)
+ case _ => false
+ }
+
+ /**
+ * True when `node` is a positional-output union -- `UnionExec` or `CometUnionExec`. Both
+ * compute output positionally from the FIRST child's attributes, so a value carried only by a
+ * LATER branch needs an explicit positional walk below. Compared by class name (the
+ * [[isDeltaScan]] idiom) to avoid a compile-time dependency; an unmatched name is still safe,
+ * caught by the generic multi-child safety net below.
+ */
+ private def isPositionalUnion(node: SparkPlan): Boolean = {
+ val name = node.getClass.getSimpleName
+ name == "UnionExec" || name == "CometUnionExec"
+ }
+
+ /**
+ * True when the scan's row-index column value is provably dead above the scan. The standard DV
+ * plan shape routes it only into a `named_struct(... row_index ...) AS _metadata` projection
+ * whose result the final projection discards; anything else (a query actually selecting
+ * `_metadata.row_index`) makes the value live and must decline. Conservative: any unrecognized
+ * consumption pattern returns false.
+ */
+ private def rowIndexUnusedAbove(plan: SparkPlan, scanExec: FileSourceScanExec): Boolean = {
+ val rowIndexAttrs = scanExec.output
+ .filter(_.name == CometDeltaNativeScan.RowIndexColumn)
+ .map(_.exprId)
+ .toSet
+ if (rowIndexAttrs.isEmpty) {
+ return true
+ }
+ // Transitive taint analysis: everything derived from the row-index attribute within the
+ // visible plan, via Project aliases or positionally across a union. The plan may be an AQE
+ // stage fragment, so tainted values escaping to the fragment's own output must decline too.
+ var tainted = rowIndexAttrs
+ var changed = true
+ while (changed) {
+ changed = false
+ plan.foreach {
+ case p: ProjectExec =>
+ p.projectList.foreach {
+ case a: Alias
+ if !tainted.contains(a.exprId) &&
+ a.references.exists(r => tainted.contains(r.exprId)) =>
+ tainted += a.exprId
+ changed = true
+ case _ =>
+ }
+ case u if isPositionalUnion(u) =>
+ // Output attributes carry the FIRST child's expression IDs, so a value tainted only in
+ // a LATER branch is otherwise invisible; walk it forward positionally instead.
+ // `children` can be re-parented by AQE after `output` is frozen, so an arity mismatch on
+ // ANY child (which would make a positional zip silently truncate) forces a decline.
+ if (u.children.exists(_.output.length != u.output.length)) {
+ return false
+ }
+ u.children.foreach { child =>
+ child.output.zip(u.output).foreach {
+ case (from, to) if tainted.contains(from.exprId) && !tainted.contains(to.exprId) =>
+ tainted += to.exprId
+ changed = true
+ case _ =>
+ }
+ }
+ case _ =>
+ }
+ }
+ val nonProjectConsumer = plan.exists {
+ case _: ProjectExec => false
+ case n if n ne scanExec =>
+ n.expressions.exists(_.references.exists(r => tainted.contains(r.exprId)))
+ case _ => false
+ }
+ val escapes = plan.output.exists(a => tainted.contains(a.exprId))
+ // Generic safety net for every OTHER multi-child node (joins, etc; positional unions are
+ // exempt, already handled precisely above). A tainted attribute a child contributes must
+ // either survive into the node's own output under the SAME expression ID or be consumed by
+ // one of the node's own expressions; otherwise decline (e.g. a LEFT SEMI/ANTI join dropping
+ // the side carrying the tainted attribute).
+ val multiChildLeak = plan.exists {
+ case u if isPositionalUnion(u) => false
+ case n if n.children.size >= 2 =>
Review Comment:
**[P1] Treat write sinks as consumers of the reader's row index**
Could we make the unused-row-index proof account for unary write sinks? This is a reader eligibility issue: Spark's existing writer simply persists the values returned by the reader.
For example, with the Delta contrib/native scan enabled and the native Parquet writer disabled, this workload reads the source row position into an ordinary output column:
```scala
val root = java.nio.file.Files.createTempDirectory("comet-row-index")
val src = root.resolve("src").toString
val dst = root.resolve("dst").toString
spark.range(32).coalesce(1).write.format("delta")
.option("delta.enableDeletionVectors", "true").save(src)
spark.sql(s"DELETE FROM delta.`$src` WHERE id IN (1, 7, 13)").collect()
spark.read.format("delta").load(src)
.selectExpr("id", "_metadata.row_index AS ri")
.write.parquet(dst)
spark.read.parquet(dst).where("id = 31").show() // expected: (31, 31)
spark.read.parquet(dst).selectExpr("sum(ri)").show() // expected: 475
```
The real physical write plan is `DataWritingCommandExec -> WriteFilesExec -> Project -> ... -> Delta scan`. Both write nodes have one child, empty output, and no expression reference to `ri`; they consume their child's rows positionally. The two Projects propagate the row-index dependency, but neither `nonProjectConsumer`, the root-output `escapes` check, nor this multi-child-only guard sees the write consuming it. The exact current `rowIndexUnusedAbove` returns `true` for the Parquet write, while the equivalent SELECT correctly returns `false`.
That permits the DV reader to supply the [synthetic Long zero](https://github.com/apache/datafusion-comet/blob/b033153da230f0239c6998532d063020751052de/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala#L474-L491). On the same actual Delta fixture, the current native reader retained all 29 correct surviving IDs but returned `ri=0` for every row: 28 incorrect row-index values, sum 0, and `(31,0)` rather than `(31,31)`. The unchanged writer would persist those incorrect input values.
Validation: the exact liveness method was exercised on real cached Spark 4.0.2-based/Delta 4.0.0 write plans with AQE off/on, including an independent default-codegen check. Separately, the production native DV attachment and Parquet reader were exercised on the actual file and DV, with the source-verified scan layout. The remaining admission path was source-traced; a full current-profile Spark/Comet JNI write was **not** executed.
Please treat write sinks as consumers of tainted input attributes, or conservatively decline this shape, and add a regression checking both the eligibility decision and saved `ri` values. The liveness logic already exists at `1d3557cf`; this is not introduced by the latest `b033153d` update.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,548 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. Strip the parquet.field.id metadata
+ // createPhysicalSchema also stamps: files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations.
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
Review Comment:
**[P1] Guard Unicode case-insensitive Delta reads before using the shared reader**
With column mapping disabled and `spark.sql.caseSensitive=false`, this call sends the logical names unchanged to the shared native Parquet reader. Its [ASCII-only matching](https://github.com/apache/datafusion-comet/blob/b033153da230f0239c6998532d063020751052de/native/core/src/parquet/schema_adapter.rs#L208-L215) does not preserve Spark's handling of non-ASCII case variants in physical column names.
A normal converted Delta table can expose this:
```scala
import org.apache.spark.sql.functions.{col, lit}
spark.conf.set("spark.sql.caseSensitive", false)
val root = java.nio.file.Files.createTempDirectory("comet-unicode")
val path = root.resolve("data").toString
val table = "comet_unicode_" + java.util.UUID.randomUUID().toString.replace("-", "")
spark.range(1, 2).select(col("id"), lit(71).as("É"))
.coalesce(1).write.parquet(path)
spark.range(2, 3).select(col("id"), lit(72).as("é"))
.coalesce(1).write.mode("append").parquet(path)
spark.sql(s"CREATE TABLE $table (id BIGINT, `É` INT) USING PARQUET LOCATION '$path'")
spark.sql(s"CONVERT TO DELTA $table NO STATISTICS")
spark.read.format("delta").load(path).selectExpr("id", "`É`").show()
```
Stock Spark/Delta returns `[(1,71),(2,72)]`. The current native reader on these exact files returns `[(1,71),(2,NULL)]`. Adding `É IS NOT NULL` or `É > 70` loses the second row. A covering Spark filter cannot recover the stored value after the scan has replaced it with NULL.
This is an ordinary `CONVERT` transaction with protocol `(1,2)`, mapping `none`, no defaults or DVs, and two local files. The declared required/data schemas are both `[id, É]`, projection is `[0,1]`, and field-ID matching is off. The case difference appears only in the second file's footer. Current admission has no Unicode-name exclusion, and the new Delta handler installs the same `SparkPhysicalExprAdapterFactory` exercised by the probe.
Validation: the standard conversion command and reference reads ran on cached Spark 4.0.2-based/Delta 4.0.0. Current native-reader execution was compared against those results: nine mismatches across `É/é`, `Σ/σ`, and `Б/б` for plain reads and the two predicates; all 15 ASCII/case-sensitive controls matched. The complete current claim path was source-traced, not executed through a full current-profile Spark/JNI pipeline.
Could we fix the shared name matching or conservatively fall back for affected Delta schemas, with an asserted native/fallback regression for this conversion history? The matcher predates the PR, but this PR newly routes these previously Spark-read Delta tables through it. The same exposure exists at `1d3557cf`, so it is not introduced by the latest update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3835744306
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, SparkPlan}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so
+ * `CometPlanAdaptiveDynamicPruningFilters`'s rewrite (via [[CometScanWithPlanData]]) survives
+ * plan copies, the lesson from CometIcebergNativeScanExec (a transient field is dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node, our
+ // own withDynamicPruningFilters, or a generic Catalyst expression rewrite going through
+ // TreeNode.makeCopy, automatically gets a helper consistent with ITS runtimeFilters. A
+ // stored helper field would desync from rewritten filters (the #3510 class of bug). The
+ // cost is that file listing runs once per executed instance (planning listed separately in
+ // the rule extension); correctness over the duplicate driver-side listing.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ override lazy val outputPartitioning: Partitioning =
+ UnknownPartitioning(perPartitionData.length)
Review Comment:
Fixed as described: outputPartitioning is now a plain def returning UnknownPartitioning(0) while any runtime filter still holds an adaptive broadcast placeholder, so AQE validation never forces perPartitionData. Rewrote the DPP test to filter at query time and added your UNION ALL shape as a regression. That shape didn't reproduce the crash pre-fix on my Spark 3.5.9 / Delta 3.3.2 profile, so it likely needs your Spark 4.0.2 harness, but the guard matches your analysis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5399703210
### [P2] Preserve the unsigned-small-integer safety fallback
At `6d20ad69`, the [Delta claim path](https://github.com/apache/datafusion-comet/blob/6d20ad69fd39f52c35230d2fa5373aa215e7ed5e/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanContrib.scala#L79-L83) bypasses `CometScanTypeChecker`, including the default-on `spark.comet.scan.unsignedSmallIntSafetyCheck`. This exposes the result mismatch that the existing core gate protects against.
I verified a catalog Parquet table with an `INT32` column annotated `UINT_8`, containing `[-1, 0, 127, 255, 256]`, converted successfully using `CONVERT TO DELTA`:
- Spark 4.0.2 / Delta 4.0.0 infers `ShortType` and reads `[-1, 0, 127, 255, 256]`.
- The current core schema checker rejects the scan with the safety setting enabled, but Delta's `declineReason` returns `None` and the production `CometScanContrib.firstClaimFrom` path returns `CometDeltaNativeScanExec`.
- Parquet 58.4 decodes the same file as `UInt8`, producing `[255, 0, 127, 255, 0]`; widening to `Int16` preserves those changed values. The production DataFusion/Comet source path decodes the physical type first and casts afterward, rather than requesting `Int16` during decoding.
This is specifically about **out-of-range/malformed `UINT_8` payloads**, not valid-range unsigned values. Verification used current admission code, a real converted Delta table, and the locked-version decoder/cast; it was not a full Comet/JNI query run.
Could we preserve the existing unsigned-small-integer fallback before claiming the scan, and add a converted-Parquet regression with the safety setting enabled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3881808035
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme (finding 8).
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
Review Comment:
Fixed. The native side of the comparison was dropping empty values, but native itself decides short-vs-global purely on key presence, so an empty short alias counts as set. Your shape declines now and removing the empty shorts passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4081019635
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,583 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ *
+ * `private[delta]` (not `private`): [[DeltaScanSupport.declineReason]]'s non-ASCII
+ * case-insensitive name gate reuses this exact conversion to compute the names native sees
+ * under column mapping, rather than re-deriving physical names with separate logic.
+ */
+ private[delta] def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. Strip the parquet.field.id metadata
+ // createPhysicalSchema also stamps: files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations.
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully. `memo` is the same claim-memo instance
+ * [[DeltaScanSupport.declineReason]] populated on this claim; its `hadoopConf` and
+ * `dvDescriptors` are reused here rather than recomputed.
+ */
+ def convert(
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaScanSupport.DeltaClaimMemo): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = memo.hadoopConf
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
Review Comment:
> Could `common.source` use the original physical scan's `SparkPlan.id` in both builders?
Tracked in #5943. The key keeps its current shape in this change, and the issue carries the regression you describe with disjoint selected files and an equivalent-scan control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4081174394
##########
.github/workflows/delta_contrib_test.yml:
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Delta Contrib Tests
+
+# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
+# live in the umbrella workflow.
+on:
+ workflow_call:
+
+permissions:
+ contents: read
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: rust-lld cannot resolve -ljvm against the Zulu JDK
+ # layout installed by setup-java (same rationale as pr_build_linux.yml).
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+ # The container's default locale is POSIX, which makes the JVM's file-path encoder
+ # reject non-ASCII partition directory names the suites create.
+ LANG: "C.UTF-8"
+ LC_ALL: "C.UTF-8"
+
+jobs:
+
+ contrib-delta:
+ name: Delta contrib (Spark ${{ matrix.profile.spark }})
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ strategy:
+ matrix:
+ profile:
+ - spark: "3.5"
+ java_version: "17"
+ - spark: "4.0"
+ java_version: "17"
+ - spark: "4.1"
+ java_version: "17"
+ # spark-4.2 is intentionally absent: the contrib profile is dormant
+ # until a Delta release supports Spark 4.2.
+ fail-fast: false
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: ${{ matrix.profile.java_version }}
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-${{ matrix.profile.spark }}
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Restore Cargo cache
+ uses: actions/cache/restore@v6
+ with:
+ path: |
+ ~/.cargo/registry
+ ~/.cargo/git
+ native/target
+ key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
+ restore-keys: |
+ ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-
+
+ - name: Build native library (CI profile)
+ run: |
+ cd native
+ # `delta` is in the default feature set, so no feature flag is needed.
+ cargo build --profile ci
+ env:
+ # Must match the flags spark_sql_test_reusable.yml builds with:
+ # cargo folds RUSTFLAGS into its fingerprints, so any divergence
+ # would make the shared cargo cache restore without ever hitting.
+ RUSTFLAGS: "-Ctarget-cpu=x86-64-v3 -Clink-arg=-fuse-ld=bfd"
+
+ # No cache save: this workflow never runs on a push to main, so the restore above
+ # falls back to the cache build_linux saves there.
+
+ - name: Stage native library at release path
+ run: |
+ # Maven's -Prelease profile (activated below) expects libcomet.so
+ # under native/target/release/; --profile ci builds it under
+ # native/target/ci/ instead (same as the other native-building
+ # workflows), so copy it into place.
+ mkdir -p native/target/release
+ cp native/target/ci/libcomet.so native/target/release/libcomet.so
+
+ - name: Install Comet core jars
+ run: |
+ ./mvnw -B -q -Prelease -Pspark-${{ matrix.profile.spark }} install -pl common,spark -DskipTests -Dspotless.check.skip=true
+
+ - name: Run Delta contrib test suites
+ run: |
+ SPARK_HOME=$(pwd) COMET_CONF_DIR=$(pwd)/conf ./mvnw -B -Prelease -Pspark-${{ matrix.profile.spark }},delta test -pl contrib/delta-spark
+
+ # The container jobs above have no Docker socket, so CometDeltaS3Suite cancels itself
+ # there. This job runs it on the plain runner, where Testcontainers can start MinIO.
+ contrib-delta-s3:
+ name: Delta contrib S3 suite (Spark 3.5, MinIO)
+ runs-on: ubuntu-24.04
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
Review Comment:
> Please use a setup path that elevates package installation on the plain runner while retaining Docker access.
Done on 15 September. The MinIO job no longer uses `setup-builder`; it installs its packages with `sudo` on the plain runner, sets up the JDK and Rust itself, and keeps the Docker socket for Testcontainers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4007003431
##########
spark/pom.xml:
##########
@@ -585,6 +585,19 @@ under the License.
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
</plugin>
+ <plugin>
Review Comment:
> Binding the execution inside a `delta` profile in this pom would keep it to the builds that need it. Also the comment says `contrib/delta`; the consumer is `contrib/delta-spark`.
The `test-jar` execution now lives in a `delta` profile in `spark/pom.xml`, so it only exists when the contrib is built, and the comment names `contrib/delta-spark`. The release questions stay with #5882 as you proposed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3989744779
##########
native/core/src/execution/planner/delta_spark_scan.rs:
##########
@@ -0,0 +1,792 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! JVM-planned Delta handler for the generic `OpStruct::ContribScan` dispatcher, feature-gated
+//! behind `delta`.
+//!
+//! delta-spark has already done log replay, snapshot resolution, and partition pruning by the
+//! time the scan reaches Comet, so the envelope carries a concrete file list (plus deletion
+//! vector descriptors) and the read path reuses the exact same shared parquet scan builder as
+//! `NativeScan` -- inheriting row-group stats pruning, page-index pruning, and filter pushdown.
+//! Sibling of the kernel-planned handler in `delta_scan.rs`; the two claim different
+//! `type_url`s within the same `ContribScan` envelope.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::execution::object_store::ObjectStoreUrl;
+use object_store::path::Path;
+use object_store::ObjectStore;
+use url::Url;
+
+use datafusion_comet_proto::spark_operator::{
+ ContribScan, DeltaSparkScan, Operator, SparkFilePartition, SparkPartitionedFile,
+};
+use prost::Message;
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::execution::planner::PhysicalPlanner;
+use crate::execution::planner::PlanCreationResult;
+use crate::parquet::objectstore::s3_blob_fs_support::normalize_object_store_url;
+use crate::parquet::parquet_support::{
+ hash_object_store_configs, object_store_url_key, prepare_object_store_with_config_hash,
+};
+
+/// Type name the JVM-planned Delta contrib claims within the `ContribScan` envelope. The
+/// contrib jar packs a `DeltaSparkScan` with a `type_url` of
+/// `type.googleapis.com/comet.contrib.delta_spark.DeltaSparkScan`; dispatch keys on the
+/// contrib-owned suffix, same convention as the kernel path's `delta_scan.rs`.
+const DELTA_SPARK_SCAN_TYPE_NAME: &str = "comet.contrib.delta_spark.DeltaSparkScan";
+
+/// Contrib entry point for the `OpStruct::ContribScan` dispatcher. Returns `Some(result)` when
+/// the envelope carries a JVM-planned Delta scan, or `None` when the `type_url` belongs to some
+/// other contrib.
+pub(crate) fn try_plan_contrib_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ contrib: &ContribScan,
+) -> Option<PlanCreationResult> {
+ if !contrib.type_url.ends_with(DELTA_SPARK_SCAN_TYPE_NAME) {
+ return None;
+ }
+ Some(
+ DeltaSparkScan::decode(contrib.value.as_slice())
+ .map_err(|e| {
+ GeneralError(format!(
+ "Failed to decode DeltaSparkScan from contrib_scan: {e}"
+ ))
+ })
+ .and_then(|scan| plan_delta_spark_scan(planner, spark_plan, &scan)),
+ )
+}
+
+fn plan_delta_spark_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ scan: &DeltaSparkScan,
+) -> PlanCreationResult {
+ // Delta data files are plain parquet; the read path deliberately reuses
+ // the same shared parquet scan builder as NativeScan so Delta inherits
+ // row-group stats pruning, page-index pruning, and filter pushdown. Only
+ // the file list arrives in Delta-specific form. Note delta_common's
+ // column_mapping_mode is informational in M1: the actual field-id
+ // matching switch is common.use_field_id, same as the Iceberg path.
+ let common = scan
+ .common
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing common data".into()))?;
+
+ let delta_partition = scan
+ .file_partition
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing file_partition".into()))?;
+
+ let spark_partition = SparkFilePartition {
+ partitioned_file: delta_partition
+ .partitioned_file
+ .iter()
+ .map(|f| {
+ f.file.clone().ok_or_else(|| {
+ GeneralError("DeltaSparkPartitionedFile missing inner file".into())
+ })
+ })
+ .collect::<Result<Vec<_>, _>>()?,
+ };
+
+ // Defense-in-depth against a stale or bypassed JVM gate: DeltaScanSupport.declineReason
+ // (multiStoreReason) already declines data files spanning multiple object-store authorities
+ // at planning time, but prepare_scan_store_and_files below resolves this whole partition's
+ // ObjectStoreUrl from the FIRST file only and then strips every other file down to its bare
+ // object-store path -- a file that actually lives under a different authority would
+ // silently read through the first file's store handle. Checked here rather than inside
+ // prepare_scan_store_and_files itself, which is shared with plain NativeScan and out of
+ // scope for this Delta-specific invariant.
+ check_same_object_store_authority(&spark_partition.partitioned_file)?;
+
+ let (object_store_url, mut files) =
+ planner.prepare_scan_store_and_files(common, &spark_partition)?;
+
+ // Translate deletion vectors into per-file ParquetAccessPlans so deleted
+ // rows are skipped inside the reader (composing, by intersection, with
+ // page-index pruning). Fetching the bitmaps and footers is async I/O;
+ // create_plan runs on the JNI task thread outside the tokio context, so
+ // block_on here is safe and keeps the scan a plain DataSourceExec.
+ if delta_partition
+ .partitioned_file
+ .iter()
+ .any(|f| f.dv.is_some())
+ {
+ let object_store_options: HashMap<String, String> = common
+ .object_store_options
+ .iter()
+ .map(|(k, v)| (k.clone(), v.clone()))
+ .collect();
+ let runtime_env = planner.session_ctx.runtime_env();
+ // `object_store_options` is the same map for every file this partition resolves a store
+ // for, so its hash is loop-invariant too: computed once here rather than once per file
+ // inside `prepare_object_store_with_configs`.
+ let object_store_config_hash = hash_object_store_configs(&object_store_options);
+
+ // Resolve every object store this partition's files touch -- the data
+ // files' shared authority (check_same_object_store_authority above
+ // has already verified every data file in this partition resolves to
+ // the same authority, so prepare_scan_store_and_files's
+ // first-file-only resolution is safe here) plus any on-disk deletion
+ // vector's authority, which may legitimately differ from the data
+ // files' and carries its own resolved store -- before entering the
+ // async DV runtime below. This MUST happen here, on the JNI thread
+ // outside the tokio runtime:
+ // constructing a cold S3 store issues its own internal
+ // Handle::block_on calls (credential-provider / bucket-region
+ // resolution), which panics when nested inside the
+ // get_runtime().block_on(...) a few lines down. See
+ // delta_dv::attach_access_plans's doc comment for the invariant this
+ // maintains -- the async path never builds a store.
+ let mut resolved_stores: HashMap<ObjectStoreUrl, Arc<dyn ObjectStore>> = HashMap::new();
+ // Tracks, per resolved ObjectStoreUrl, the userinfo (and raw URL, for the error
+ // message) of the first URL that resolved to it. This closure is the ONE place in this
+ // scan that sees both data-file and deletion-vector URLs together, so the
+ // store-identity collision check (see check_store_identity's doc comment) lives here
+ // rather than as an extension of check_same_object_store_authority above, which sees
+ // only data files and would hard-error the legitimate cross-bucket DV shape.
+ let mut store_identities: HashMap<ObjectStoreUrl, (String, String)> = HashMap::new();
+ let mut resolve_store =
+ |url: String| -> Result<(Path, Arc<dyn ObjectStore>), ExecutionError> {
+ let parsed_url = Url::parse(&url).map_err(|e| {
+ GeneralError(format!(
+ "Error parsing URL {}: {e}",
+ redacted_url_display(&url)
+ ))
+ })?;
+ let user_info = url_user_info(&parsed_url);
+ // The same s3a/alias rewrite the shared resolution path applies first, so the
+ // cheap key and cached-hit path below agree with what
+ // prepare_object_store_with_config_hash registers under. Re-parses the already
+ // parsed string only so the parse error above stays redacted.
+ let normalized = normalize_object_store_url(&url, &object_store_options)?;
+
+ // Cheap, I/O-free cache key: no config hashing, no global object-store-cache
+ // lock, no runtime_env registration. Checked against the LOCAL `resolved_stores`
+ // map below before ever paying for the expensive resolution path -- most files
+ // in a partition share the same authority as an already-resolved file.
+ let (url_key, _is_hdfs_scheme) = object_store_url_key(&normalized);
+ let parsed_url = normalized.url;
+ let store_url = ObjectStoreUrl::parse(url_key)?;
+ check_store_identity(&store_url, &user_info, &url, &mut store_identities)?;
+ if let Some(store) = resolved_stores.get(&store_url) {
Review Comment:
> Please use the same backend-aware identity for both lookup and insertion, preserving the distinction between native and Hadoop stores, and add a repeated-resolution regression.
Fixed. The registration URL derivation now lives in one function, `object_store_registration_url`, which `prepare_object_store_with_config_hash` and the Delta partition resolver both call, so the memo's lookup key, the identity-check key and the insert key are the same value by construction: the physical `file://` for the native local store, otherwise `{scheme}+comet-{hash}-{native|hdfs}://{authority}`. The closure is now a small resolver whose result says whether the memo hit, the store is fetched from the runtime under that same key so any drift errors instead of registering twice, and the partition logs its reference and hit counts at debug level.
Regressions: two S3 keys of one bucket resolve as a miss then a hit on one entry and the same `Arc`, a second bucket is a miss with a second entry, the same holds for local files and for a libhdfs-routed name node with the store seeded in the process cache, and a separate test pins that the helper returns the URL the prepare function registers under for s3a, file and an hdfs-listed scheme. The S3 memo test fails on the previous key.
This head is also rebased onto #5453, with the INT96 leaf stamp layered onto the instrumented metadata fetch and the object-store backend carried through the Delta arm, and onto #5850, which needed a `delta` entry in the CI policy table for the contrib job to route at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-6135529043
> > @dwsmith1983 could you fix conflicts?
>
> nm, I took the liberty of fixing them so we can get this merged
Sorry about that. It was around 3/4am so I wasn't up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5529092956
@sunchao dustin.william.smith@gmail.com
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5528110202
Fixed in 4a57686cc. The per file policies are still computed against the physical struct so the footer's INT96 ordinal attribution stays valid, but the wrapper now derives a requested mask by pairing the physical type with the requested logical type through exactly the pairings the struct conversion narrows (struct children by name under the adapter's case rules, list elements, map entries with matching key ordering); every physical leaf outside that mask is marked corrected and never checked. Anything the conversion does not narrow (large lists, fixed size lists, dictionaries, mismatched map ordering) keeps every leaf, so the mask is a superset of what the reader materializes. Your probe is the Spark regression: a converted metadata-free file with s.d modern and s.ts ancient under EXCEPTION read modes, select s.d reads natively and matches Spark, while selecting s.ts still fails loudly. Rust unit tests cover the same at the wrapper level plus a list of struct variant and the non
narrowed shapes.
Batteries at this head: 236/236 on both spark-3.5 and spark-4.0, 343 native tests, clippy and fmt clean. Independent review of the change caught that the first cut recursed through list and dictionary pairings the conversion never narrows and a first-match tie-break mismatch on folded name collisions; both fixed before this push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5535233691
Rebased onto main now that #5653 has merged. The branch is back to the single Delta commit (79b7b785c) and the description no longer lists the base commits. No code changes beyond the rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3920764778
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1707 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Always zero-I/O (plain propagated-conf read, no keystore): native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ /**
+ * `CommonConfigurationKeysPublic.HADOOP_SECURITY_CREDENTIAL_CLEAR_TEXT_FALLBACK`, default
+ * `true`, verified via `javap` against `hadoop-common` 3.3.4's
+ * `Configuration#getPasswordFromConfig`: `getPassword` only falls back to reading a plaintext
+ * conf value once `getBoolean(<this key>, true)` holds -- with the flag off, a plaintext value
+ * is invisible to every `getPassword`-based resolver, even when no credential provider is
+ * configured at all.
+ */
+ private val ClearTextFallbackKey = "hadoop.security.credential.clear-text-fallback"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * How Hadoop's OWN consumer reads one of the keys compared by [[s3ConfigDivergenceReason]],
+ * which decides how [[s3KeyDivergenceReason]] computes the Hadoop-effective side of its
+ * equality check. Exactly two consumer families exist among [[AllS3ConfigKeys]] in `hadoop-aws`
+ * 3.3.4, each verified via `javap`/CFR against the real call sites (cited per key on
+ * [[S3ConfigKeyConsumers]]). The tier must mirror the key's ACTUAL consumer: resolving a
+ * [[PropagatedOptionConsumer]] key through the wider `lookupPassword` cascade is NOT fail-safe
+ * for a value-EQUALITY comparator -- a long-form alias value Hadoop itself never reads can
+ * EQUAL native's resolution while Hadoop's true propagate-then-plain-get value differs, turning
+ * a real divergence into a wrongly-claimed scan (the endpoint `${...}`-redirect shape pinned in
+ * `DeltaScanContribSuite`).
+ */
+ private[delta] sealed trait S3ConfigConsumer
+
+ /**
+ * Read via `S3AUtils#lookupPassword(bucket, conf, baseKey)`, verified via `javap` against
+ * `hadoop-aws` 3.3.4: builds `longBucketKey = "fs.s3a.bucket." + bucket + "." + baseKey` (the
+ * FULL, already-`fs.s3a`-prefixed base key appended after the bucket segment) and reads it via
+ * `Configuration#getPassword` BEFORE the short-bucket key, keeping the long value whenever
+ * `getPassword` returns non-empty and only falling through to short-then-global otherwise.
+ * `getPassword` is Hadoop-credential-provider-aware and skips plaintext conf entirely when
+ * [[ClearTextFallbackKey]] is false. Modeled by [[hadoopLookupPasswordEffective]].
+ */
+ private[delta] case object LookupPasswordConsumer extends S3ConfigConsumer
+
+ /**
+ * Read via `S3AUtils#propagateBucketOptions` followed by a plain `Configuration#get`-family
+ * call (`getTrimmed`/`getBoolean`/`getClasses`) against the propagated view: the short bucket
+ * form wins only by having overwritten the global key during propagation, the long bucket form
+ * folds into an unread `fs.s3a.fs.s3a.*` key, and neither a credential provider nor
+ * [[ClearTextFallbackKey]] is ever consulted. Modeled as a plain `Configuration#get` on the
+ * [[propagateBucketOptions]] result, which also expands `${...}` references under that
+ * propagated view exactly like the real consumer.
+ */
+ private[delta] case object PropagatedOptionConsumer extends S3ConfigConsumer
+
+ /**
+ * Every `fs.s3a.*` base key that governs whether a claimed native scan actually behaves like
+ * Hadoop's own reader would, paired with the consumer family Hadoop resolves it through -- ONE
+ * list, with each key's resolution tier declared beside it, so a key can never sit in the
+ * comparator without a deliberate classification (adding one without picking a tier does not
+ * compile). The entries are every per-bucket `fs.s3a.*` base key native's S3 client's
+ * `get_config` (s3.rs) resolves, verified directly against its call sites:
+ * `extract_s3_config_options` (endpoint.region, path.style.access, endpoint,
+ * requester.pays.enabled), `lookup_provider_class` (the Comet-specific
+ * credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name).
+ *
+ * Tier assignments, each verified via `javap`/CFR against `hadoop-aws` 3.3.4:
+ * - access.key/secret.key/session.token: `S3AUtils#getAWSAccessKeys` and
+ * `MarshalledCredentialBinding#fromFileSystem` (reached from
+ * `TemporaryAWSCredentialsProvider`) resolve all three via `S3AUtils#lookupPassword` --
+ * [[LookupPasswordConsumer]].
+ * - aws.credentials.provider and assumed.role.credentials.provider:
+ * `S3AUtils#buildAWSProviderList` -> `loadAWSProviderClasses` -> plain
+ * `Configuration#getClasses` -- [[PropagatedOptionConsumer]].
+ * - assumed.role.arn/session.name: `AssumedRoleCredentialProvider`'s constructor reads both
+ * via plain `Configuration#getTrimmed` -- [[PropagatedOptionConsumer]].
+ * - endpoint (`S3AFileSystem`: `getTrimmed`), endpoint.region (`DefaultS3ClientFactory`:
+ * `getTrimmed`), path.style.access (`S3AFileSystem`: `getBoolean`) --
+ * [[PropagatedOptionConsumer]].
+ * - requester.pays.enabled: not read anywhere in `hadoop-aws` 3.3.4 (the constant does not
+ * even exist in its `Constants` class); later releases read it via plain `getBoolean`
+ * against the propagated conf, so the plain tier is both the faithful forward model and
+ * inert on 3.3.4 -- [[PropagatedOptionConsumer]].
+ * - comet.credential.provider.class: Comet's own activation key, plain conf read on both
+ * sides, never a Hadoop key at all -- [[PropagatedOptionConsumer]].
+ *
+ * SYNC NOTE: the key list must stay a superset of native's `NATIVE_S3A_CONFIG_PROPERTIES`
+ * constant (`native/core/src/parquet/objectstore/s3.rs`, property suffixes without the
+ * `fs.s3a.` prefix) -- `DeltaScanContribSuite`'s discovery-harness test asserts this
+ * mechanically against [[AllS3ConfigKeys]]. Literal strings, not the
+ * [[AwsCredentialsProviderKey]] / [[AssumedRoleCredentialsProviderKey]] vals declared below,
+ * purely to avoid a forward reference inside this `object` body; kept textually identical to
+ * those two constants.
+ */
+ private[delta] val S3ConfigKeyConsumers: Seq[(String, S3ConfigConsumer)] = Seq(
+ "fs.s3a.access.key" -> LookupPasswordConsumer,
+ "fs.s3a.secret.key" -> LookupPasswordConsumer,
+ "fs.s3a.session.token" -> LookupPasswordConsumer,
+ "fs.s3a.aws.credentials.provider" -> PropagatedOptionConsumer,
+ "fs.s3a.assumed.role.arn" -> PropagatedOptionConsumer,
+ "fs.s3a.assumed.role.session.name" -> PropagatedOptionConsumer,
+ "fs.s3a.assumed.role.credentials.provider" -> PropagatedOptionConsumer,
+ "fs.s3a.endpoint" -> PropagatedOptionConsumer,
+ "fs.s3a.endpoint.region" -> PropagatedOptionConsumer,
+ "fs.s3a.path.style.access" -> PropagatedOptionConsumer,
+ "fs.s3a.requester.pays.enabled" -> PropagatedOptionConsumer,
+ "fs.s3a.comet.credential.provider.class" -> PropagatedOptionConsumer)
+
+ /** The compared keys alone, in [[S3ConfigKeyConsumers]] order (discovery-harness surface). */
+ private[delta] val AllS3ConfigKeys: Seq[String] = S3ConfigKeyConsumers.map(_._1)
+
+ /**
+ * The short-bucket-then-global value resolved for `baseKey` under `bucket` from `hadoopConf`,
+ * skipping an empty value at either alias exactly like [[plainValue]]. NOT used by
+ * [[s3ConfigDivergenceReason]]/[[s3KeyDivergenceReason]] any more -- every key checked there
+ * resolves per its declared [[S3ConfigKeyConsumers]] tier (see [[s3KeyDivergenceReason]]),
+ * neither of which matches this function's read. This function's one remaining caller is
+ * [[shortThenGlobalOrReason]], which reads provider-CLASS strings (from the ORIGINAL,
+ * unpropagated conf) for name-support validation in [[providerClassReason]]/
+ * [[assumedRoleProviderClassReason]] -- by the time those run, [[s3ConfigDivergenceReason]] has
+ * already proven Hadoop's and native's effective values agree for the same key, so whichever of
+ * the two (equal) values this narrower read returns does not affect correctness there. NEVER
+ * used to compute native's own effective value -- see [[nativeShortThenGlobal]] for that.
+ */
+ private def shortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket` from the ORIGINAL, unpropagated `hadoopConf` -- `NativeConfig
+ * .extractObjectStoreOptions` forwards `Configuration#get`'s substituted value for every
+ * `fs.s3a.*` entry with no bucket-option propagation step of its own, so the original conf is
+ * the right input here. Unlike [[shortThenGlobal]]/[[plainValue]], this mirrors `get_config`
+ * faithfully: PRESENCE of the short-bucket key alone -- never its emptiness -- decides whether
+ * native falls back to the global key (`get_config` is a plain `HashMap::get`, which returns
+ * `Some` for a key explicitly set to `""`), so an explicitly empty or whitespace-only
+ * short-bucket value resolves to `Some("")` here and never falls through to global -- the
+ * OPPOSITE of Hadoop's own `getPassword`/`lookupPassword` semantics (see
+ * [[hadoopLookupPasswordEffective]]), which treat empty as absent and keep trying the next
+ * alias. The ONLY function used to compute native's effective value in
+ * [[s3KeyDivergenceReason]].
+ *
+ * Deliberately does NOT apply `get_config_trimmed`'s `.trim()` here: [[s3KeyDivergenceReason]]
+ * trims both this value and Hadoop's effective value together, symmetrically, at the point they
+ * are compared, rather than one-sidedly here -- trimming only the native side would flag a
+ * spurious divergence for a value neither side's whitespace actually changes the behavior of
+ * once each side's own downstream parsing normalizes it (e.g. Hadoop's own multi-line
+ * `fs.s3a.aws.credentials.provider` default, which both Hadoop and native additionally trim per
+ * comma-separated entry after splitting), while a one-sided trim would make an
+ * otherwise-identical default value look diverged for every bucket, never claiming natively at
+ * all.
+ */
+ private def nativeShortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ Option(hadoopConf.get(shortKey)).orElse(Option(hadoopConf.get(baseKey)))
+ }
+
+ /**
+ * Faithful in-memory replica of `S3AUtils#propagateBucketOptions` (`hadoop-aws`), which
+ * `S3AFileSystem#initialize` calls FIRST, before any option or credential is read:
+ * `Configuration conf = propagateBucketOptions(originalConf, bucket); ...; setConf(conf);` --
+ * every subsequent `conf.get`/`getPassword` call in that filesystem instance, including
+ * `${...}` variable substitution, resolves against this propagated view, not the original conf.
+ * `hadoop-aws` is not on this module's runtime classpath (see the string-literal-keys note
+ * above), so `S3AUtils#propagateBucketOptions` cannot be called directly; this reproduces its
+ * logic verbatim using only `hadoop-common`'s `Configuration`:
+ * {{{
+ * public static Configuration propagateBucketOptions(Configuration source, String bucket) {
+ * final String bucketPrefix = FS_S3A_BUCKET_PREFIX + bucket + '.';
+ * final Configuration dest = new Configuration(source);
+ * for (Map.Entry<String, String> entry : source) {
+ * final String key = entry.getKey();
+ * final String value = entry.getValue(); // the (unexpanded) value
+ * if (!key.startsWith(bucketPrefix) || bucketPrefix.equals(key)) continue;
+ * final String stripped = key.substring(bucketPrefix.length());
+ * if (stripped.startsWith("bucket.") || "impl".equals(stripped)) {
+ * // ignored
+ * } else {
+ * final String generic = FS_S3A_PREFIX + stripped;
+ * dest.set(generic, value, ...); // overwrites any existing global value
+ * }
+ * }
+ * return dest;
+ * }
+ * }}}
+ * Note the LONG bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) folds to an unread
+ * `fs.s3a.fs.s3a.<key>` key here too, exactly like the real method -- `stripped` already starts
+ * with `fs.s3a.` in that case, so prepending `fs.s3a.` again produces a key nothing ever reads.
+ */
+ private def propagateBucketOptions(hadoopConf: Configuration, bucket: String): Configuration = {
+ val bucketPrefix = s"fs.s3a.bucket.$bucket."
+ val dest = new Configuration(hadoopConf)
+ hadoopConf.iterator().asScala.foreach { entry =>
+ val key = entry.getKey
+ if (key.startsWith(bucketPrefix) && key != bucketPrefix) {
+ val stripped = key.substring(bucketPrefix.length)
+ if (!stripped.startsWith("bucket.") && stripped != "impl") {
+ dest.set(s"fs.s3a.$stripped", entry.getValue)
+ }
+ }
+ }
+ dest
+ }
+
+ /**
+ * Canonical and deprecated Hadoop S3A encryption-algorithm config keys, verified via `javap`
+ * against `hadoop-aws` 3.3.4's `org.apache.hadoop.fs.s3a.Constants`: `S3_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.encryption.algorithm"` (canonical) and `SERVER_SIDE_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.server-side-encryption-algorithm"` (DEPRECATED -- note the hyphen before "algorithm",
+ * unlike the corresponding `*.key` constants below, which both use a `.key` suffix).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, same
+ * rationale as [[HadoopCredentialProviderPathKey]].
+ */
+ private val S3EncryptionAlgorithmKey = "fs.s3a.encryption.algorithm"
+ private val DeprecatedS3EncryptionAlgorithmKey = "fs.s3a.server-side-encryption-algorithm"
+
+ /**
+ * The exact strings `S3AEncryptionMethods#getMethod` accepts, verified via `javap`/CFR against
+ * `hadoop-aws` 3.3.4's `S3AEncryptionMethods` enum: `NONE("")`, `SSE_S3("AES256", serverSide =
+ * true, requiresSecret = false)`, `SSE_KMS("SSE-KMS", serverSide = true, requiresSecret =
+ * false)`, `SSE_C("SSE-C", serverSide = true, requiresSecret = true)`, `CSE_KMS("CSE-KMS",
+ * serverSide = false, requiresSecret = true)`, `CSE_CUSTOM("CSE-CUSTOM", serverSide = false,
+ * requiresSecret = true)`. `getMethod` parses case-insensitively
+ * (`values().find(_.getMethod.equalsIgnoreCase(algorithm))`), matched below the same way.
+ *
+ * ALLOWLIST, not a blocklist (replaces the former SSE-C-only blocklist): only the algorithms S3
+ * decrypts transparently on GET/HEAD given read permission alone, with NO extra request header
+ * and NO client-side step, are safe for a native scan that forwards none of Hadoop's
+ * `fs.s3a.encryption.*`/`fs.s3a.server-side-encryption*` options --
+ * - `AES256` (SSE_S3, `serverSide = true`): plain server-side encryption, transparent on GET.
+ * - `SSE-KMS` (SSE_KMS, `serverSide = true`): server-side, KMS-managed key, transparent on
+ * GET given KMS decrypt permission (no header).
+ * - `DSSE-KMS`: NOT present in this enum on `hadoop-aws` 3.3.4 (confirmed by the six values
+ * listed above) -- `S3AEncryptionMethods.getMethod("DSSE-KMS")` throws
+ * `IOException("Unknown encryption algorithm DSSE-KMS")` on this version, so
+ * `S3AUtils#buildEncryptionSecrets` (and therefore Hadoop's own reader) already fails
+ * before ever reading such a table under 3.3.4, meaning this string can never actually be
+ * the resolved value on the declared target version -- admitting it here is inert there.
+ * Included anyway, forward-compatible, for a newer `hadoop-aws` on the runtime classpath (a
+ * later Hadoop release; this module has no compile-time `hadoop-aws` dependency, see the
+ * string-literal-keys note above) where DSSE-KMS is a real, dual-layer, server-side
+ * algorithm decrypted transparently on GET the same way SSE-KMS is. Every other value
+ * declines: `SSE-C` (SSE_C is `serverSide = true` in Hadoop's own enum, but `requiresSecret
+ * \= true` -- S3 rejects a GET/HEAD for an SSE-C object outright (400 Bad Request) unless
+ * the customer key is resent as a request header on every call, so a native scan that never
+ * learns the key cannot succeed at all, where Hadoop's own reader -- whose request factory
+ * attaches the key -- would), `CSE-KMS`/`CSE-CUSTOM` (`serverSide = false`: client-side
+ * encryption decrypts object bytes locally in the SDK layer, which the native Parquet
+ * reader has no equivalent of -- it would read raw ciphertext), and any future/unknown
+ * value (a value `S3AEncryptionMethods.getMethod` itself would reject is certainly not one
+ * of the three confirmed-transparent algorithms above; declining is the only safe default
+ * for anything this gate cannot positively confirm).
+ */
+ private val AllowedEncryptionAlgorithms: Set[String] = Set("AES256", "SSE-KMS", "DSSE-KMS")
+
+ /**
+ * `bucket`'s effective encryption-algorithm key and value under `hadoopConf`, or `None` when
+ * neither the canonical nor deprecated key is set anywhere consulted. Mirrors
+ * `S3AUtils#buildEncryptionSecrets`'s real resolution order, verified via `javap`/CFR
+ * decompilation of `hadoop-aws` 3.3.4's `S3AUtils.class`:
+ * {{{
+ * String algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.server-side-encryption-algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.server-side-encryption-algorithm");
+ * }}}
+ * i.e. bucket-tier (canonical, then deprecated), THEN global-tier (canonical, then deprecated)
+ * -- the two tiers are never interleaved key-by-key, so this must stay two explicit bucket-tier
+ * lookups followed by two explicit global-tier lookups, not a single
+ * [[hadoopLookupPasswordEffective]] call per key (which would let an unset canonical bucket key
+ * fall through straight to the canonical GLOBAL value ahead of a SET deprecated bucket key, the
+ * wrong answer).
+ *
+ * THE FIX for the SSE-C long-bucket-alias gap is entirely inside the bucket tier:
+ * `lookupBucketSecret` itself is long-then-short, decompiled from `hadoop-aws` 3.3.4's
+ * `S3AUtils.class`:
+ * {{{
+ * // longBucketKey = fs.s3a.bucket.B.fs.s3a.<key>
+ * String longBucketKey = String.format(BUCKET_PATTERN, bucket, baseKey);
+ * String initialVal = getPassword(conf, longBucketKey, null, null);
+ * // shortBucketKey = fs.s3a.bucket.B.<key>
+ * String shortBucketKey = String.format(BUCKET_PATTERN, bucket, subkey);
+ * // keeps initialVal (the LONG value) if non-empty
+ * return getPassword(conf, shortBucketKey, initialVal, null);
+ * }}}
+ * i.e. the SAME long-bucket-key construction and long-wins-if-nonempty semantics as
+ * `S3AUtils#lookupPassword` (see [[LookupPasswordConsumer]]/[[hadoopLookupPasswordEffective]])
+ * -- the encryption algorithm is NOT one of the keys that flows through
+ * `S3AUtils#propagateBucketOptions` (which folds an unrelated per-bucket LONG form into an
+ * unread key). An earlier version of this function modeled the bucket tier as SHORT-only,
+ * documented as "the LONG bucket form is genuinely never consulted for this key" -- that
+ * documentation was wrong (this decompilation supersedes it): a bucket configured only via
+ * `fs.s3a.bucket.B.fs.s3a.encryption.algorithm=SSE-C` bypassed the SSE-C gate entirely, because
+ * Hadoop's own reader DOES read that long form (and picks SSE-C), while this function reported
+ * `None` (nothing set) and the allowlist check below never even ran.
+ *
+ * The canonical-vs-deprecated distinction below is frequently moot in practice: `hadoop-aws`'s
+ * `S3AFileSystem.addDeprecatedKeys()` statically registers `fs.s3a.server-side-encryption-*` as
+ * `Configuration`-level deprecated aliases of `fs.s3a.encryption.*` (verified via `javap`), a
+ * registration that lives in a static field on Hadoop's `Configuration` class -- process-wide
+ * once `S3AFileSystem`'s class has loaded anywhere in the JVM, which a real scan has always
+ * already done by the time this gate runs, since reading the S3 table at all requires loading
+ * that class. Once active, `Configuration#get` resolves either literal key to the identical
+ * value transparently, making the two-key cascade below redundant (but harmless) for that case;
+ * it remains the operative path only when nothing else in the process has loaded
+ * `S3AFileSystem` yet.
+ *
+ * ALSO walks the Hadoop-credential-provider (JCEKS) path via [[resolveViaCredentialAliases]]
+ * for each of the four lookups below, matching `lookupBucketSecret`/`lookupPassword`'s real
+ * per-alias `getPassword` calls (quoted above) exactly: both are `getPassword`, not plain
+ * `Configuration#get`, so a bucket storing the algorithm name ONLY in a JCEKS keystore is
+ * exactly as real a Hadoop deployment shape for this key as it is for the credential keys
+ * [[hadoopLookupPasswordEffective]] already covers -- there is nothing algorithm-specific that
+ * makes JCEKS storage implausible here, so an earlier version of this function skipping it
+ * (documented at the time as "the algorithm NAME is not credential-sensitive data, so storing
+ * it in a keystore is not a realistic Hadoop deployment pattern") was an unjustified, narrower
+ * read than Hadoop's own resolver actually performs, under-declining a bucket whose algorithm
+ * is keystore-only. [[resolveViaCredentialAliases]]'s Arm B/C split still means this is zero
+ * extra I/O for the common case: keystore I/O only happens when a Hadoop credential-provider
+ * path is actually configured for the bucket, contained in that function's own try/catch.
+ * `bucketTier`/`globalTier` return `Left` (propagated straight through by [[orElseTier]]) when
+ * [[resolveViaCredentialAliases]] cannot safely verify a tier at all (an S3A-scoped provider
+ * path, or a corrupt/unreadable global keystore) -- correctly short-circuiting the whole
+ * cascade with a decline rather than silently falling through to a later tier that might look
+ * unset only because the true value was unverifiable.
+ */
+ private def effectiveEncryptionAlgorithm(
+ hadoopConf: Configuration,
+ bucket: String): Either[String, Option[(String, String)]] = {
+ def bucketTier(baseKey: String): Either[String, Option[(String, String)]] = {
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(longKey, shortKey))
+ .map(_.map(baseKey -> _))
+ }
+ def globalTier(baseKey: String): Either[String, Option[(String, String)]] =
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(baseKey))
+ .map(_.map(baseKey -> _))
+
+ // Short-circuits on Left (unverifiable tier) or Right(Some(_)) (resolved); only Right(None)
+ // (tier definitively unset) falls through to `next`, mirroring buildEncryptionSecrets's
+ // sequential `if (algorithm == null) algorithm = ...` cascade exactly.
+ def orElseTier(
+ current: Either[String, Option[(String, String)]],
+ next: => Either[String, Option[(String, String)]])
+ : Either[String, Option[(String, String)]] =
+ current match {
+ case Left(reason) => Left(reason)
+ case Right(Some(value)) => Right(Some(value))
+ case Right(None) => next
+ }
+
+ orElseTier(
+ bucketTier(S3EncryptionAlgorithmKey),
+ orElseTier(
+ bucketTier(DeprecatedS3EncryptionAlgorithmKey),
+ orElseTier(
+ globalTier(S3EncryptionAlgorithmKey),
+ globalTier(DeprecatedS3EncryptionAlgorithmKey))))
+ }
+
+ private def unsupportedEncryptionAlgorithmDeclineReason(
+ bucket: String,
+ algorithmKey: String,
+ algorithm: String): String =
+ s"Native Delta scan does not support $algorithmKey=$algorithm for $bucket " +
+ "(the native S3 client only supports unencrypted objects and S3's transparent " +
+ "server-side algorithms -- AES256/SSE-S3, SSE-KMS, and DSSE-KMS decrypt on GET/HEAD given " +
+ "read permission alone, with no extra request header; SSE-C additionally requires the " +
+ "customer-provided key resent as a header on every GET/HEAD request, which the native S3 " +
+ "client's extract_s3_config_options never forwards, and CSE-KMS/CSE-CUSTOM decrypt object " +
+ "bytes client-side, a layer the native Parquet reader does not have -- any of these would " +
+ "fail outright or silently read ciphertext where Hadoop's own reader succeeds)"
+
+ /**
+ * First reason any bucket among `uris` is configured for an encryption algorithm the native S3
+ * client cannot safely read, or `None` when claimable. Allowlist-based (see
+ * [[AllowedEncryptionAlgorithms]]): only `AES256`/`SSE-KMS`/`DSSE-KMS` (and unset/empty) pass;
+ * every other resolved value -- `SSE-C`, `CSE-KMS`, `CSE-CUSTOM`, or any unrecognized future
+ * algorithm string -- declines. Deliberately NOT a blocklist keyed on `SSE-C` alone: an
+ * allowlist is safe by construction against a Hadoop release adding a new encryption method
+ * this gate has never heard of, where a blocklist would silently admit it. Never interpolates a
+ * resolved key value, only key names, the bucket, and the (non-secret) algorithm name. Declines
+ * on a `Left` from [[effectiveEncryptionAlgorithm]] too (an unverifiable credential-provider
+ * arm, e.g. an S3A-scoped provider path or a corrupt/unreadable global keystore) -- the
+ * algorithm cannot be ruled safe when it cannot be read at all.
+ */
+ private[delta] def unsupportedEncryptionAlgorithmReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ effectiveEncryptionAlgorithm(hadoopConf, bucket) match {
+ case Left(reason) => Some(reason)
+ case Right(None) => None
+ case Right(Some((key, value))) =>
+ if (!AllowedEncryptionAlgorithms.exists(_.equalsIgnoreCase(value))) {
+ Some(unsupportedEncryptionAlgorithmDeclineReason(bucket, key, value))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Canonical Hadoop S3A HTTP-proxy host config key, verified via CFR decompilation of
+ * `hadoop-aws` 3.3.4's `S3AUtils.class` (`initProxySupport`):
+ * {{{
+ * String proxyHost = conf.getTrimmed("fs.s3a.proxy.host", "");
+ * int proxyPort = conf.getInt("fs.s3a.proxy.port", -1);
+ * if (!proxyHost.isEmpty()) {
+ * ...
+ * String proxyUsername =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.username", null, null);
+ * String proxyPassword =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.password", null, null);
+ * ...
+ * }
+ * }}}
+ * `fs.s3a.proxy.host`/`fs.s3a.proxy.port` resolve via a PLAIN, non-bucket-scoped, non-JCEKS
+ * `Configuration#getTrimmed`/`getInt` call -- NOT `lookupPassword` -- against whatever conf
+ * `S3AFileSystem#initialize` already ran through `propagateBucketOptions` before
+ * `createAwsConf`/`initProxySupport` ever runs; only the SIBLING `fs.s3a.proxy.username`/
+ * `fs.s3a.proxy.password` keys go through `lookupPassword` (bucket long/short/global,
+ * JCEKS-aware). So the host is bucket-aware only via `propagateBucketOptions`'s short-bucket-
+ * form folding, never the long-bucket form, and never a credential-provider read -- the SAME
+ * shape as `endpoint`/`path.style.access` ([[PropagatedOptionConsumer]], see
+ * [[S3ConfigKeyConsumers]]'s doc), not the credential family. `hadoop-aws` 3.4.x (the Spark 4.x
+ * profiles' version) moves this code to `AWSClientConfig#createProxyConfiguration`/
+ * `#createAsyncProxyConfiguration` but keeps the exact same reads, verified via `javap` against
+ * 3.4.2: `conf.getTrimmed("fs.s3a.proxy.host", "")` for the host, `S3AUtils.lookupPassword` for
+ * username/password only.
+ *
+ * [[proxyGateReason]] therefore resolves the host EXACTLY like its real consumer -- plain
+ * `Configuration#getTrimmed` on the [[propagateBucketOptions]] result, no provider arms, no
+ * [[ClearTextFallbackKey]] handling -- rather than through the wider `lookupPassword` cascade
+ * an earlier version reused here. The wider cascade was wrong in BOTH directions for this key:
+ * with only the GLOBAL Hadoop provider path set and [[ClearTextFallbackKey]] false,
+ * `getPassword` hides a plaintext host that `getTrimmed` serves to Hadoop anyway (a missed
+ * decline, the exact bypass this gate exists to close -- native has NO HTTP-proxy support of
+ * any kind, no `fs.s3a.proxy.*` key is read anywhere in `s3.rs`); and an S3A-scoped provider
+ * path or a lone long-form bucket alias declined a bucket whose real consumer can never see a
+ * host from either source (pure over-refusal -- no keystore can supply the host to a plain
+ * `getTrimmed`, and the long form folds into the unread `fs.s3a.fs.s3a.proxy.host`).
+ */
+ private val S3ProxyHostKey = "fs.s3a.proxy.host"
+
+ private def unsupportedProxyReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key configured for $bucket (the native S3 client has " +
+ "no HTTP proxy support at all -- no fs.s3a.proxy.* key is read anywhere in its object " +
+ "store layer -- so a claimed scan would connect to S3 directly instead of routing through " +
+ "the configured proxy, either bypassing an egress/network-segmentation policy or simply " +
+ "failing to reach the endpoint)"
+
+ /**
+ * First reason any bucket among `uris` has an HTTP proxy configured via [[S3ProxyHostKey]], or
+ * `None` when claimable. Reads the host EXACTLY like its real consumer (see
+ * [[S3ProxyHostKey]]'s doc): plain `Configuration#getTrimmed` on the [[propagateBucketOptions]]
+ * result, so this gate is always zero-I/O -- no credential provider is ever consulted for the
+ * host, because none ever supplies it to Hadoop either. Ordered alongside
+ * [[unsupportedEncryptionAlgorithmReason]] among the conf-only gates, ahead of
+ * [[s3ConfigDivergenceReason]]. The try/catch guards `Configuration#get`'s
+ * `IllegalStateException` on a `${...}` substitution cycle, same as [[s3KeyDivergenceReason]].
+ * Never interpolates a resolved value: the proxy HOST is not secret, but naming it here would
+ * be a strange place to first surface it, and proxy CREDENTIALS
+ * (`fs.s3a.proxy.username`/`fs.s3a.proxy.password`, not read by this gate at all -- the whole
+ * point of gating on the host is that a non-empty host declines before any proxy credential
+ * would ever need to be forwarded) must never appear in a decline reason regardless.
+ */
+ private[delta] def proxyGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI],
+ propagatedConfCache: MutableMap[String, Configuration] = MutableMap.empty)
+ : Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ try {
+ val propagatedConf =
+ propagatedConfCache.getOrElseUpdate(
+ bucket,
+ propagateBucketOptions(hadoopConf, bucket))
+ if (propagatedConf.getTrimmed(S3ProxyHostKey, "").nonEmpty) {
+ Some(unsupportedProxyReason(bucket, S3ProxyHostKey))
+ } else {
+ None
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableValueReason(bucket, S3ProxyHostKey, e))
+ }
+ }
+ }
+ }
+
+ /**
+ * `baseKey`'s long, short, then global per-bucket aliases, in Hadoop's own resolution order.
+ */
+ private def longThenShortThenGlobalAliases(bucket: String, baseKey: String): Seq[String] = {
+ val suffix = baseKey.stripPrefix("fs.s3a.")
+ Seq(s"fs.s3a.bucket.$bucket.fs.s3a.$suffix", s"fs.s3a.bucket.$bucket.$suffix", baseKey)
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * The three-way Hadoop credential-provider-path precheck shared by every `getPassword`-based
+ * resolution below, factored out of what used to be [[hadoopLookupPasswordEffective]]'s own
Review Comment:
Trimmed. Those comments and a few similar ones now describe present behavior only; anything narrating how the code got here is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3920770142
##########
native/proto/src/proto/operator.proto:
##########
@@ -77,6 +77,8 @@ message Operator {
// zero runtime cost.
ContribScan contrib_scan = 200;
}
+
+ reserved "delta_scan";
Review Comment:
Dropped. You are right that it never reserved anything real.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3869546109
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,981 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.ListBuffer
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands. Anything else on the protocol declines the
+ * table. Note `deletionVectors` and `columnMapping` are declined separately (below) so their
+ * fallback reasons are specific.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: this is
+ * the first gate on every V1 scan and must stay inert when delta-spark is absent from the
+ * classpath, where `classOf[DeltaParquetFileFormat]` would raise NoClassDefFoundError inside
+ * CometScanRule and take down every parquet scan in the session. A name match proves
+ * delta-spark is present, so Delta types used past this gate are safe.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Returns the first reason this Delta scan cannot go native, or None when claimable. Only
+ * called when [[isDeltaScan]] is true. `scanHelper` is the same [[CometScanExec]] the caller
+ * builds to drive [[CometDeltaNativeScan.convert]] on a claim, reused here to resolve the
+ * scan's selected files for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Descriptor deserialization is expensive; hoisted once so it runs at most once per claim
+ // attempt. `lazy` since most scans are not DV-shaped and gates that return earlier should not
+ // pay for it.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported (the same type checker core's own
+ // FileSourceScanExec path runs) so scan-time type gates -- notably the default-on
+ // COMET_PARQUET_UNSIGNED_SMALL_INT_CHECK safety fallback for ShortType, plus the
+ // collation and shredded-variant-struct gates -- apply identically whether the scan is
+ // claimed through core or through this contrib. A pure in-memory schema check with no
+ // file or store I/O, so it runs first, ahead of every other gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated.
+ val cmMode = metadata.columnMappingMode.name
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder uses the
+ // required schema verbatim as output: name-sensitive expressions (e.g. to_json) would leak
+ // physical names into results. Needs a rename adapter for the logical schema; decline until
+ // then.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bound native's memory for expanded DV row selectors (delta_dv.rs), bounded above by
+ // 2*cardinality + #row-groups; the descriptor's cardinality is a sound, pessimistic upper
+ // bound. The conf below makes an over-pessimistic decline recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities; the native builder resolves the
+ // scan's ObjectStoreUrl from the FIRST selected file only, so a later file under a different
+ // store would silently read through the wrong handle. Force file listing (scanHelper is
+ // already built for the claim path) and decline rather than risk it.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // Credentials that resolve ONLY through a Hadoop credential provider (JCEKS et al.) are
+ // invisible to the plain-conf extraction forwarded to the native S3 client; reuses hadoopConf
+ // from the encryption gate above.
+ val credentialReason = credentialAliasReason(hadoopConf, dataFileUris ++ dvUris)
+ if (credentialReason.isDefined) {
+ return credentialReason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, deserialized once and
+ * normalized to absolute on-disk paths via `copyWithAbsolutePath`. Returns `Seq.empty` for the
+ * plain shape ([[CometDeltaNativeScan.isDvShape]] false). Shared by the DV cardinality gate and
+ * [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset. Hoisted so [[declineReason]]'s
+ * root-path and selected-file scheme gates share one parsed set.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged: such a URI cannot come from a Hadoop-backed source. Factored out of
+ * [[declineReason]]'s two scheme gates so both share one predicate, unit-testable without a
+ * Spark session.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt). See [[declineReason]]'s call site for ordering vs. the authority gates
+ * below.
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme plus the raw
+ * authority -- userinfo, host, port -- lowercased so `S3A://Bucket:1234` and
+ * `s3a://bucket:1234` collapse), or `None` when they share one. `file://` paths carry no
+ * authority, so local scans across many directories are unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the individually-parsed host/port/userinfo fields: `getAuthority` already
+ * includes userinfo, so containers on the same storage account don't collapse together; and
+ * `getHost` (and `getUserInfo`/`getPort`) return `null` for the whole authority when it doesn't
+ * conform to RFC 3986's `reg-name` syntax (e.g. an underscore in a GCS bucket name,
+ * `gs://my_bucket`), silently collapsing distinct buckets into one empty-host key. A `null`
+ * authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`: as with [[uriAuthority]], the structured getters return
+ * `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an underscore
+ * in a GCS bucket name), hiding a real userinfo component. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme` + `://` + a literal `***` + `@host[:port]`, for
+ * embedding in a decline reason. NEVER interpolate `uri.getAuthority` or [[uriUserInfo]]
+ * directly into a reason string: doing so would leak credentials embedded as URI userinfo (e.g.
+ * `s3a://AKIA...:secret@bucket`) into the SQL plan's explain output, fallback-reason logging,
+ * or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle -- normally a missing-object
+ * error, but silently wrong data if a same-named object exists under both.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted by [[credentialAliasReason]] below. `hadoop-aws` is
+ * NOT on this module's runtime classpath (`spark-hadoop-cloud` is test-scope only), so
+ * `org.apache.hadoop.fs.s3a.Constants` must never be referenced here -- doing so would raise
+ * `NoClassDefFoundError` and take down every Delta scan in a session with no S3 dependency at
+ * all, not just S3 ones.
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key
+ * (`fs.s3a.bucket.B.<base key minus its "fs.s3a." prefix>`) -- see [[s3aCredentialAliases]] for
+ * why both must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ /**
+ * The aliases the native S3 credentials provider chain tries per bucket (short bucket form,
+ * then global), PLUS the LONG bucket form Hadoop's `S3AUtils#lookupPassword` also consults.
+ * Hadoop resolves long before short before global, keeping a non-empty long value over the
+ * short/global ones (`S3AUtils#getPassword` returns a non-empty `val` unchanged); native reads
+ * only short+global. So a JCEKS entry set ONLY under the long alias is resolved by Hadoop but
+ * invisible to native -- a shadowed value under ANY alias here means the caller must decline.
+ * List order only affects which alias name appears in the reason.
+ */
+ private def s3aCredentialAliases(bucket: String): Seq[String] =
+ Seq(
+ s"fs.s3a.bucket.$bucket.fs.s3a.access.key",
+ s"fs.s3a.bucket.$bucket.fs.s3a.secret.key",
+ s"fs.s3a.bucket.$bucket.fs.s3a.session.token",
+ s"fs.s3a.bucket.$bucket.access.key",
+ s"fs.s3a.bucket.$bucket.secret.key",
+ s"fs.s3a.bucket.$bucket.session.token",
+ "fs.s3a.access.key",
+ "fs.s3a.secret.key",
+ "fs.s3a.session.token")
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority (host, minus any
+ * userinfo or port), or `None` when `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority
+ * manually (last `@`, then last `:`) rather than using `URI#getHost`: the same RFC 3986
+ * `reg-name` pitfall as [[uriAuthority]] applies (an underscore, valid in an S3 bucket name,
+ * makes `getHost` return `null` for the whole authority).
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ /**
+ * The three per-bucket S3A credential base keys the native S3 client's `get_config` (`s3.rs`)
+ * resolves: short bucket key first, then global (native never reads the long bucket key).
+ */
+ private val PlainCredentialBaseKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket`. Used by the credential-provider-class gates below, which read general S3A options
+ * the same way native does.
+ */
+ private def effectiveOptionValue(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ private def plainLongFormCredentialDivergenceReason(bucket: String, longKey: String): String =
+ "Native Delta scan cannot forward long-form bucket credentials for " +
+ s"$bucket ($longKey is set but the native S3 client only reads the short bucket and " +
+ "global keys, so its credentials would differ from Hadoop's)"
+
+ /**
+ * Zero-I/O plain-value divergence check (see [[s3aCredentialAliases]] for the long-before-short
+ * precedence). Native's `get_config` never reads the long form, so it can silently diverge from
+ * Hadoop's effective value. Declines when the long form is set and the two sides disagree
+ * (short set to the SAME value as long passes). Never interpolates a resolved value, only the
+ * key name.
+ *
+ * This applies to credentials ONLY. Hadoop resolves them via `S3AUtils#lookupPassword`, where
+ * the long bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) WINS when set -- a real, silent
+ * divergence from native, which never reads it. Every OTHER `fs.s3a` option instead flows
+ * through `S3AUtils#propagateBucketOptions`, which strips only ONE `fs.s3a.bucket.B.` prefix
+ * layer: a long-form key there folds into `fs.s3a.fs.s3a.<key>`, a key nothing else in Hadoop
+ * ever reads, so the long form is inert for those options -- Hadoop's own effective value
+ * already reduces to short.orElse(global), matching native. No gate is needed (or present) for
+ * that wider set; see DeltaScanContribSuite's pinned control test for the long-form endpoint
+ * case.
+ */
+ private def plainLongFormCredentialReason(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ PlainCredentialBaseKeys.foldLeft(Option.empty[String]) { (declined, baseKey) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val long = plainValue(hadoopConf, longKey)
+ if (long.isEmpty) {
+ None
+ } else {
+ val short = plainValue(hadoopConf, shortKey)
+ val global = plainValue(hadoopConf, baseKey)
+ val hadoopEffective = long.orElse(short).orElse(global)
+ val nativeEffective = short.orElse(global)
+ if (hadoopEffective != nativeEffective) {
+ Some(plainLongFormCredentialDivergenceReason(bucket, longKey))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * String-literal mirror of every credential-provider class name s3.rs's
+ * `build_aws_credential_provider_metadata` and `is_anonymous_credential_provider` recognize
+ * (Hadoop S3A names plus AWS SDK v1/v2 names). `hadoop-aws` is NOT on this module's runtime
+ * classpath (see the note above [[HadoopCredentialProviderPathKey]]), so these are string
+ * literals, never `org.apache.hadoop.fs.s3a.auth.*` or AWS SDK provider `classOf` references.
+ */
+ private val SupportedCredentialProviderClasses: Set[String] = Set(
+ "org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider",
+ "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider",
+ "com.amazonaws.auth.ContainerCredentialsProvider",
+ "com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper",
+ "software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider",
+ "com.amazonaws.auth.InstanceProfileCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider",
+ "com.amazonaws.auth.EnvironmentVariableCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider",
+ "com.amazonaws.auth.WebIdentityTokenCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider",
+ "com.amazonaws.auth.profile.ProfileCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider",
+ "com.amazonaws.auth.AnonymousAWSCredentials")
+
+ private val AnonymousCredentialProviderClasses: Set[String] = Set(
+ "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider",
+ "com.amazonaws.auth.AnonymousAWSCredentials")
+
+ private val HadoopAssumedRoleProviderClass =
+ "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider"
+
+ private val AwsCredentialsProviderKey = "fs.s3a.aws.credentials.provider"
+ private val AssumedRoleCredentialsProviderKey = "fs.s3a.assumed.role.credentials.provider"
+
+ /**
+ * Splits a Hadoop-style comma-separated credential-provider-class list the same way s3.rs's
+ * `parse_credential_provider_names` does: split on comma, trim each entry, drop empties.
+ */
+ private def parseProviderClassNames(value: String): Seq[String] =
+ value.split(",").map(_.trim).filter(_.nonEmpty).toSeq
+
+ private def unsupportedProviderReason(bucket: String, key: String, className: String): String =
+ s"Native Delta scan does not support the credential provider class $className " +
+ s"configured via $key for $bucket (the native S3 client only supports a fixed set of " +
+ "provider classes; an unsupported class would fail at scan execution time, after the " +
+ "scan was already claimed, rather than at planning time)"
+
+ private def mixedAnonymousProviderReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key for $bucket naming an anonymous credential " +
+ "provider together with any other provider (the native S3 client rejects this " +
+ "combination at scan execution time)"
+
+ private def anonymousAssumedRoleProviderReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support an anonymous credential provider in $key for " +
+ s"$bucket (the native S3 client does not allow an anonymous provider as the base " +
+ "credentials for an assumed-role chain)"
+
+ private def unsupportedProviderNameReason(
+ bucket: String,
+ key: String,
+ names: Seq[String]): Option[String] =
+ names
+ .find(name => !SupportedCredentialProviderClasses.contains(name))
+ .map(unsupportedProviderReason(bucket, key, _))
+
+ /**
+ * Decline reason when `bucket`'s effective (short-bucket-then-global; see
+ * [[effectiveOptionValue]]) `assumed.role.credentials.provider` names an unsupported class, or
+ * an anonymous one (native's `build_assume_role_credential_provider_metadata` rejects ANY
+ * anonymous entry here, not just a mix). Unset defaults to native's own hardcoded
+ * `[SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider]` fallback, both
+ * always supported, so `None` is safe.
+ */
+ private def assumedRoleProviderClassReason(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ effectiveOptionValue(hadoopConf, bucket, AssumedRoleCredentialsProviderKey) match {
+ case None => None
+ case Some(value) =>
+ val names = parseProviderClassNames(value)
+ unsupportedProviderNameReason(bucket, AssumedRoleCredentialsProviderKey, names).orElse {
+ if (names.exists(AnonymousCredentialProviderClasses.contains)) {
+ Some(anonymousAssumedRoleProviderReason(bucket, AssumedRoleCredentialsProviderKey))
+ } else {
+ None
+ }
+ }
+ }
+ }
+
+ /**
+ * Decline reason when `bucket`'s effective `aws.credentials.provider` names a class native's
+ * `build_aws_credential_provider_metadata` (s3.rs) does not recognize, mixes an anonymous
+ * provider with any other provider (native's `build_credential_provider` rejects this
+ * combination), or -- when `AssumedRoleCredentialProvider` is among the names -- its
+ * `assumed.role.credentials.provider` sub-chain has the same problem. An unset/empty value is
+ * fine: native falls back to its own default AWS SDK provider chain. Checked so an unsupported
+ * class or invalid combination declines at planning time instead of erroring during scan
+ * execution, after the scan was already claimed.
+ */
+ private def providerClassReason(hadoopConf: Configuration, bucket: String): Option[String] = {
+ effectiveOptionValue(hadoopConf, bucket, AwsCredentialsProviderKey).flatMap { value =>
+ val names = parseProviderClassNames(value)
+ unsupportedProviderNameReason(bucket, AwsCredentialsProviderKey, names)
+ .orElse {
+ if (names.length > 1 && names.exists(AnonymousCredentialProviderClasses.contains)) {
+ Some(mixedAnonymousProviderReason(bucket, AwsCredentialsProviderKey))
+ } else {
+ None
+ }
+ }
+ .orElse {
+ if (names.contains(HadoopAssumedRoleProviderClass)) {
+ assumedRoleProviderClassReason(hadoopConf, bucket)
+ } else {
+ None
+ }
+ }
+ }
+ }
+
+ /**
+ * Returns the first reason any bucket among `uris` names an unsupported (or invalidly combined)
+ * credential-provider class, or `None` when every bucket's provider configuration is one native
+ * can build. See [[providerClassReason]].
+ */
+ private[delta] def providerClassGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) declined else providerClassReason(hadoopConf, bucket)
+ }
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def shadowedCredentialAliasReason(bucket: String, alias: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($alias resolves through $HadoopCredentialProviderPathKey but is not present " +
+ "as a plain configuration value, so the native S3 client would have no credentials)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * Compares each [[s3aCredentialAliases]] alias's plain `Configuration#get` value against
+ * `Configuration#getPassword` (providers first, plain conf as fallback); a resolved value that
+ * differs means the alias is invisible or wrong to native's plain-conf extraction, so decline.
+ * Only called once the GLOBAL provider path is established as the sole path for `bucket`.
+ *
+ * Runs inside try/catch: `getPassword` performs real keystore I/O, and a corrupt or unreadable
+ * store must decline this bucket rather than abort planning for the whole session.
+ */
+ private def verifyGlobalProviderAliases(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ try {
+ s3aCredentialAliases(bucket).foldLeft(Option.empty[String]) { (declined, alias) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ val resolved =
+ Option(hadoopConf.getPassword(alias)).map(new String(_)).filter(_.nonEmpty)
+ resolved match {
+ case Some(value) if !Option(hadoopConf.get(alias)).contains(value) =>
+ Some(shadowedCredentialAliasReason(bucket, alias))
+ case _ => None
+ }
+ }
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableCredentialProviderReason(bucket, e))
+ }
+ }
+
+ /**
+ * The decline reason for `bucket` alone, or `None` when its credentials are safe to forward.
+ * `globalPathSet`/`s3aPathSet` are hoisted by the caller since they don't vary per bucket.
+ *
+ * Checked FIRST, regardless of provider-path config: [[plainLongFormCredentialReason]] is a
+ * zero-I/O, provider-unrelated check that is decisive on its own.
+ *
+ * Zero-I/O precheck: `None` immediately when none of the four provider-path keys are set -- a
+ * config-map lookup only, no keystore access.
+ *
+ * Arm A: the S3A-scoped provider-path keys point S3A's own resolution at a provider
+ * `Configuration#getPassword` does not consult, so any being set declines with no keystore
+ * read.
+ *
+ * Arm B: only the global path is set, which `getPassword` DOES consult; delegates to
+ * [[verifyGlobalProviderAliases]].
+ */
+ private def bucketCredentialAliasReason(
+ hadoopConf: Configuration,
+ bucket: String,
+ globalPathSet: Boolean,
+ s3aPathSet: Boolean): Option[String] = {
+ plainLongFormCredentialReason(hadoopConf, bucket).orElse {
+ val bucketPathKey = s3aBucketProviderPathKey(bucket)
+ val bucketLongPathKey = s3aBucketLongProviderPathKey(bucket)
+ val bucketPathSet = nonEmptyConf(hadoopConf, bucketPathKey)
+ val bucketLongPathSet = nonEmptyConf(hadoopConf, bucketLongPathKey)
+ if (!globalPathSet && !s3aPathSet && !bucketPathSet && !bucketLongPathSet) {
+ None
+ } else if (s3aPathSet || bucketPathSet || bucketLongPathSet) {
+ val offendingKey =
+ if (s3aPathSet) S3aCredentialProviderPathKey
+ else if (bucketPathSet) bucketPathKey
+ else bucketLongPathKey
+ Some(s3aScopedProviderPathReason(bucket, offendingKey))
+ } else {
+ verifyGlobalProviderAliases(hadoopConf, bucket)
+ }
+ }
+ }
+
+ /**
+ * Returns the first reason a native S3 scan cannot faithfully forward this table's Hadoop
+ * credentials, or `None` when claimable. `uris` is the scan's data-file and DV URIs; only
+ * `s3`/`s3a` authorities matter here (ABFS/WASB mooted by the userinfo gate, GCS out of scope).
+ * `Configuration#getPassword` checks every provider FIRST, falling back to plain conf only when
+ * unset, so a keystore-only or shadowing alias is invisible/wrong to native's plain-conf
+ * extraction; when in doubt, decline. Never interpolates a resolved value, only key names.
+ */
+ private[delta] def credentialAliasReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ if (buckets.isEmpty) {
+ return None
Review Comment:
Extended the gate to the connector's legacy prefix: google.cloud.auth.* keys now decline the same way fs.gs.auth.* does, still scoped to gs paths and naming keys only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5443012067
Thanks for taking a look, and thanks Chao for the corrections, no arguments from me there. This did prompt a few improvements though.
The default feature choice came out of earlier review discussion, but you're right it belonged in the description. It's there now along with the measured cost, about 82 KB on a roughly 380 MB debug dylib, and it's unreachable without the contrib jar plus the config.
The DV decoder concern was fair, so there's now a malformed-input matrix in the native tests: truncation at every byte offset, CRC and magic corruption, length and cardinality lies, and bit-flip sweeps. Everything errors cleanly, no panics.
On version skew, Chao was right that linkage errors used to propagate. The claim hook now contains LinkageError as a logged decline, so a mismatched contrib jar falls back to Spark instead of breaking planning. There's a test that fakes a NoSuchMethodError contrib to pin that.
Also cleaned up the description itself, a few garbled words and stale pre-convergence module names, which is probably what made the benchmark section look cut off. CI numbers are in there too: the contrib jobs are path gated and the suite runs 4 to 5 minutes per Spark profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3871815190
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,548 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. Strip the parquet.field.id metadata
+ // createPhysicalSchema also stamps: files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations.
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
Review Comment:
Took your suggestion to fix the shared matcher instead of gating. Native case-insensitive matching now mirrors Java's equalsIgnoreCase (per char, simple mappings, including the U+0130 special case where simple and full mappings diverge), applied across the read path. The schema gate is gone since it couldn't see footer-side names anyway. Your Kelvin repro reproduced the (2,NULL) here and now reads correctly natively, as does the earlier accented pair.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3881827167
##########
contrib/delta-spark/dev/run-delta-regression.sh:
##########
@@ -0,0 +1,169 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Run Delta Lake's own Spark test suites against a Comet build with the
+# native Delta scan enabled. Clones delta at $DELTA_VERSION into $WORKDIR,
+# injects Comet into the test SparkSession (DeltaSQLCommandTest) and the
+# test classpath (unmanagedJars), then runs the given testOnly selectors.
+#
+# Usage:
+# COMET_JARS=/path/comet-spark.jar,/path/comet-contrib-delta.jar,/path/flatbuffers.jar \
+# ./run-delta-regression.sh <workdir> 'org.apache.spark.sql.delta.DeletionVectorsSuite' [...]
+#
+# Env:
+# DELTA_VERSION delta tag to test against (default 3.3.2)
+# COMET_JARS comma-separated jars added to the test classpath (required)
+# JAVA_HOME JDK for sbt (17 recommended)
+set -euo pipefail
+
+DELTA_VERSION="${DELTA_VERSION:-3.3.2}"
+WORKDIR="${1:?usage: run-delta-regression.sh <workdir> <suite> [...suites]}"
+shift
+[ $# -ge 1 ] || { echo "no suites given" >&2; exit 2; }
+: "${COMET_JARS:?COMET_JARS must list the comet jars}"
+
+# Resolve to an absolute path before the cd below: the log path is built from
+# $WORKDIR after we're already inside the Delta checkout, so a relative
+# argument would otherwise be re-anchored under $DELTA_DIR.
+mkdir -p "$WORKDIR"
+WORKDIR=$(cd "$WORKDIR" && pwd)
+
+IFS=',' read -ra _jars <<< "$COMET_JARS"
+for j in "${_jars[@]}"; do
+ [ -f "$j" ] || { echo "COMET_JARS entry not found: $j" >&2; exit 2; }
Review Comment:
Fixed, jar paths are canonicalized during validation before the cd, same treatment as the workdir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3905341388
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1627 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Zero-I/O in the common case, like the encryption gate above: native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The three keys where Hadoop's OWN resolver (`S3AUtils#lookupPassword`) is long-bucket-aware,
+ * verified via `javap` against `hadoop-aws` 3.3.4: `S3AUtils#lookupPassword(bucket, conf,
+ * baseKey)` builds `longBucketKey = "fs.s3a.bucket." + bucket + "." + baseKey` (note: the FULL
+ * `baseKey`, already `fs.s3a`-prefixed, appended after the bucket segment) and reads it via
+ * `getPassword` BEFORE the short-bucket key, keeping the long value whenever `getPassword`
+ * returns non-empty for it and only falling through to short-then-global otherwise. Kept as a
+ * named subset purely for documentation/discovery purposes (see
+ * [[hadoopLookupPasswordEffective]]'s doc); every key in [[AllS3ConfigKeys]] below is resolved
+ * through the SAME long-then-short-then-global function regardless of membership here -- see
+ * that function's doc for why extending long-bucket-awareness to keys where it does not
+ * strictly apply is safe.
+ */
+ private val CredentialValueKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ /**
+ * Every `fs.s3a.*` base key that governs whether a claimed native scan actually behaves like
+ * Hadoop's own reader would, checked by [[s3ConfigDivergenceReason]] via
+ * [[hadoopLookupPasswordEffective]] (see that function's doc for the resolution order and the
+ * deliberate over-decline rationale for the non-credential entries below). Two groups, kept
+ * merged into one list and one resolution path -- NOT two lists with two different resolution
+ * mechanisms, which is what let `fs.s3a.encryption.algorithm` silently fall through the
+ * short-then-global-only path while Hadoop's real resolver for it was long-bucket-aware (the
+ * SSE-C long-bucket-alias gap): the ORIGINAL [[CredentialValueKeys]] three, plus every OTHER
+ * per-bucket `fs.s3a.*` base key native's S3 client's `get_config` (s3.rs) resolves, verified
+ * directly against its call sites: `extract_s3_config_options` (endpoint.region,
+ * path.style.access, endpoint, requester.pays.enabled), `lookup_provider_class` (the
+ * Comet-specific credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name).
+ *
+ * SYNC NOTE: this list's non-credential entries must stay a superset of native's
+ * `NATIVE_S3A_CONFIG_PROPERTIES` constant (`native/core/src/parquet/objectstore/s3.rs`,
+ * property suffixes without the `fs.s3a.` prefix) -- `DeltaScanContribSuite`'s
+ * discovery-harness test asserts this mechanically. Literal strings, not the
+ * [[AwsCredentialsProviderKey]] / [[AssumedRoleCredentialsProviderKey]] vals declared below,
+ * purely to avoid a forward reference inside this `object` body; kept textually identical to
+ * those two constants.
+ */
+ private[delta] val AllS3ConfigKeys: Seq[String] = Seq(
+ "fs.s3a.access.key",
+ "fs.s3a.secret.key",
+ "fs.s3a.session.token",
+ "fs.s3a.aws.credentials.provider",
+ "fs.s3a.assumed.role.arn",
+ "fs.s3a.assumed.role.session.name",
+ "fs.s3a.assumed.role.credentials.provider",
+ "fs.s3a.endpoint",
+ "fs.s3a.endpoint.region",
+ "fs.s3a.path.style.access",
+ "fs.s3a.requester.pays.enabled",
+ "fs.s3a.comet.credential.provider.class")
+
+ /**
+ * The short-bucket-then-global value resolved for `baseKey` under `bucket` from `hadoopConf`,
+ * skipping an empty value at either alias exactly like [[plainValue]]. NOT used by
+ * [[s3ConfigDivergenceReason]]/[[s3KeyDivergenceReason]] any more -- every key checked there
+ * resolves through [[hadoopLookupPasswordEffective]] uniformly (long-then-short-then-global),
+ * which is a strict superset of what this function reads. This function's one remaining caller
+ * is [[shortThenGlobalOrReason]], which reads provider-CLASS strings (from the ORIGINAL,
+ * unpropagated conf) for name-support validation in [[providerClassReason]]/
+ * [[assumedRoleProviderClassReason]] -- by the time those run, [[s3ConfigDivergenceReason]] has
+ * already proven Hadoop's and native's effective values agree for the same key, so whichever of
+ * the two (equal) values this narrower read returns does not affect correctness there. NEVER
+ * used to compute native's own effective value -- see [[nativeShortThenGlobal]] for that.
+ */
+ private def shortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket` from the ORIGINAL, unpropagated `hadoopConf` -- `NativeConfig
+ * .extractObjectStoreOptions` forwards `Configuration#get`'s substituted value for every
+ * `fs.s3a.*` entry with no bucket-option propagation step of its own, so the original conf is
+ * the right input here. Unlike [[shortThenGlobal]]/[[plainValue]], this mirrors `get_config`
+ * faithfully: PRESENCE of the short-bucket key alone -- never its emptiness -- decides whether
+ * native falls back to the global key (`get_config` is a plain `HashMap::get`, which returns
+ * `Some` for a key explicitly set to `""`), so an explicitly empty or whitespace-only
+ * short-bucket value resolves to `Some("")` here and never falls through to global -- the
+ * OPPOSITE of Hadoop's own `getPassword`/`lookupPassword` semantics (see
+ * [[hadoopLookupPasswordEffective]]), which treat empty as absent and keep trying the next
+ * alias. The ONLY function used to compute native's effective value in
+ * [[s3KeyDivergenceReason]].
+ *
+ * Deliberately does NOT apply `get_config_trimmed`'s `.trim()` here: [[s3KeyDivergenceReason]]
+ * trims both this value and Hadoop's effective value together, symmetrically, at the point they
+ * are compared, rather than one-sidedly here -- trimming only the native side would flag a
+ * spurious divergence for a value neither side's whitespace actually changes the behavior of
+ * once each side's own downstream parsing normalizes it (e.g. Hadoop's own multi-line
+ * `fs.s3a.aws.credentials.provider` default, which both Hadoop and native additionally trim per
+ * comma-separated entry after splitting), while a one-sided trim would make an
+ * otherwise-identical default value look diverged for every bucket, never claiming natively at
+ * all.
+ */
+ private def nativeShortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ Option(hadoopConf.get(shortKey)).orElse(Option(hadoopConf.get(baseKey)))
+ }
+
+ /**
+ * Faithful in-memory replica of `S3AUtils#propagateBucketOptions` (`hadoop-aws`), which
+ * `S3AFileSystem#initialize` calls FIRST, before any option or credential is read:
+ * `Configuration conf = propagateBucketOptions(originalConf, bucket); ...; setConf(conf);` --
+ * every subsequent `conf.get`/`getPassword` call in that filesystem instance, including
+ * `${...}` variable substitution, resolves against this propagated view, not the original conf.
+ * `hadoop-aws` is not on this module's runtime classpath (see the string-literal-keys note
+ * above), so `S3AUtils#propagateBucketOptions` cannot be called directly; this reproduces its
+ * logic verbatim using only `hadoop-common`'s `Configuration`:
+ * {{{
+ * public static Configuration propagateBucketOptions(Configuration source, String bucket) {
+ * final String bucketPrefix = FS_S3A_BUCKET_PREFIX + bucket + '.';
+ * final Configuration dest = new Configuration(source);
+ * for (Map.Entry<String, String> entry : source) {
+ * final String key = entry.getKey();
+ * final String value = entry.getValue(); // the (unexpanded) value
+ * if (!key.startsWith(bucketPrefix) || bucketPrefix.equals(key)) continue;
+ * final String stripped = key.substring(bucketPrefix.length());
+ * if (stripped.startsWith("bucket.") || "impl".equals(stripped)) {
+ * // ignored
+ * } else {
+ * final String generic = FS_S3A_PREFIX + stripped;
+ * dest.set(generic, value, ...); // overwrites any existing global value
+ * }
+ * }
+ * return dest;
+ * }
+ * }}}
+ * Note the LONG bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) folds to an unread
+ * `fs.s3a.fs.s3a.<key>` key here too, exactly like the real method -- `stripped` already starts
+ * with `fs.s3a.` in that case, so prepending `fs.s3a.` again produces a key nothing ever reads.
+ */
+ private def propagateBucketOptions(hadoopConf: Configuration, bucket: String): Configuration = {
+ val bucketPrefix = s"fs.s3a.bucket.$bucket."
+ val dest = new Configuration(hadoopConf)
+ hadoopConf.iterator().asScala.foreach { entry =>
+ val key = entry.getKey
+ if (key.startsWith(bucketPrefix) && key != bucketPrefix) {
+ val stripped = key.substring(bucketPrefix.length)
+ if (!stripped.startsWith("bucket.") && stripped != "impl") {
+ dest.set(s"fs.s3a.$stripped", entry.getValue)
+ }
+ }
+ }
+ dest
+ }
+
+ /**
+ * Canonical and deprecated Hadoop S3A encryption-algorithm config keys, verified via `javap`
+ * against `hadoop-aws` 3.3.4's `org.apache.hadoop.fs.s3a.Constants`: `S3_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.encryption.algorithm"` (canonical) and `SERVER_SIDE_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.server-side-encryption-algorithm"` (DEPRECATED -- note the hyphen before "algorithm",
+ * unlike the corresponding `*.key` constants below, which both use a `.key` suffix).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, same
+ * rationale as [[HadoopCredentialProviderPathKey]].
+ */
+ private val S3EncryptionAlgorithmKey = "fs.s3a.encryption.algorithm"
+ private val DeprecatedS3EncryptionAlgorithmKey = "fs.s3a.server-side-encryption-algorithm"
+
+ /**
+ * The exact strings `S3AEncryptionMethods#getMethod` accepts, verified via `javap`/CFR against
+ * `hadoop-aws` 3.3.4's `S3AEncryptionMethods` enum: `NONE("")`, `SSE_S3("AES256", serverSide =
+ * true, requiresSecret = false)`, `SSE_KMS("SSE-KMS", serverSide = true, requiresSecret =
+ * false)`, `SSE_C("SSE-C", serverSide = true, requiresSecret = true)`, `CSE_KMS("CSE-KMS",
+ * serverSide = false, requiresSecret = true)`, `CSE_CUSTOM("CSE-CUSTOM", serverSide = false,
+ * requiresSecret = true)`. `getMethod` parses case-insensitively
+ * (`values().find(_.getMethod.equalsIgnoreCase(algorithm))`), matched below the same way.
+ *
+ * ALLOWLIST, not a blocklist (replaces the former SSE-C-only blocklist): only the algorithms S3
+ * decrypts transparently on GET/HEAD given read permission alone, with NO extra request header
+ * and NO client-side step, are safe for a native scan that forwards none of Hadoop's
+ * `fs.s3a.encryption.*`/`fs.s3a.server-side-encryption*` options --
+ * - `AES256` (SSE_S3, `serverSide = true`): plain server-side encryption, transparent on GET.
+ * - `SSE-KMS` (SSE_KMS, `serverSide = true`): server-side, KMS-managed key, transparent on
+ * GET given KMS decrypt permission (no header).
+ * - `DSSE-KMS`: NOT present in this enum on `hadoop-aws` 3.3.4 (confirmed by the six values
+ * listed above) -- `S3AEncryptionMethods.getMethod("DSSE-KMS")` throws
+ * `IOException("Unknown encryption algorithm DSSE-KMS")` on this version, so
+ * `S3AUtils#buildEncryptionSecrets` (and therefore Hadoop's own reader) already fails
+ * before ever reading such a table under 3.3.4, meaning this string can never actually be
+ * the resolved value on the declared target version -- admitting it here is inert there.
+ * Included anyway, forward-compatible, for a newer `hadoop-aws` on the runtime classpath (a
+ * later Hadoop release; this module has no compile-time `hadoop-aws` dependency, see the
+ * string-literal-keys note above) where DSSE-KMS is a real, dual-layer, server-side
+ * algorithm decrypted transparently on GET the same way SSE-KMS is. Every other value
+ * declines: `SSE-C` (SSE_C is `serverSide = true` in Hadoop's own enum, but `requiresSecret
+ * \= true` -- S3 rejects a GET/HEAD for an SSE-C object outright (400 Bad Request) unless
+ * the customer key is resent as a request header on every call, so a native scan that never
+ * learns the key cannot succeed at all, where Hadoop's own reader -- whose request factory
+ * attaches the key -- would), `CSE-KMS`/`CSE-CUSTOM` (`serverSide = false`: client-side
+ * encryption decrypts object bytes locally in the SDK layer, which the native Parquet
+ * reader has no equivalent of -- it would read raw ciphertext), and any future/unknown
+ * value (a value `S3AEncryptionMethods.getMethod` itself would reject is certainly not one
+ * of the three confirmed-transparent algorithms above; declining is the only safe default
+ * for anything this gate cannot positively confirm).
+ */
+ private val AllowedEncryptionAlgorithms: Set[String] = Set("AES256", "SSE-KMS", "DSSE-KMS")
+
+ /**
+ * `bucket`'s effective encryption-algorithm key and value under `hadoopConf`, or `None` when
+ * neither the canonical nor deprecated key is set anywhere consulted. Mirrors
+ * `S3AUtils#buildEncryptionSecrets`'s real resolution order, verified via `javap`/CFR
+ * decompilation of `hadoop-aws` 3.3.4's `S3AUtils.class`:
+ * {{{
+ * String algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupBucketSecret(bucket, conf, "fs.s3a.server-side-encryption-algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.encryption.algorithm");
+ * if (algorithm == null)
+ * algorithm = lookupPassword(null, conf, "fs.s3a.server-side-encryption-algorithm");
+ * }}}
+ * i.e. bucket-tier (canonical, then deprecated), THEN global-tier (canonical, then deprecated)
+ * -- the two tiers are never interleaved key-by-key, so this must stay two explicit bucket-tier
+ * lookups followed by two explicit global-tier lookups, not a single
+ * [[hadoopLookupPasswordEffective]] call per key (which would let an unset canonical bucket key
+ * fall through straight to the canonical GLOBAL value ahead of a SET deprecated bucket key, the
+ * wrong answer).
+ *
+ * THE FIX for the SSE-C long-bucket-alias gap is entirely inside the bucket tier:
+ * `lookupBucketSecret` itself is long-then-short, decompiled from `hadoop-aws` 3.3.4's
+ * `S3AUtils.class`:
+ * {{{
+ * // longBucketKey = fs.s3a.bucket.B.fs.s3a.<key>
+ * String longBucketKey = String.format(BUCKET_PATTERN, bucket, baseKey);
+ * String initialVal = getPassword(conf, longBucketKey, null, null);
+ * // shortBucketKey = fs.s3a.bucket.B.<key>
+ * String shortBucketKey = String.format(BUCKET_PATTERN, bucket, subkey);
+ * // keeps initialVal (the LONG value) if non-empty
+ * return getPassword(conf, shortBucketKey, initialVal, null);
+ * }}}
+ * i.e. the SAME long-bucket-key construction and long-wins-if-nonempty semantics as
+ * `S3AUtils#lookupPassword` (see [[CredentialValueKeys]]/[[hadoopLookupPasswordEffective]]) --
+ * the encryption algorithm is NOT one of the keys that flows through
+ * `S3AUtils#propagateBucketOptions` (which folds an unrelated per-bucket LONG form into an
+ * unread key). An earlier version of this function modeled the bucket tier as SHORT-only,
+ * documented as "the LONG bucket form is genuinely never consulted for this key" -- that
+ * documentation was wrong (this decompilation supersedes it): a bucket configured only via
+ * `fs.s3a.bucket.B.fs.s3a.encryption.algorithm=SSE-C` bypassed the SSE-C gate entirely, because
+ * Hadoop's own reader DOES read that long form (and picks SSE-C), while this function reported
+ * `None` (nothing set) and the allowlist check below never even ran.
+ *
+ * The canonical-vs-deprecated distinction below is frequently moot in practice: `hadoop-aws`'s
+ * `S3AFileSystem.addDeprecatedKeys()` statically registers `fs.s3a.server-side-encryption-*` as
+ * `Configuration`-level deprecated aliases of `fs.s3a.encryption.*` (verified via `javap`), a
+ * registration that lives in a static field on Hadoop's `Configuration` class -- process-wide
+ * once `S3AFileSystem`'s class has loaded anywhere in the JVM, which a real scan has always
+ * already done by the time this gate runs, since reading the S3 table at all requires loading
+ * that class. Once active, `Configuration#get` resolves either literal key to the identical
+ * value transparently, making the two-key cascade below redundant (but harmless) for that case;
+ * it remains the operative path only when nothing else in the process has loaded
+ * `S3AFileSystem` yet.
+ *
+ * ALSO walks the Hadoop-credential-provider (JCEKS) path via [[resolveViaCredentialAliases]]
+ * for each of the four lookups below, matching `lookupBucketSecret`/`lookupPassword`'s real
+ * per-alias `getPassword` calls (quoted above) exactly: both are `getPassword`, not plain
+ * `Configuration#get`, so a bucket storing the algorithm name ONLY in a JCEKS keystore is
+ * exactly as real a Hadoop deployment shape for this key as it is for the credential keys
+ * [[hadoopLookupPasswordEffective]] already covers -- there is nothing algorithm-specific that
+ * makes JCEKS storage implausible here, so an earlier version of this function skipping it
+ * (documented at the time as "the algorithm NAME is not credential-sensitive data, so storing
+ * it in a keystore is not a realistic Hadoop deployment pattern") was an unjustified, narrower
+ * read than Hadoop's own resolver actually performs, under-declining a bucket whose algorithm
+ * is keystore-only. [[resolveViaCredentialAliases]]'s Arm B/C split still means this is zero
+ * extra I/O for the common case: keystore I/O only happens when a Hadoop credential-provider
+ * path is actually configured for the bucket, contained in that function's own try/catch.
+ * `bucketTier`/`globalTier` return `Left` (propagated straight through by [[orElseTier]]) when
+ * [[resolveViaCredentialAliases]] cannot safely verify a tier at all (an S3A-scoped provider
+ * path, or a corrupt/unreadable global keystore) -- correctly short-circuiting the whole
+ * cascade with a decline rather than silently falling through to a later tier that might look
+ * unset only because the true value was unverifiable.
+ */
+ private def effectiveEncryptionAlgorithm(
+ hadoopConf: Configuration,
+ bucket: String): Either[String, Option[(String, String)]] = {
+ def bucketTier(baseKey: String): Either[String, Option[(String, String)]] = {
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(longKey, shortKey))
+ .map(_.map(baseKey -> _))
+ }
+ def globalTier(baseKey: String): Either[String, Option[(String, String)]] =
+ resolveViaCredentialAliases(hadoopConf, bucket, Seq(baseKey)).map(_.map(baseKey -> _))
+
+ // Short-circuits on Left (unverifiable tier) or Right(Some(_)) (resolved); only Right(None)
+ // (tier definitively unset) falls through to `next`, mirroring buildEncryptionSecrets's
+ // sequential `if (algorithm == null) algorithm = ...` cascade exactly.
+ def orElseTier(
+ current: Either[String, Option[(String, String)]],
+ next: => Either[String, Option[(String, String)]])
+ : Either[String, Option[(String, String)]] =
+ current match {
+ case Left(reason) => Left(reason)
+ case Right(Some(value)) => Right(Some(value))
+ case Right(None) => next
+ }
+
+ orElseTier(
+ bucketTier(S3EncryptionAlgorithmKey),
+ orElseTier(
+ bucketTier(DeprecatedS3EncryptionAlgorithmKey),
+ orElseTier(
+ globalTier(S3EncryptionAlgorithmKey),
+ globalTier(DeprecatedS3EncryptionAlgorithmKey))))
+ }
+
+ private def unsupportedEncryptionAlgorithmDeclineReason(
+ bucket: String,
+ algorithmKey: String,
+ algorithm: String): String =
+ s"Native Delta scan does not support $algorithmKey=$algorithm for $bucket " +
+ "(the native S3 client only supports unencrypted objects and S3's transparent " +
+ "server-side algorithms -- AES256/SSE-S3, SSE-KMS, and DSSE-KMS decrypt on GET/HEAD given " +
+ "read permission alone, with no extra request header; SSE-C additionally requires the " +
+ "customer-provided key resent as a header on every GET/HEAD request, which the native S3 " +
+ "client's extract_s3_config_options never forwards, and CSE-KMS/CSE-CUSTOM decrypt object " +
+ "bytes client-side, a layer the native Parquet reader does not have -- any of these would " +
+ "fail outright or silently read ciphertext where Hadoop's own reader succeeds)"
+
+ /**
+ * First reason any bucket among `uris` is configured for an encryption algorithm the native S3
+ * client cannot safely read, or `None` when claimable. Allowlist-based (see
+ * [[AllowedEncryptionAlgorithms]]): only `AES256`/`SSE-KMS`/`DSSE-KMS` (and unset/empty) pass;
+ * every other resolved value -- `SSE-C`, `CSE-KMS`, `CSE-CUSTOM`, or any unrecognized future
+ * algorithm string -- declines. Deliberately NOT a blocklist keyed on `SSE-C` alone: an
+ * allowlist is safe by construction against a Hadoop release adding a new encryption method
+ * this gate has never heard of, where a blocklist would silently admit it. Never interpolates a
+ * resolved key value, only key names, the bucket, and the (non-secret) algorithm name. Declines
+ * on a `Left` from [[effectiveEncryptionAlgorithm]] too (an unverifiable credential-provider
+ * arm, e.g. an S3A-scoped provider path or a corrupt/unreadable global keystore) -- the
+ * algorithm cannot be ruled safe when it cannot be read at all.
+ */
+ private[delta] def unsupportedEncryptionAlgorithmReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ effectiveEncryptionAlgorithm(hadoopConf, bucket) match {
+ case Left(reason) => Some(reason)
+ case Right(None) => None
+ case Right(Some((key, value))) =>
+ if (!AllowedEncryptionAlgorithms.exists(_.equalsIgnoreCase(value))) {
+ Some(unsupportedEncryptionAlgorithmDeclineReason(bucket, key, value))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Canonical Hadoop S3A HTTP-proxy host config key, verified via CFR decompilation of
+ * `hadoop-aws` 3.3.4's `S3AUtils.class` (`initProxySupport`):
+ * {{{
+ * String proxyHost = conf.getTrimmed("fs.s3a.proxy.host", "");
+ * int proxyPort = conf.getInt("fs.s3a.proxy.port", -1);
+ * if (!proxyHost.isEmpty()) {
+ * ...
+ * String proxyUsername =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.username", null, null);
+ * String proxyPassword =
+ * S3AUtils.lookupPassword(bucket, conf, "fs.s3a.proxy.password", null, null);
+ * ...
+ * }
+ * }}}
+ * `fs.s3a.proxy.host`/`fs.s3a.proxy.port` resolve via a PLAIN, non-bucket-scoped, non-JCEKS
+ * `Configuration#getTrimmed`/`getInt` call -- NOT `lookupPassword` -- against whatever conf
+ * `S3AFileSystem#initialize` already ran through `propagateBucketOptions` before
+ * `createAwsConf`/`initProxySupport` ever runs; only the SIBLING `fs.s3a.proxy.username`/
+ * `fs.s3a.proxy.password` keys go through `lookupPassword` (bucket long/short/global,
+ * JCEKS-aware). So the host is bucket-aware only via `propagateBucketOptions`'s short-bucket-
+ * form folding, never the long-bucket form, and never a credential-provider read -- the SAME
+ * shape as `endpoint`/`path.style.access` (the "propagated option" family, see
+ * [[AllS3ConfigKeys]]'s doc), not the credential family.
+ *
+ * Resolved here through [[hadoopLookupPasswordEffective]] anyway (long-then-short-then-global,
+ * credential-provider-aware) for the identical reason [[AllS3ConfigKeys]]'s non-credential
+ * entries are: checking MORE than Hadoop's real, narrower resolution can only ever find a value
+ * Hadoop itself ignores, which can only turn an already-necessary decline into one found
+ * slightly more eagerly -- never a missed one. A missed decline is exactly the failure mode
+ * this gate exists to close (native has NO HTTP-proxy support of any kind -- no
+ * `fs.s3a.proxy.*` key is read anywhere in `s3.rs`), so over-approximating is the conservative
+ * direction here, unlike a value-EQUALITY comparator (e.g. [[s3KeyDivergenceReason]]) where
+ * over-approximating could manufacture a false divergence between two sides that would
+ * otherwise agree.
+ */
+ private val S3ProxyHostKey = "fs.s3a.proxy.host"
+
+ private def unsupportedProxyReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key configured for $bucket (the native S3 client has " +
+ "no HTTP proxy support at all -- no fs.s3a.proxy.* key is read anywhere in its object " +
+ "store layer -- so a claimed scan would connect to S3 directly instead of routing through " +
+ "the configured proxy, either bypassing an egress/network-segmentation policy or simply " +
+ "failing to reach the endpoint)"
+
+ /**
+ * First reason any bucket among `uris` has an HTTP proxy configured via [[S3ProxyHostKey]], or
+ * `None` when claimable. Ordered alongside [[unsupportedEncryptionAlgorithmReason]] among the
+ * other conf-only gates that are zero-I/O in the common case, ahead of
+ * [[s3ConfigDivergenceReason]]: [[hadoopLookupPasswordEffective]] only performs real keystore
+ * I/O, inside its own try/catch, when a Hadoop credential-provider path is actually configured
+ * for this bucket (see that function's doc for the Arm A/B/C dispatch). Never interpolates a
+ * resolved value: the proxy HOST is not secret, but naming it here would be a strange place to
+ * first surface it, and proxy CREDENTIALS (`fs.s3a.proxy.username`/`fs.s3a.proxy.password`, not
+ * read by this gate at all -- the whole point of gating on the host is that a non-empty host
+ * declines before any proxy credential would ever need to be forwarded) must never appear in a
+ * decline reason regardless.
+ */
+ private[delta] def proxyGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI],
+ propagatedConfCache: MutableMap[String, Configuration] = MutableMap.empty)
+ : Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ try {
+ val propagatedConf =
+ propagatedConfCache.getOrElseUpdate(
+ bucket,
+ propagateBucketOptions(hadoopConf, bucket))
+ hadoopLookupPasswordEffective(propagatedConf, bucket, S3ProxyHostKey) match {
+ case Left(reason) => Some(reason)
+ case Right(Some(_)) => Some(unsupportedProxyReason(bucket, S3ProxyHostKey))
+ case Right(None) => None
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableValueReason(bucket, S3ProxyHostKey, e))
+ }
+ }
+ }
+ }
+
+ /**
+ * `baseKey`'s long, short, then global per-bucket aliases, in Hadoop's own resolution order.
+ */
+ private def longThenShortThenGlobalAliases(bucket: String, baseKey: String): Seq[String] = {
+ val suffix = baseKey.stripPrefix("fs.s3a.")
+ Seq(s"fs.s3a.bucket.$bucket.fs.s3a.$suffix", s"fs.s3a.bucket.$bucket.$suffix", baseKey)
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * The three-way Hadoop credential-provider-path precheck shared by every `getPassword`-based
+ * resolution below, factored out of what used to be [[hadoopLookupPasswordEffective]]'s own
+ * body so [[effectiveEncryptionAlgorithm]] can reuse it for its own bucket-tier-only and
+ * global-tier-only lookups (see that function's doc for why the algorithm needs its resolution
+ * split into separate tiers rather than one combined long-then-short-then-global list). This
+ * precheck is a property of the BUCKET alone, independent of which base key or alias list a
+ * caller goes on to resolve: Arm A -- an S3A-scoped or bucket-scoped Hadoop credential-provider
+ * path configures a provider `Configuration#getPassword` does not consult
+ * (`S3AUtils#lookupPassword`/`#lookupBucketSecret` build their own S3A-only provider list per
+ * bucket, which this plain-conf-reading gate cannot mirror) -- returns
+ * [[UnverifiableProvider]], NO keystore I/O. Arm B -- only the GLOBAL Hadoop provider path is
+ * set (which `getPassword` DOES consult) -- returns [[GlobalProviderOnly]]; the caller must
+ * wrap its own `getPassword` calls in try/catch, since real keystore I/O only happens once this
+ * arm is reached. Arm C -- no provider path is set anywhere -- returns [[NoProvider]], zero-I/O
+ * plain conf only.
+ */
+ private sealed trait CredentialProviderArm
+ private case class UnverifiableProvider(offendingKey: String) extends CredentialProviderArm
+ private case object GlobalProviderOnly extends CredentialProviderArm
+ private case object NoProvider extends CredentialProviderArm
+
+ private def credentialProviderArm(
+ hadoopConf: Configuration,
+ bucket: String): CredentialProviderArm = {
+ val bucketPathKey = s3aBucketProviderPathKey(bucket)
+ val bucketLongPathKey = s3aBucketLongProviderPathKey(bucket)
+ val s3aPathSet = nonEmptyConf(hadoopConf, S3aCredentialProviderPathKey)
+ val bucketPathSet = nonEmptyConf(hadoopConf, bucketPathKey)
+ val bucketLongPathSet = nonEmptyConf(hadoopConf, bucketLongPathKey)
+ if (s3aPathSet || bucketPathSet || bucketLongPathSet) {
+ val offendingKey =
+ if (s3aPathSet) S3aCredentialProviderPathKey
+ else if (bucketPathSet) bucketPathKey
+ else bucketLongPathKey
+ UnverifiableProvider(offendingKey)
+ } else if (nonEmptyConf(hadoopConf, HadoopCredentialProviderPathKey)) {
+ GlobalProviderOnly
+ } else {
+ NoProvider
+ }
+ }
+
+ /**
+ * `aliases`' effective value under `hadoopConf`/`bucket`, tried in the given order and keeping
+ * the first non-empty result -- or `Left(reason)` when the value cannot be safely verified.
+ * Dispatches on [[credentialProviderArm]]: [[UnverifiableProvider]] declines outright (zero
+ * I/O); [[GlobalProviderOnly]] resolves every alias via `Configuration#getPassword` (real
+ * keystore I/O, contained in try/catch: a corrupt/unreadable store must decline this bucket,
+ * not abort planning for the whole session); [[NoProvider]] resolves every alias via plain
+ * [[plainValue]] reads (zero I/O). Generalizes what used to be
+ * [[hadoopLookupPasswordEffective]]'s entire body to an explicit, caller-supplied alias list --
+ * [[hadoopLookupPasswordEffective]] is now the long-then-short-then-global special case of
+ * this; [[effectiveEncryptionAlgorithm]] is the bucket-tier-only (two aliases) and
+ * global-tier-only (one alias) special cases, matching
+ * `S3AUtils#lookupBucketSecret`/`#lookupPassword`'s real per-tier `getPassword` calls exactly
+ * instead of [[plainValue]]'s JCEKS-blind read that under-declined a keystore-only algorithm.
+ */
+ private def resolveViaCredentialAliases(
+ hadoopConf: Configuration,
+ bucket: String,
+ aliases: Seq[String]): Either[String, Option[String]] =
+ credentialProviderArm(hadoopConf, bucket) match {
+ case UnverifiableProvider(offendingKey) =>
+ Left(s3aScopedProviderPathReason(bucket, offendingKey))
+ case GlobalProviderOnly =>
+ try {
+ Right(
+ aliases.iterator
+ .map(alias =>
+ Option(hadoopConf.getPassword(alias)).map(new String(_)).filter(_.nonEmpty))
+ .collectFirst { case Some(v) => v })
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Left(unverifiableCredentialProviderReason(bucket, e))
+ }
+ case NoProvider =>
+ Right(aliases.flatMap(plainValue(hadoopConf, _)).headOption)
+ }
+
+ /**
+ * `bucket`'s effective value for `baseKey` in Hadoop's own `S3AUtils#lookupPassword` resolution
+ * order -- long bucket alias, then short bucket alias, then global, each tried through a Hadoop
+ * credential provider before falling back to plain conf (see [[resolveViaCredentialAliases]]
+ * for the Arm A/B/C dispatch this delegates to) -- or `Left(reason)` when the value cannot be
+ * safely verified.
+ *
+ * USED FOR EVERY KEY IN [[AllS3ConfigKeys]], not just [[CredentialValueKeys]] -- see that val's
+ * doc for why the structural bug behind the SSE-C long-bucket-alias gap was exactly this split
+ * (some keys resolved long-bucket-aware, others short-only) and why resolving every key through
+ * this SAME long-then-short-then-global function is fail-safe even for the keys (`endpoint`,
+ * `path.style.access`, `requester.pays.enabled`, `comet.credential.provider.class`) whose real
+ * Hadoop resolver is `S3AUtils#propagateBucketOptions` + plain `Configuration#get`, which never
+ * consults the long bucket form or a credential provider at all: checking the long alias (or a
+ * credential-provider path) for one of THOSE keys can only ever find a value real Hadoop itself
+ * ignores. That can make this function return a value differing from Hadoop's TRUE effective
+ * value only in the direction of an EXTRA divergence (a long-form/keystore value set
+ * differently from short/global for a key Hadoop never reads that way) -- which
+ * [[s3KeyDivergenceReason]] turns into an extra DECLINE, never a wrongly-claimed scan, since a
+ * value Hadoop never reads cannot be the reason a wrongly-claimed native scan would misbehave.
+ * A key added to [[AllS3ConfigKeys]] in the future inherits this safety automatically; no
+ * per-key classification decision (and no repeat of the SSE-C bug) is needed.
+ *
+ * `hadoopConf` must be a [[propagateBucketOptions]] result (the caller,
+ * [[s3KeyDivergenceReason]], always passes one) so that `${...}` references embedded in any
+ * alias resolve exactly like `S3AFileSystem#initialize`'s real propagate-then-resolve order.
+ */
+ private def hadoopLookupPasswordEffective(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Either[String, Option[String]] =
+ resolveViaCredentialAliases(
+ hadoopConf,
+ bucket,
+ longThenShortThenGlobalAliases(bucket, baseKey))
+
+ private def effectiveValueDivergenceReason(bucket: String, key: String): String =
+ s"Native Delta scan cannot forward $key for $bucket (Hadoop's effective value for this key " +
+ "differs from what the native S3 client resolves, so its credentials or configuration " +
+ "would differ from Hadoop's)"
+
+ private def unverifiableValueReason(bucket: String, key: String, error: Throwable): String =
+ s"Native Delta scan cannot verify $key for $bucket (Configuration#get raised " +
+ s"${error.getClass.getName}), declining rather than risk forwarding a stale or " +
+ "diverging value"
+
+ /**
+ * `None` when `baseKey`'s Hadoop-effective and native-effective values under `bucket` agree, or
+ * a decline reason naming `baseKey` and `bucket` (never a value) when they diverge or either
+ * side cannot be safely computed.
+ *
+ * Hadoop's effective value is computed against [[propagateBucketOptions]]'s result, mirroring
+ * `S3AFileSystem#initialize`'s actual order (propagate bucket options into the conf FIRST, only
+ * THEN read/substitute options against it), via Hadoop's `lookupPassword`-order read
+ * ([[hadoopLookupPasswordEffective]]) for EVERY key in [[AllS3ConfigKeys]] -- see that
+ * function's doc for why using the same long-then-short-then-global resolution uniformly,
+ * rather than a short-then-global-only read for the keys Hadoop's real resolver does not make
+ * long-bucket-aware, is a deliberate, fail-safe over-approximation. Native's effective value is
+ * always `nativeShortThenGlobal(hadoopConf, ...)` on the ORIGINAL, unpropagated conf, matching
+ * `NativeConfig.extractObjectStoreOptions`'s actual forwarding semantics (no propagation step)
+ * AND native's `get_config` presence-based (not emptiness-based) short-vs-global fallback --
+ * see [[nativeShortThenGlobal]]. Both values are trimmed together, symmetrically, right before
+ * the equality check below (mirroring `get_config_trimmed`'s `.trim()`, which native applies
+ * regardless of which alias it read) rather than trimming [[nativeShortThenGlobal]]'s result on
+ * its own -- see [[nativeShortThenGlobal]]'s doc for why a one-sided trim there would flag a
+ * spurious divergence. Comparing against the propagated view (rather than the original conf, as
+ * an earlier version of this check did) matters because propagation can change what a `${...}`
+ * reference inside one bucket-scoped value resolves to: e.g.
+ * `fs.s3a.bucket.B.access.key=${fs.s3a.custom.ref}` with `fs.s3a.bucket.B.custom.ref=X` and
+ * global `fs.s3a.custom.ref=Y` propagates to `fs.s3a.custom.ref=X` (overwriting the global `Y`)
+ * before the access key's `${...}` reference is ever substituted, so Hadoop resolves `X` while
+ * a check against the unpropagated conf would (wrongly) also see `Y`, the same value native
+ * forwards -- masking a real divergence. Wrapped in try/catch: `Configuration#get` raises
+ * `IllegalStateException` once `${...}` substitution recurses past Hadoop's `MAX_SUBST` bound
+ * (e.g. a two-key mutual reference cycle); declining is safer than crashing planning or
+ * comparing a partially-substituted value.
+ */
+ private def s3KeyDivergenceReason(
+ hadoopConf: Configuration,
+ propagatedConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ try {
+ val hadoopEffective: Either[String, Option[String]] =
+ hadoopLookupPasswordEffective(propagatedConf, bucket, baseKey)
Review Comment:
Good catch, and it broke more than this one key: the blanket unification I landed earlier reads wider than the real consumer for plain options, and a comparator that reads wider can manufacture equality — which is the dangerous direction. The comparator now carries a per-key consumer tier: the credential trio keeps lookupPassword semantics (the SSE-C and JCEKS tests all still pass untouched), everything else resolves the way its actual consumer does — plain get on the propagated conf, with the ${} reference expanding under the propagated view. Your four-setting shape now declines naming the endpoint key, the companion without the per-bucket referent override still admits, and a lone long-form endpoint alias now admits too, since propagateBucketOptions folds it into the unread fs.s3a.fs.s3a.endpoint key. Adding a key without a tier no longer compiles, a harness test pins the password tier to exactly the trio, and each consumer assignment was checked against the 3.3.4 bytecode r
ather than the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5497653844
cc @andygrove @comphead @parthchandra @viirya @peterxcli as well, since this is a fairly large PR and would benefit from a few extra sets of eyes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4081067884
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so its rewrite (via
+ * [[CometScanWithPlanData]]) survives plan copies -- a transient field would be dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning (the CometIcebergNativeScanExec lesson).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node
+ // automatically gets a helper consistent with ITS runtimeFilters, avoiding the #3510 class of
+ // bug where a stored helper field desyncs from rewritten filters. Costs one extra file listing
+ // per executed instance; correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below, and AQE's UI
+ // plan-walk calls `.metrics` on every node MID-PLANNING, including while a DPP subquery is
+ // still an adaptive placeholder or a partition filter holds an unresolved ScalarSubquery (see
+ // `hasUnevaluableSubqueryFilter` below). That's safe ONLY because constructing `scanHelper` is a
+ // cheap case-class build with no file listing, and core's `CometScanExec.metrics` touches only
+ // `wrapped.driverMetrics` (populated by Spark's own planning) plus a static metric-node
+ // constructor -- neither file listing nor subquery resolution. If core's `metrics` ever touches
+ // either, forcing `scanHelper` here would resurrect the AQE mid-planning crashes this invariant
+ // prevents.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element, so checking runtimeFilters
+ // here avoids constructing/forcing the derived scanHelper just to read partitioning. The
+ // InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep in sync. The
+ // ScalarSubquery case is probed rather than treated as permanently unevaluable: Spark exposes no
+ // public finished/updated flag on ExecSubqueryExpression, but `eval()` doubles as one -- it only
+ // reads the cached `result` behind a `require(updated, ...)` guard, while the subquery is
+ // actually run by `updateResult()` (invoked separately during prepare/AQE), never by `eval()`.
+ // Once resolved, outputPartitioning below reports the real perPartitionData.length instead of
+ // staying at zero -- a fused native parent's buildNativeContext requires that count to match.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case s: ScalarSubquery => !isScalarSubqueryResolved(s)
+ case _ => false
+ })
+
+ // `eval()` never triggers the subquery's execution: on a resolved subquery it is a pure cached
+ // read of `result` (verified against bytecode: `Predef.require(updated(), ...)` then a plain
+ // field read), so this probe is safe to call repeatedly, including from AQE's mid-planning plan
+ // walks. Pre-resolution, the ONLY throw is `require`'s `IllegalArgumentException`; catch exactly
+ // that, since anything else escaping is a genuine bug we must not mask as unpartitioned.
+ private def isScalarSubqueryResolved(s: ScalarSubquery): Boolean =
+ try {
+ s.eval()
+ true
+ } catch {
+ case _: IllegalArgumentException => false
+ }
+
+ private def isAdaptivePlaceholder(p: SparkPlan): Boolean = p match {
+ case ReusedSubqueryExec(inner) => isAdaptivePlaceholder(inner)
+ case _: CometSubqueryAdaptiveBroadcastExec => true
+ case _: SubqueryAdaptiveBroadcastExec => true
+ case _ => false
+ }
+
+ override lazy val outputOrdering: Seq[SortOrder] = originalPlan.outputOrdering
+
+ override def dynamicPruningFilters: Seq[Expression] = runtimeFilters
+
+ override def withDynamicPruningFilters(filters: Seq[Expression]): SparkPlan = {
+ // A real copy: runtimeFilters is a constructor field included in equality, so the copy
+ // survives enclosing-block rebuilds, and the derived scanHelper picks up the rewritten
+ // filters automatically.
+ copy(runtimeFilters = filters)
+ }
+
+ /**
+ * Lazy split-mode serialization, mirroring CometNativeScanExec: common data was serialized at
+ * planning; per-partition file lists serialize here, at execution time.
+ */
+ @transient private lazy val serializedPartitionData
+ : (Array[Byte], Array[Array[Byte]], Array[Seq[String]]) = {
+ // Resolve the helper's DPP subqueries: it holds its own InSubqueryExec instances that
+ // Spark's expressions walk does not see (the helper is derived, not a child).
+ scanHelper.partitionFilters.foreach {
+ case DynamicPruningExpression(e: InSubqueryExec) if e.values().isEmpty =>
+ e.updateResult()
+ case _ =>
+ }
+
+ val commonBytes = {
+ val deltaScan = DeltaSparkScanEnvelope.unpack(nativeOp)
+ // Scalar subqueries in dataFilters were unresolved at planning; resolve them now and
+ // append them as pushed filters, as CometNativeScanExec.serializedPartitionData does.
+ // has_data_filters follows their presence, not the serialized count: a filter that fails
+ // to serialize still keeps native on the safe timestamp conversion for a filtered scan.
+ val resolved = org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .resolvedSubqueryFilters(dataFilters, output, requiredSchema, conf)
+ val common = if (!resolved.hasResolvedFilters) {
+ deltaScan.getCommon
+ } else {
+ val builder = deltaScan.getCommon.toBuilder
+ builder.setHasDataFilters(true)
+ resolved.protos.foreach(builder.addDataFilters)
+ builder.build()
+ }
+ OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaScan.getDeltaCommon)
+ .build()
+ .toByteArray
+ }
+
+ val filePartitions = scanHelper.getFilePartitions()
+
+ val tableRoot = DeltaSparkScanEnvelope.unpack(nativeOp).getDeltaCommon.getTableRoot
+ val perPartitionBytes = filePartitions.map { filePartition =>
+ org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .serializePartition(filePartition, originalPlan, tableRoot)
+ }.toArray
+
+ val perPartitionPaths = filePartitions.map(_.files.map(_.filePath.toString).toSeq).toArray
+
+ (commonBytes, perPartitionBytes, perPartitionPaths)
+ }
+
+ override def commonData: Array[Byte] = serializedPartitionData._1
+
+ override def perPartitionData: Array[Array[Byte]] = serializedPartitionData._2
+
+ def perPartitionFilePaths: Array[Seq[String]] = serializedPartitionData._3
+
+ override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+ val nativeMetrics = CometMetricNode.fromCometPlan(this)
+ val serializedPlan = CometExec.serializeNativePlan(nativeOp)
+
+ new CometExecRDD(
+ sparkContext,
+ Seq.empty,
+ Map(sourceKey -> commonData),
+ Map(sourceKey -> perPartitionData),
+ serializedPlan,
+ perPartitionData.length,
+ output.length,
+ nativeMetrics,
+ Seq.empty,
+ None,
+ Seq.empty,
+ perPartitionFilePaths = perPartitionFilePaths,
+ reportScanInputMetrics = true)
+ }
+
+ override def doCanonicalize(): CometDeltaNativeScanExec = {
+ val canonOriginal = if (originalPlan != null) {
+ val stripped = originalPlan.copy(partitionFilters =
+ CometScanUtils.filterUnusedDynamicPruningExpressions(originalPlan.partitionFilters))
+ stripped.doCanonicalize()
+ } else {
+ null
+ }
+ CometDeltaNativeScanExec(
+ nativeOp,
+ output.map(QueryPlan.normalizeExpressions(_, output)),
+ requiredSchema,
+ QueryPlan.normalizePredicates(
+ CometScanUtils.filterUnusedDynamicPruningExpressions(runtimeFilters),
+ output),
+ QueryPlan.normalizePredicates(dataFilters, output),
+ relation,
+ canonOriginal,
+ SerializedPlan(None),
+ "")
+ }
+
+ override def stringArgs: Iterator[Any] = Iterator(output, runtimeFilters)
+
+ override def equals(obj: Any): Boolean = obj match {
+ case other: CometDeltaNativeScanExec =>
+ this.originalPlan == other.originalPlan &&
+ this.serializedPlanOpt == other.serializedPlanOpt &&
+ this.runtimeFilters == other.runtimeFilters &&
+ this.dataFilters == other.dataFilters
+ case _ => false
+ }
+
+ override def hashCode(): Int =
+ java.util.Objects.hash(originalPlan, serializedPlanOpt, runtimeFilters, dataFilters)
+
+ private val driverMetricKeys =
+ Set(
+ "numFiles",
+ "filesSize",
+ "numPartitions",
+ "metadataTime",
+ "staticFilesNum",
+ "staticFilesSize",
+ "pruningTime")
+
+ // Forces `scanHelper` (see its doc above for why that -- and reading `.metrics` off it -- is
+ // safe even when AQE calls `.metrics` mid-planning against an unresolved DPP/scalar subquery).
+ override lazy val metrics: Map[String, SQLMetric] = {
+ CometMetricNode.nativeScanMetrics(session.sparkContext) ++
+ scanHelper.metrics.filter { case (k, _) => driverMetricKeys.contains(k) }
+ }
+}
+
+object CometDeltaNativeScanExec {
+
+ /** File-planning helper: reuses CometScanExec's listing/splitting/DPP machinery. */
Review Comment:
> Could we make its contract explicitly take the original scan plus the current partition and data filters, and return the final `FilePartition` objects with their metadata?
Tracked in #5946, including the regression tests over serialized paths, splits, partition values and metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5852369806
@andygrove could you approve a CI run on `de2b0133a`? sunchao approved it, and this head carries the merge with #6116, so the Delta scan applies the same missing field ids check. The last CI run was on an earlier head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5861351464
@andygrove could you approve a CI run on `0392218ba`? This head merges main to resolve the conflict with #5880 in `CometNativeScanExec`. The Delta scan now registers its input metrics listener before the native iterator, the same ordering #5880 uses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5379637457
I squashed the commits. I didn't want the co-author tags to be lost if it gets merged. @sunchao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4041682615
##########
dev/ci/compute-changes.py:
##########
@@ -387,6 +387,26 @@
".mvn/**",
"mvnw",
],
+ "delta": [
Review Comment:
This filter matches `native/**/src/**` and `spark/src/main/**`, so it adds four jobs to essentially every queued PR, each doing a native build plus a Maven install plus a suite run. My local Spark 3.5 leg alone was 8m6s of test time on top of the build. Could you put the measured runner-minutes in the description? If it lands where I think it does, it may be worth trimming to one Spark profile in the queue with the other two on the label or nightly, which is how the Iceberg and Spark SQL tiers are already split.
##########
docs/source/user-guide/latest/delta.md:
##########
@@ -0,0 +1,62 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Delta Lake (experimental)
+
+Comet can execute DSv1 Delta Lake table scans natively. Reads planned by
+delta-spark run through Comet's native Parquet scan, inheriting row-group
+pruning, page-index pruning, and filter pushdown, with deletion vectors
+applied inside the scan.
+
+Support is experimental and explicitly opt-in. Two things are required:
+
+1. The `comet-contrib-delta-spark` contrib jar on the classpath, alongside
+ `delta-spark`. It is never bundled into `comet-spark`.
+2. `spark.comet.scan.delta.enabled=true`. The default is `false`, so
+ the jar alone does nothing.
+
+Unsupported tables and features fall back to Spark's reader. See the
+[contrib module README](https://github.com/apache/datafusion-comet/blob/main/contrib/delta-spark/README.md)
+for the supported Spark/Delta version matrix and build instructions.
+
+Unlike the core native scan, the Delta scan resolves each data file's datetime
+calendar-rebase policy from the file's own writer metadata
+(`org.apache.spark.legacyDateTime` and friends), the same way Spark's reader
+does, selecting the `datetimeRebaseModeInRead` spec for dates and INT64
+timestamps and the `int96RebaseModeInRead` spec for INT96 timestamps, at any
+nesting depth: dates written with the legacy hybrid Julian/Gregorian calendar
+are rebased exactly, timestamps are rebased exactly when the file records a
+fixed UTC writer time zone, and ancient values whose calendar cannot be
+applied natively (non-UTC legacy writer zones, or files that do not declare a
+policy under the `EXCEPTION` read mode) raise an error rather than silently
+returning shifted values. Modern values are unaffected: dates from 1582-10-15
+onward, and timestamps from 1900-01-01T00:00:00Z onward (Spark's own
+rebase cutoff). Disable `spark.comet.scan.delta.enabled` for such tables to
+read them through Spark.
+
+## Configuration
+
+<!--BEGIN:CONFIG_TABLE[delta]-->
Review Comment:
This page carries `CONFIG_TABLE` markers, but `DeltaSparkConfigProvider`'s own comment says the doc build cannot see the provider with the current module layout, so nothing fills them. The table is hand-maintained while looking generated, which is how it will drift. The guard in `DeltaScanContribSuite` covers only `COMET_DELTA_NATIVE_ENABLED`'s key and doc string, not `maxDeletedRowsPerFile` and not the default values, and it cancels rather than fails when it cannot find the file. Could the guard cover every entry in `DeltaScanConf.all` including defaults, and fail instead of cancel on a missing file? Or drop the markers so nobody reads the table as generated.
##########
native/core/src/parquet/datetime_rebase.rs:
##########
@@ -0,0 +1,2794 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Per-file datetime calendar-rebase handling for the parquet scan.
+//!
+//! Spark 2.4 and earlier wrote dates and timestamps in the hybrid Julian + Gregorian calendar;
+//! Spark 3.0+ uses the proleptic Gregorian calendar and records the calendar policy of every
+//! file it writes in the parquet footer's key-value metadata (`org.apache.spark.version`,
+//! `org.apache.spark.legacyDateTime`, `org.apache.spark.legacyINT96`,
+//! `org.apache.spark.timeZone`). Spark's reader resolves the rebase policy from EACH FILE's
+//! writer metadata (`DataSourceUtils.datetimeRebaseSpec` / `int96RebaseSpec`) -- the session's
+//! `spark.sql.parquet.datetimeRebaseModeInRead` conf only applies to files whose metadata does
+//! not decide the policy on its own -- so a reader that ignores the metadata silently returns
+//! values shifted by up to ten days for dates before 1582-10-15 (e.g. `1500-01-01` reads as
+//! `1500-01-10`).
+//!
+//! This module mirrors that per-file resolution: [`resolve_file_rebase_policies`] computes the
+//! date / INT64-timestamp / INT96-timestamp policies from a file's arrow schema metadata (the
+//! parquet key-value pairs survive the parquet -> arrow schema conversion), and
+//! [`wrap_datetime_rebase`] wraps the per-file rewritten expressions' column references in a
+//! [`SparkDatetimeRebaseExpr`] that rebases values exactly where that is possible without the
+//! JVM's historical timezone tables (dates always; timestamps for a fixed UTC writer zone) and
+//! refuses -- rather than silently corrupting -- ancient values it cannot rebase. Nested
+//! columns are rebuilt leaf by leaf (struct / list / map / fixed-size list / dictionary), each
+//! leaf under its own policy, with nulls and offsets preserved. Modern values are always the
+//! identity under every policy: from 1582-10-15 onward for dates, and from
+//! [`LAST_SWITCH_JULIAN_TS_SECONDS`] (1900-01-01T00:00:00Z, Spark's
+//! `RebaseDateTime.lastSwitchJulianTs`) onward for timestamps.
+//!
+//! Spark applies `datetimeRebaseSpec` to INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns
+//! and `int96RebaseSpec` to INT96 columns. The two physical types are indistinguishable in the
+//! arrow schema DataFusion hands the expression adapter (both surface as `Timestamp(us, "UTC")`
+//! after INT96 coercion), so Comet's parquet reader factory stamps the file's INT96 leaf
+//! ordinals -- taken from the parquet footer's own `SchemaDescriptor` -- into the key-value
+//! metadata under [`INT96_LEAVES_METADATA_KEY`] before the arrow schema is derived (see
+//! [`stamp_int96_leaves`] and `eager_page_index_reader_factory.rs`), and the adapter attributes
+//! every timestamp leaf to its spec from that stamp. Without a stamp, the two specs are merged:
+//! agreement decides, disagreement degrades to [`RebasePolicy::CheckAncient`].
+//!
+//! The wrapper sits BENEATH the schema adapter's nested narrowing (the struct -> struct convert
+//! that keeps only the requested children), which is what keeps those ordinals physical -- but
+//! it means the wrapper sees every physical child, requested or not. Spark only ever decodes
+//! the requested nested schema, so [`FileRebasePolicies::restrict_to_requested`] marks the
+//! physical leaves the narrowing drops as the identity: an unrequested ancient `s.ts` never
+//! blocks `select s.d`, exactly as in Spark.
+//!
+//! Currently only enabled by the Delta scan arms via
+//! `SparkParquetOptions::rebase_from_file_metadata`, which also carries the session read modes
+//! ([`SessionRebaseModes`], forwarded from the JVM) that decide the policy for files without
+//! Spark writer metadata; the plain NativeScan keeps its documented no-rebase behavior (see
+//! the compatibility guide and issue #5010).
+
+use std::collections::HashMap;
+use std::fmt::{self, Display};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Date32Array, FixedSizeListArray, GenericListArray, MapArray,
+ OffsetSizeTrait, PrimitiveArray, RecordBatch, StructArray,
+};
+use arrow::datatypes::{
+ ArrowPrimitiveType, ArrowTimestampType, DataType, Date32Type, FieldRef, Schema, SchemaRef,
+ TimeUnit, TimestampMicrosecondType, TimestampMillisecondType, TimestampNanosecondType,
+ TimestampSecondType,
+};
+use arrow::error::ArrowError;
+use datafusion::common::tree_node::{Transformed, TreeNode};
+use datafusion::common::{DataFusionError, Result as DataFusionResult};
+use datafusion::physical_expr::expressions::Column;
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::ColumnarValue;
+use parquet::basic::Type as ParquetPhysicalType;
+use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaData};
+use parquet::schema::types::SchemaDescriptor;
+
+use super::name_fold::fold_names;
+use super::schema_adapter::parse_field_id;
+
+/// Footer key naming the Spark release that wrote the file; absent for non-Spark writers.
+const SPARK_VERSION_METADATA_KEY: &str = "org.apache.spark.version";
+/// Present (empty value) when the file's dates and INT64 timestamps were written with
+/// `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_DATETIME_KEY: &str = "org.apache.spark.legacyDateTime";
+/// Present (empty value) when the file's INT96 timestamps were written with
+/// `spark.sql.parquet.int96RebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_INT96_KEY: &str = "org.apache.spark.legacyINT96";
+/// The writer session's time zone, stamped alongside either legacy flag.
+const SPARK_TIMEZONE_KEY: &str = "org.apache.spark.timeZone";
+
+/// Key-value metadata entry Comet's parquet reader factory adds to a file's footer metadata
+/// (in memory only, never written back) so the expression adapter can tell INT96 timestamp
+/// columns from INT64 ones after both have been coerced to the same arrow type. Value:
+/// `"<leaf count>:<comma-separated INT96 leaf ordinals>"`, where leaves are the file's
+/// primitive columns in `SchemaDescriptor::columns()` order -- the same depth-first order
+/// parquet-rs assigns arrow leaves, so an arrow-side depth-first walk lines up with it. The
+/// leaf count lets the reader detect a stamp that does not describe the schema it is paired
+/// with (see [`Int96Attribution::from_schema`]).
+pub(crate) const INT96_LEAVES_METADATA_KEY: &str = "comet.int96_leaf_columns";
+
+/// Day of the Gregorian cutover (1582-10-15) as days since the epoch; rebasing is the identity
+/// from this day onward. Same value as Spark's `RebaseDateTime.lastSwitchJulianDay`.
+const LAST_SWITCH_JULIAN_DAY: i32 = -141427;
+
+/// Spark's `RebaseDateTime.lastSwitchJulianTs` (and `lastSwitchGregorianTs`) in seconds since
+/// the epoch: 1900-01-01T00:00:00Z. Spark derives it as the latest switch instant across every
+/// zone in its `julian-gregorian-rebase-micros.json` table (`getLastSwitchTs`, which also
+/// asserts the calendars' difference is zero for every zone from then on): most zones ran on
+/// local mean time before 1900, so the last instant at which rebasing changes a value in ANY
+/// zone is 1900-01-01T00:00:00Z, not the 1582 cutover. `createTimestampRebaseFuncInRead`
+/// under `EXCEPTION` throws exactly for `micros < lastSwitchJulianTs` (after converting
+/// `TIMESTAMP_MILLIS` to micros), and `rebaseJulianToGregorianMicros` is the identity from it
+/// onward in every zone. The value is in seconds so it scales exactly to any timestamp unit.
+pub(crate) const LAST_SWITCH_JULIAN_TS_SECONDS: i64 = -2_208_988_800;
+
+/// The per-century differences between the Julian and proleptic Gregorian calendars, and the
+/// Julian-calendar switch days at which each difference starts to apply. Copied verbatim from
+/// Spark's `RebaseDateTime.julianGregDiffs` / `julianGregDiffSwitchDay` (which Spark generated
+/// from `localRebaseJulianToGregorianDays`); `rebase_julian_to_gregorian_days` must stay
+/// value-for-value equal to Spark's `rebaseJulianToGregorianDays`.
+const JULIAN_GREG_DIFFS: [i32; 14] = [2, 1, 0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, 0];
+const JULIAN_GREG_DIFF_SWITCH_DAY: [i32; 14] = [
+ -719164, -682945, -646420, -609895, -536845, -500320, -463795, -390745, -354220, -317695,
+ -244645, -208120, -171595, -141427,
+];
+
+/// Proleptic-Gregorian days since 1970-01-01 for a nominal civil date, via Howard Hinnant's
+/// `days_from_civil`. `d` may exceed the month's length; the excess rolls into the following
+/// month exactly like `LocalDate.of(y, m, 1).plusDays(d - 1)` in Spark's
+/// `localRebaseJulianToGregorianDays` (how the non-existent proleptic date `1000-02-29`,
+/// valid in the Julian calendar, lands on `1000-03-01`).
+fn days_from_civil(y: i64, m: i64, d: i64) -> i64 {
+ let y = if m <= 2 { y - 1 } else { y };
+ let era = y.div_euclid(400);
+ let yoe = y - era * 400; // [0, 399]
+ let mp = (m + 9) % 12; // [0, 11], March = 0
+ let doy = (153 * mp + 2) / 5 + d - 1;
+ let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
+ era * 146097 + doe - 719468
+}
+
+/// Julian-calendar civil date `(year, month, day)` for a day count since 1970-01-01 that labels
+/// days in the Julian calendar (astronomical year numbering: 1 BCE is year 0). Standard
+/// Julian-day-number conversion (E.G. Richards' algorithm), exact for any day.
+fn julian_day_to_civil(days: i64) -> (i64, i64, i64) {
+ // Integer (noon) Julian Day Number of this civil day: 1970-01-01 is JDN 2440588.
+ let jdn = days + 2_440_588;
+ let f = jdn + 1401;
+ let e = 4 * f + 3;
+ let g = e.rem_euclid(1461) / 4;
+ let h = 5 * g + 2;
+ let day = h.rem_euclid(153) / 5 + 1;
+ let month = (h / 153 + 2).rem_euclid(12) + 1;
+ let year = e.div_euclid(1461) - 4716 + (14 - month) / 12;
+ (year, month, day)
+}
+
+/// Exact port of Spark's `RebaseDateTime.rebaseJulianToGregorianDays`: reinterprets a day count
+/// written in the hybrid Julian + Gregorian calendar as the proleptic Gregorian day count of the
+/// same nominal civil date. Identity for days from 1582-10-15 onward. Days before the tables'
+/// range (before Julian `0001-01-01`) take the calendar-arithmetic path, mirroring Spark's
+/// `localRebaseJulianToGregorianDays` fallback.
+pub(crate) fn rebase_julian_to_gregorian_days(days: i32) -> i32 {
+ if days < JULIAN_GREG_DIFF_SWITCH_DAY[0] {
+ let (y, m, d) = julian_day_to_civil(days as i64);
+ (days_from_civil(y, m, 1) + (d - 1)) as i32
+ } else {
+ // Spark's rebaseDays: linear search from the most recent switch day.
+ let mut i = JULIAN_GREG_DIFF_SWITCH_DAY.len();
+ loop {
+ i -= 1;
+ if i == 0 || days >= JULIAN_GREG_DIFF_SWITCH_DAY[i] {
+ break;
+ }
+ }
+ days + JULIAN_GREG_DIFFS[i]
+ }
+}
+
+/// Timezone strings from `org.apache.spark.timeZone` that denote a fixed zero-offset zone in
+/// both `java.util.TimeZone` and `java.time`. Only for these is timestamp rebasing the pure
+/// nominal-date shift [`SparkDatetimeRebaseExpr::rebase_timestamp_utc`] computes; any other (or
+/// absent) zone needs the JVM's historical timezone tables and stays on the
+/// refuse-ancient-values path.
+const UTC_EQUIVALENT_TIMEZONES: [&str; 6] = ["UTC", "Etc/UTC", "GMT", "Etc/GMT", "Z", "+00:00"];
+
+/// How the writer's session time zone (if recorded) affects timestamp rebasing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum WriterTimeZone {
+ /// A fixed zero-offset zone: rebasing reduces to the exact nominal-date shift.
+ Utc,
+ /// Any other zone, or none recorded (pre-3.0 files): ancient values cannot be rebased
+ /// without the JVM's historical timezone data.
+ OtherOrUnknown,
+}
+
+/// One session-level datetime rebase read mode (a `LegacyBehaviorPolicy` value of
+/// `spark.sql.parquet.datetimeRebaseModeInRead` / `int96RebaseModeInRead`), consulted by
+/// [`resolve_file_rebase_policies`] ONLY for files whose footer metadata does not decide the
+/// policy on its own -- exactly the `getOrElse` fallback in Spark's
+/// `DataSourceUtils.getRebaseSpec`. Files that carry `org.apache.spark.version` ignore these
+/// modes entirely, on every Spark version.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) enum RebaseReadMode {
+ /// Refuse ancient values (Spark raises `SparkUpgradeException`); maps to
+ /// [`RebasePolicy::CheckAncient`]. The default mirrors the conservative posture used
+ /// before the conf was plumbed through (and Spark 3.x's own conf default).
+ #[default]
+ Exception,
+ /// Read values as proleptic Gregorian without rebasing.
+ Corrected,
+ /// Rebase from the hybrid Julian + Gregorian calendar.
+ Legacy,
+}
+
+impl RebaseReadMode {
+ /// Parses a `LegacyBehaviorPolicy` conf value. `SQLConf` validates and upper-cases the
+ /// session conf, but a per-relation `datetimeRebaseMode` option arrives verbatim, so the
+ /// match is case-insensitive. Anything unrecognized -- including the empty string a proto
+ /// producer that predates the field sends -- falls back to [`RebaseReadMode::Exception`],
+ /// which refuses ancient values rather than silently corrupting them.
+ pub(crate) fn from_conf_value(value: &str) -> Self {
+ match value.to_ascii_uppercase().as_str() {
+ "CORRECTED" => RebaseReadMode::Corrected,
+ "LEGACY" => RebaseReadMode::Legacy,
+ _ => RebaseReadMode::Exception,
+ }
+ }
+}
+
+/// The session's effective datetime rebase read modes, one per spec class (INT64
+/// dates/timestamps vs INT96 timestamps), forwarded from the JVM at planning time.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) struct SessionRebaseModes {
+ /// `spark.sql.parquet.datetimeRebaseModeInRead` (or the relation's `datetimeRebaseMode`).
+ pub datetime: RebaseReadMode,
+ /// `spark.sql.parquet.int96RebaseModeInRead` (or the relation's `int96RebaseMode`).
+ pub int96: RebaseReadMode,
+}
+
+/// Calendar policy of one file's date or timestamp columns, resolved from writer metadata the
+/// same way Spark's `DataSourceUtils.getRebaseSpec` resolves it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum RebasePolicy {
+ /// Written in the proleptic Gregorian calendar; values pass through untouched.
+ Corrected,
+ /// Written in the hybrid Julian + Gregorian calendar; values must be rebased.
+ Legacy(WriterTimeZone),
+ /// Policy could not be pinned down (contradictory flags, or a non-Spark writer under the
+ /// `EXCEPTION` read mode): modern values -- identical under either calendar -- pass,
+ /// ancient values raise. Mirrors Spark's `EXCEPTION` behavior (`SparkUpgradeException`).
+ CheckAncient,
+}
+
+/// Which of a file's leaf columns are physically INT96, from the stamp the parquet reader
+/// factory adds under [`INT96_LEAVES_METADATA_KEY`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) enum Int96Attribution {
+ /// No stamp, or a stamp whose leaf count does not match the schema it arrived with: the
+ /// INT64 and INT96 timestamp specs cannot be told apart per column and are merged.
+ Unknown,
+ /// Sorted leaf ordinals (depth-first over the file schema's primitive columns) that are
+ /// INT96; every other timestamp leaf is INT64.
+ Known(Vec<usize>),
+}
+
+impl Int96Attribution {
+ /// Parses the stamp out of `schema`'s metadata and validates its leaf count against the
+ /// schema's own depth-first leaf count, so a stamp that does not describe this schema (a
+ /// crafted footer key, or a cached-metadata mismatch) degrades to [`Self::Unknown`].
+ fn from_schema(schema: &Schema) -> Self {
+ let Some(stamp) = schema.metadata().get(INT96_LEAVES_METADATA_KEY) else {
+ return Int96Attribution::Unknown;
+ };
+ let Some((count, ordinals)) = stamp.split_once(':') else {
+ return Int96Attribution::Unknown;
+ };
+ let schema_leaves: usize = schema
+ .fields()
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ if count.parse::<usize>().ok() != Some(schema_leaves) {
+ return Int96Attribution::Unknown;
+ }
+ let parsed: Option<Vec<usize>> = if ordinals.is_empty() {
+ Some(Vec::new())
+ } else {
+ ordinals
+ .split(',')
+ .map(|o| o.parse::<usize>().ok().filter(|o| *o < schema_leaves))
+ .collect()
+ };
+ match parsed {
+ Some(mut leaves) => {
+ leaves.sort_unstable();
+ Int96Attribution::Known(leaves)
+ }
+ None => Int96Attribution::Unknown,
+ }
+ }
+
+ /// `Some(true)` / `Some(false)` when the leaf is known to be INT96 / INT64, `None` when
+ /// the attribution is unknown.
+ fn is_int96(&self, leaf: usize) -> Option<bool> {
+ match self {
+ Int96Attribution::Unknown => None,
+ Int96Attribution::Known(leaves) => Some(leaves.binary_search(&leaf).is_ok()),
+ }
+ }
+}
+
+/// The [`INT96_LEAVES_METADATA_KEY`] value describing `schema`: its leaf count and the
+/// ordinals of its INT96 primitive columns.
+pub(crate) fn int96_leaf_stamp(schema: &SchemaDescriptor) -> String {
+ let ordinals: Vec<String> = schema
+ .columns()
+ .iter()
+ .enumerate()
+ .filter(|(_, column)| column.physical_type() == ParquetPhysicalType::INT96)
+ .map(|(ordinal, _)| ordinal.to_string())
+ .collect();
+ format!("{}:{}", schema.num_columns(), ordinals.join(","))
+}
+
+/// Returns a copy of `metadata` whose key-value metadata carries the [`int96_leaf_stamp`] of
+/// its own schema, or `None` when it already does (the common case after the first open of a
+/// file, since the caller caches the stamped copy). Any pre-existing entry under the key --
+/// a file cannot legitimately carry one -- is replaced, never trusted. Only the file-level
+/// key-value list changes; row groups and page indexes are carried over as-is. The parquet
+/// API cannot carry a file decryptor, nor `FileMetaData`'s crate-private encryption fields
+/// (encryption algorithm, footer signing key metadata), across this rebuild, so callers must
+/// not stamp opens that supply decryption properties -- and the only consumer, the Delta
+/// scan, declines every encrypted-parquet configuration before planning, so a parquet
+/// modular encryption file never reaches this path with or without those properties.
+pub(crate) fn stamp_int96_leaves(metadata: &ParquetMetaData) -> Option<ParquetMetaData> {
+ let file_metadata = metadata.file_metadata();
+ let stamp = int96_leaf_stamp(file_metadata.schema_descr());
+ let existing = file_metadata
+ .key_value_metadata()
+ .and_then(|kvs| kvs.iter().find(|kv| kv.key == INT96_LEAVES_METADATA_KEY))
+ .and_then(|kv| kv.value.as_deref());
+ if existing == Some(stamp.as_str()) {
+ return None;
+ }
+ let mut key_values: Vec<KeyValue> = file_metadata
+ .key_value_metadata()
+ .map(|kvs| {
+ kvs.iter()
+ .filter(|kv| kv.key != INT96_LEAVES_METADATA_KEY)
+ .cloned()
+ .collect()
+ })
+ .unwrap_or_default();
+ key_values.push(KeyValue::new(INT96_LEAVES_METADATA_KEY.to_string(), stamp));
+ let stamped_file_metadata = FileMetaData::new(
+ file_metadata.version(),
+ file_metadata.num_rows(),
+ file_metadata.created_by().map(str::to_string),
+ Some(key_values),
+ file_metadata.schema_descr_ptr(),
+ file_metadata.column_orders().cloned(),
+ );
+ Some(
+ ParquetMetaData::new(stamped_file_metadata, metadata.row_groups().to_vec())
+ .into_builder()
+ .set_column_index(metadata.column_index().cloned())
+ .set_offset_index(metadata.offset_index().cloned())
+ .build(),
+ )
+}
+
+/// Per-file rebase policies for the three affected column classes, plus the INT96
+/// attribution that selects between the two timestamp specs per leaf.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct FileRebasePolicies {
+ /// `DATE` columns, governed by `org.apache.spark.legacyDateTime` alone.
+ pub date: RebasePolicy,
+ /// INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns: the datetime spec (same
+ /// resolution as `date`), as Spark's `ParquetVectorUpdaterFactory` selects for INT64.
+ pub int64_timestamp: RebasePolicy,
+ /// INT96 columns: the INT96 spec (`org.apache.spark.legacyINT96`, min version 3.1.0).
+ pub int96_timestamp: RebasePolicy,
+ /// Which timestamp leaves are INT96. See [`Int96Attribution`].
+ pub int96_leaves: Int96Attribution,
+ /// Sorted depth-first leaf ordinals -- over the physical file schema, the same ordinals
+ /// `int96_leaves` uses -- that the query does not read: nested children the schema
+ /// adapter's struct narrowing drops before any value leaves the scan. Spark never decodes
+ /// them either, so their policy is the identity whatever the file's calendar. Empty until
+ /// [`Self::restrict_to_requested`] runs (every leaf requested).
+ pub unrequested_leaves: Vec<usize>,
+}
+
+impl FileRebasePolicies {
+ /// True when some policy is not the plain proleptic-Gregorian pass-through, i.e. when the
+ /// per-column wrap in [`wrap_datetime_rebase`] can install anything at all.
+ pub(crate) fn any_rebase_needed(&self) -> bool {
+ self.date != RebasePolicy::Corrected
+ || self.int64_timestamp != RebasePolicy::Corrected
+ || self.int96_timestamp != RebasePolicy::Corrected
+ }
+
+ fn is_requested(&self, leaf: usize) -> bool {
+ self.unrequested_leaves.binary_search(&leaf).is_err()
+ }
+
+ /// The policy of the `Date32` leaf at depth-first ordinal `leaf`: the file's date policy,
+ /// or the identity when the query does not read that leaf.
+ fn date_policy(&self, leaf: usize) -> RebasePolicy {
+ if self.is_requested(leaf) {
+ self.date
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+
+ /// The policy of the timezone-carrying timestamp leaf at depth-first ordinal `leaf`: the
+ /// identity when the query does not read it; otherwise its physical type's spec when the
+ /// attribution is known, or else the two specs merged -- agreement decides, disagreement
+ /// degrades to [`RebasePolicy::CheckAncient`], which still passes every modern value and
+ /// refuses only ancient ones.
+ fn timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ Some(false) => self.int64_timestamp,
+ None if self.int64_timestamp == self.int96_timestamp => self.int64_timestamp,
+ None => RebasePolicy::CheckAncient,
+ }
+ }
+
+ /// These policies with every physical leaf the query does not read marked the identity.
+ /// `requested` pairs each top-level field of `physical_schema` (by position) with the type
+ /// of the logical field the schema adapter narrows it to -- `None` for a column without a
+ /// logical counterpart, whose leaves are left as they are (no expression reads it anyway).
+ /// Nested children pair the way the adapter's struct convert selects them (see
+ /// [`push_unrequested_leaves`]); the INT96 attribution is untouched, since the ordinals
+ /// stay physical. `requested` is parallel to the schema's fields; should a caller pass a
+ /// shorter slice, the trailing columns simply keep every leaf (the safe direction).
+ pub(crate) fn restrict_to_requested(
+ mut self,
+ physical_schema: &Schema,
+ requested: &[Option<&DataType>],
+ case_sensitive: bool,
+ use_field_id: bool,
+ ) -> Self {
+ debug_assert_eq!(requested.len(), physical_schema.fields().len());
+ let matching = FieldMatching {
+ case_sensitive,
+ use_field_id,
+ };
+ let mut next_leaf = 0;
+ let mut unrequested = Vec::new();
+ for (field, requested) in physical_schema.fields().iter().zip(requested) {
+ match requested {
+ Some(logical) => push_unrequested_leaves(
+ field.data_type(),
+ logical,
+ &mut next_leaf,
+ matching,
+ &mut unrequested,
+ ),
+ None => next_leaf += leaf_count(field.data_type()),
+ }
+ }
+ // Emitted in depth-first order, so already sorted for `is_requested`'s binary search.
+ self.unrequested_leaves = unrequested;
+ self
+ }
+}
+
+/// The field-matching rules of the schema adapter's nested narrowing
+/// (`parquet_convert_struct_to_struct`): names fold per `case_sensitive`, and Parquet field ids
+/// select fields when `use_field_id` is set.
+#[derive(Debug, Clone, Copy)]
+struct FieldMatching {
+ case_sensitive: bool,
+ use_field_id: bool,
+}
+
+/// Appends to `out` the depth-first leaf ordinals of `physical` (counting from `next_leaf`,
+/// which advances past every leaf of `physical`) that reading it as `requested` drops.
+///
+/// Recurses through exactly the pairings `parquet_convert_array` narrows, and no others: a
+/// struct child is dropped only when NO requested child selects it by either rule the struct
+/// convert uses -- folded name, or Parquet field id when ids are in play -- and an ambiguous
+/// child (several requested children select it) is kept; `List` pairs with `List` by element
+/// type, and `Map` with a `Map` of the same key ordering by its entries, positionally. Any
+/// other pairing -- a leaf, a `LargeList` / `FixedSizeList` / dictionary, a map whose ordering
+/// differs, or a shape mismatch -- is handed to arrow's cast or passed through whole by the
+/// convert, so it keeps every leaf. Keeping a superset of what the narrowing reads is always
+/// safe (a spurious check at worst); dropping a leaf the narrowing reads would skip its
+/// rebase, so every doubt resolves to "requested".
+fn push_unrequested_leaves(
+ physical: &DataType,
+ requested: &DataType,
+ next_leaf: &mut usize,
+ matching: FieldMatching,
+ out: &mut Vec<usize>,
+) {
+ match (physical, requested) {
+ (DataType::Struct(physical_fields), DataType::Struct(requested_fields)) => {
+ let names: Vec<&str> = physical_fields
+ .iter()
+ .chain(requested_fields.iter())
+ .map(|f| f.name().as_str())
+ .collect();
+ // A fold failure means the names could not be compared at all; keeping every leaf
+ // requested is the safe superset, the same as the pass-through pairings below.
+ let Ok(folded) = fold_names(&names, matching.case_sensitive) else {
+ *next_leaf += leaf_count(physical);
+ return;
+ };
+ let (physical_folded, requested_folded) = folded.split_at(physical_fields.len());
+ for (i, child) in physical_fields.iter().enumerate() {
+ let child_id = if matching.use_field_id {
+ parse_field_id(child)
+ } else {
+ None
+ };
+ let mut selectors = requested_fields.iter().enumerate().filter(|(j, r)| {
+ requested_folded[*j] == physical_folded[i]
+ || (child_id.is_some() && parse_field_id(r) == child_id)
+ });
+ match (selectors.next(), selectors.next()) {
+ (None, _) => {
+ let n = leaf_count(child.data_type());
+ out.extend(*next_leaf..*next_leaf + n);
+ *next_leaf += n;
+ }
+ (Some((_, requested_child)), None) => push_unrequested_leaves(
+ child.data_type(),
+ requested_child.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (Some(_), Some(_)) => *next_leaf += leaf_count(child.data_type()),
+ }
+ }
+ }
+ (DataType::List(physical_item), DataType::List(requested_item)) => push_unrequested_leaves(
+ physical_item.data_type(),
+ requested_item.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (
+ DataType::Map(physical_entries, physical_sorted),
+ DataType::Map(requested_entries, requested_sorted),
+ ) if physical_sorted == requested_sorted => {
+ match (physical_entries.data_type(), requested_entries.data_type()) {
+ (DataType::Struct(physical_kv), DataType::Struct(requested_kv))
+ if physical_kv.len() == requested_kv.len() =>
+ {
+ for (p, r) in physical_kv.iter().zip(requested_kv.iter()) {
+ push_unrequested_leaves(
+ p.data_type(),
+ r.data_type(),
+ next_leaf,
+ matching,
+ out,
+ );
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+}
+
+/// The writer time zone recorded in `metadata`, classified for timestamp rebasing. Mirrors the
+/// `Option(lookupFileMeta(SPARK_TIMEZONE_METADATA_KEY))` lookup Spark's `getRebaseSpec` performs
+/// for every LEGACY resolution, conf-fallback included; Spark substitutes the JVM default zone
+/// when the key is absent (`RebaseSpec.timeZone`), which is unavailable natively, so an absent or
+/// non-UTC zone classifies as [`WriterTimeZone::OtherOrUnknown`] (dates still rebase fully --
+/// the day rebase is zone-free -- while ancient timestamps refuse rather than guess).
+fn writer_time_zone(metadata: &HashMap<String, String>) -> WriterTimeZone {
+ match metadata.get(SPARK_TIMEZONE_KEY) {
+ Some(tz) if UTC_EQUIVALENT_TIMEZONES.contains(&tz.as_str()) => WriterTimeZone::Utc,
+ _ => WriterTimeZone::OtherOrUnknown,
+ }
+}
+
+/// One spec resolution, mirroring Spark's `DataSourceUtils.getRebaseSpec` exactly: a Spark
+/// version below `min_version` (lexicographic comparison, same as the Scala `String.<`) or a
+/// present legacy flag means LEGACY; a Spark version at/after `min_version` without the flag
+/// means CORRECTED; no Spark version at all falls back to `conf_mode`, the session read conf
+/// forwarded from the JVM (`getRebaseSpec`'s `modeByConfig` fallback, its ONLY use of the
+/// conf): CORRECTED passes values through, LEGACY rebases (with the writer zone from the
+/// file's `org.apache.spark.timeZone` key, same lookup as the metadata-driven LEGACY path),
+/// and EXCEPTION refuses ancient values as [`RebasePolicy::CheckAncient`].
+fn resolve_spec(
+ metadata: &HashMap<String, String>,
+ min_version: &str,
+ legacy_key: &str,
+ conf_mode: RebaseReadMode,
+) -> RebasePolicy {
+ match metadata.get(SPARK_VERSION_METADATA_KEY) {
+ None => match conf_mode {
+ RebaseReadMode::Corrected => RebasePolicy::Corrected,
+ RebaseReadMode::Legacy => RebasePolicy::Legacy(writer_time_zone(metadata)),
+ RebaseReadMode::Exception => RebasePolicy::CheckAncient,
+ },
+ Some(version) => {
+ if version.as_str() < min_version || metadata.contains_key(legacy_key) {
+ RebasePolicy::Legacy(writer_time_zone(metadata))
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+ }
+}
+
+/// Resolves the per-file rebase policies from a file's arrow schema: the parquet footer's
+/// key-value pairs in its metadata decide the specs (the datetime spec uses min version
+/// `3.0.0` and the INT96 spec `3.1.0`, matching `DataSourceUtils.datetimeRebaseSpec` /
+/// `int96RebaseSpec`; `session_modes` supplies the per-spec conf fallback for files without
+/// Spark writer metadata), and the reader factory's INT96 stamp -- validated against the
+/// schema's leaf structure -- attributes each timestamp leaf to its spec.
+pub(crate) fn resolve_file_rebase_policies(
+ physical_file_schema: &Schema,
+ session_modes: SessionRebaseModes,
+) -> FileRebasePolicies {
+ let metadata = physical_file_schema.metadata();
+ let datetime_spec = resolve_spec(
+ metadata,
+ "3.0.0",
+ SPARK_LEGACY_DATETIME_KEY,
+ session_modes.datetime,
+ );
+ let int96_spec = resolve_spec(
+ metadata,
+ "3.1.0",
+ SPARK_LEGACY_INT96_KEY,
+ session_modes.int96,
+ );
+ FileRebasePolicies {
+ date: datetime_spec,
+ int64_timestamp: datetime_spec,
+ int96_timestamp: int96_spec,
+ int96_leaves: Int96Attribution::from_schema(physical_file_schema),
+ unrequested_leaves: Vec::new(),
+ }
+}
+
+/// Number of primitive leaves `dt` contains in a depth-first walk -- the same count and order
+/// parquet-rs uses when it maps the file's `SchemaDescriptor` columns onto the arrow schema, so
+/// arrow-side leaf ordinals line up with [`int96_leaf_stamp`]'s.
+fn leaf_count(dt: &DataType) -> usize {
+ match dt {
+ DataType::Struct(fields) => fields.iter().map(|f| leaf_count(f.data_type())).sum(),
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_count(f.data_type()),
+ DataType::Dictionary(_, value) => leaf_count(value),
+ DataType::RunEndEncoded(_, value) => leaf_count(value.data_type()),
+ DataType::Union(fields, _) => fields.iter().map(|(_, f)| leaf_count(f.data_type())).sum(),
+ _ => 1,
+ }
+}
+
+/// Appends the policy of every leaf of `dt`, in depth-first order, to `out`, consuming leaf
+/// ordinals from `next_leaf` (exactly [`leaf_count`] of them). Only `Date32` and
+/// timezone-carrying timestamps have a policy to apply, and only when the query reads the
+/// leaf; timezone-free timestamps are `TIMESTAMP_NTZ`, which Spark never rebases, and every
+/// other leaf is the identity ([`RebasePolicy::Corrected`]).
+fn leaf_policies(
+ dt: &DataType,
+ next_leaf: &mut usize,
+ policies: &FileRebasePolicies,
+ out: &mut Vec<RebasePolicy>,
+) {
+ match dt {
+ DataType::Date32 => {
+ out.push(policies.date_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Timestamp(_, Some(_)) => {
+ out.push(policies.timestamp_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Struct(fields) => {
+ for f in fields {
+ leaf_policies(f.data_type(), next_leaf, policies, out);
+ }
+ }
+ // Mirrors `leaf_count` variant for variant, so a rebase-affected leaf inside a nested
+ // type `rebase_array` cannot rebuild (views, run-end, union -- never produced from a
+ // parquet schema) still gets its real policy and makes `rebase_array` refuse loudly
+ // instead of being stamped the identity.
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_policies(f.data_type(), next_leaf, policies, out),
+ DataType::Dictionary(_, value) => leaf_policies(value, next_leaf, policies, out),
+ DataType::RunEndEncoded(_, value) => {
+ leaf_policies(value.data_type(), next_leaf, policies, out)
+ }
+ DataType::Union(fields, _) => {
+ for (_, f) in fields.iter() {
+ leaf_policies(f.data_type(), next_leaf, policies, out);
+ }
+ }
+ _ => {
+ *next_leaf += 1;
+ out.push(RebasePolicy::Corrected);
+ }
+ }
+}
+
+/// Wraps every column reference in `expr` whose physical file type contains a rebase-affected
+/// leaf under a policy that needs handling with a [`SparkDatetimeRebaseExpr`] carrying that
+/// column's per-leaf policies, so both the per-file projection and the pushed-down predicate
+/// evaluate rebased values. Columns whose leaves are all the identity -- unaffected types,
+/// affected types under [`RebasePolicy::Corrected`], or leaves the query does not read (see
+/// [`FileRebasePolicies::restrict_to_requested`]) -- pass through unwrapped. (The pruning
+/// predicates derived from the wrapped predicate treat the wrapper as an opaque expression and
+/// skip pruning on those columns -- conservative, since file-level statistics are in the
+/// file's own calendar.)
+pub(crate) fn wrap_datetime_rebase(
+ expr: Arc<dyn PhysicalExpr>,
+ physical_schema: &SchemaRef,
+ policies: &FileRebasePolicies,
+) -> DataFusionResult<Arc<dyn PhysicalExpr>> {
+ expr.transform(|e| {
+ let Some(col) = e.downcast_ref::<Column>() else {
+ return Ok(Transformed::no(e));
+ };
+ // Missing columns were already replaced with literals; any surviving reference is
+ // physical-schema-indexed. Out-of-range means a non-file column (defensive): skip.
+ let Some(field) = physical_schema.fields().get(col.index()) else {
+ return Ok(Transformed::no(e));
+ };
+ // This column's first leaf ordinal: the leaves of every preceding top-level field.
+ let mut next_leaf: usize = physical_schema.fields()[..col.index()]
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ let mut column_leaf_policies = Vec::with_capacity(leaf_count(field.data_type()));
+ leaf_policies(
+ field.data_type(),
+ &mut next_leaf,
+ policies,
+ &mut column_leaf_policies,
+ );
+ if column_leaf_policies
+ .iter()
+ .all(|p| *p == RebasePolicy::Corrected)
+ {
+ return Ok(Transformed::no(e));
+ }
+ Ok(Transformed::yes(Arc::new(SparkDatetimeRebaseExpr {
+ child: e,
+ field: Arc::clone(field),
+ leaf_policies: column_leaf_policies,
+ }) as Arc<dyn PhysicalExpr>))
+ })
+ .map(|t| t.data)
+}
+
+/// Applies a file's calendar-rebase policies to one column: rebases exactly where possible,
+/// raises on ancient values it cannot rebase, and passes modern values (the identity under
+/// every policy) through untouched. Nested columns are rebuilt leaf by leaf with nulls and
+/// offsets preserved. See the module doc for the policy table.
+#[derive(Debug, Eq)]
+struct SparkDatetimeRebaseExpr {
+ child: Arc<dyn PhysicalExpr>,
+ /// The physical file field this expression reads (type preserved by the rebase).
+ field: FieldRef,
+ /// One policy per primitive leaf of `field`'s type, in depth-first order (a single entry
+ /// for a flat column). At least one is not [`RebasePolicy::Corrected`].
+ leaf_policies: Vec<RebasePolicy>,
+}
+
+impl SparkDatetimeRebaseExpr {
+ /// The refusal error, as an [`ArrowError`] so `try_unary` closures can raise it directly;
+ /// it converts into a `DataFusionError` at the `?` in `evaluate`.
+ fn rebase_error(&self, detail: &str) -> ArrowError {
+ ArrowError::ComputeError(format!(
+ "Native scan cannot rebase ancient values in column '{}': the file was written \
+ with the legacy (hybrid Julian/Gregorian) calendar, or does not declare which \
+ calendar it used, and {detail}. Reading it natively would return silently \
+ shifted values; disable the native Delta scan \
+ (spark.comet.scan.delta.enabled=false) to let Spark read this table",
+ self.field.name(),
+ ))
+ }
+
+ fn internal_error(&self, detail: impl Display) -> DataFusionError {
+ DataFusionError::Internal(format!(
+ "SparkDatetimeRebaseExpr on column '{}': {detail}",
+ self.field.name()
+ ))
+ }
+
+ /// Rebases a timestamp column written at a fixed zero-offset zone: shift the nominal day
+ /// with the exact date table, keep the time of day. Matches Spark's
+ /// `rebaseJulianToGregorianMicros` for UTC, where the hybrid calendar's day boundaries sit
+ /// exactly on multiples of a day and no timezone transition can apply (UTC's last switch
+ /// instant in Spark's rebase table is the 1582-10-15 cutover itself).
+ fn rebase_timestamp_utc(&self, v: i64, units_per_day: i64) -> Result<i64, ArrowError> {
+ // Compare in days, not units: the cutover day times a nanosecond day does not fit i64.
+ let day = v.div_euclid(units_per_day);
+ if day >= LAST_SWITCH_JULIAN_DAY as i64 {
+ return Ok(v);
+ }
+ let time_of_day = v - day * units_per_day;
+ let day = i32::try_from(day).map_err(|_| {
+ self.rebase_error("the value is outside the rebaseable timestamp range")
+ })?;
+ let rebased = rebase_julian_to_gregorian_days(day) as i64;
+ rebased
+ .checked_mul(units_per_day)
+ .and_then(|d| d.checked_add(time_of_day))
+ .ok_or_else(|| self.rebase_error("the rebased value overflows the timestamp range"))
+ }
+
+ /// Whether `array` holds no null and no value before the Gregorian cutover, so no rebase
+ /// and no rejection applies and the batch can pass through untouched. The vectorised
+ /// minimum makes this one cheap pass on null-free values; arrays with nulls take the
+ /// per-value path below, which skips nulls without a second pass.
+ fn all_modern<T: ArrowPrimitiveType<Native = i64>>(
+ array: &PrimitiveArray<T>,
+ cutover: i64,
+ ) -> bool {
+ array.null_count() == 0 && arrow::compute::min(array).is_none_or(|min| min >= cutover)
Review Comment:
The `null_count() == 0` guard here undoes the pass-through this function exists for. `arrow::compute::min` already skips nulls, so the guard is redundant, and with it any column holding a single null takes the `try_unary` path and allocates a fresh values buffer on every batch, even when every valid value is modern. I proved it with a probe asserting `Arc::ptr_eq` on a nullable all-modern batch, which fails on this head under all three non-`Corrected` policies.
This is reachable by default rather than exotic. On Spark 3.4 and 3.5 `datetimeRebaseModeInRead` defaults to `EXCEPTION`, which maps to `CheckAncient` for any file with no Spark writer metadata, so every requested date and timestamp leaf of a Delta table written by delta-rs, Trino, Flink or DuckDB gets the wrapper. Spark 4.x defaults to `CORRECTED`, so the exposure is 3.4 and 3.5, and it is Delta-arm only since `rebase_from_file_metadata` gates it.
Measured on an 8192-row `TimestampMicrosecond` batch under `CheckAncient`, release build, 20k iterations:
| variant | null-free | one null |
|---|---|---|
| this head | 0.18 ns/row | 0.71 ns/row, plus a fresh 64 KB buffer per batch |
| `min` alone | 0.14 | 0.78, since null-aware `min` is the slow part |
| hybrid below | 0.16 | 0.43, same `Arc` returned |
So dropping the guard on its own is not the answer. Keeping the vectorised `min` only when the array is null-free, and falling back to the validity-aware loop this commit replaced, gets the best of both:
```rust
match array.nulls() {
None => arrow::compute::min(array).is_none_or(|min| min >= cutover),
Some(nulls) => array
.values()
.iter()
.zip(nulls.iter())
.all(|(&v, valid)| !valid || v >= cutover),
}
```
All 43 tests in the module pass with that, plus the probe. The same shape applies to the date copy at line 913.
One more thing. `6039c304` also weakened the test that would have caught this: `modern_batches_pass_through_without_a_new_buffer` used to loop `for input in [&dates, &masked]` asserting `Arc::ptr_eq` on both, and it now pulls `masked` out and only checks `out.null_count() == 1`. Could that assertion go back?
##########
.github/workflows/delta_contrib_test.yml:
##########
@@ -0,0 +1,169 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Delta Contrib Tests
+
+# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
+# live in the umbrella workflow.
+on:
+ workflow_call:
+
+permissions:
+ contents: read
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: rust-lld cannot resolve -ljvm against the Zulu JDK
+ # layout installed by setup-java (same rationale as pr_build_linux.yml).
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+ # The container's default locale is POSIX, which makes the JVM's file-path encoder
+ # reject non-ASCII partition directory names the suites create.
+ LANG: "C.UTF-8"
+ LC_ALL: "C.UTF-8"
+
+jobs:
+
+ contrib-delta:
+ name: Delta contrib (Spark ${{ matrix.profile.spark }})
+ runs-on: ubuntu-24.04
+ container:
Review Comment:
Thanks for the dedicated runner job, that solves the Docker socket problem.
One thing still bothers me though. `CometDeltaS3Suite.beforeAll` catches a MinIO startup failure, flips `dockerAvailable` back to false, and lets every test cancel, which scalatest reports as green. That fail-soft is right for the container jobs, but `contrib-delta-s3` exists only to run this suite, so a zero-coverage run and a real one look identical from the outside. Could the suite take an env var, something like `COMET_DELTA_S3_REQUIRED=1`, that this job sets to turn the cancel into a hard failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5373688862
Reposting the two remaining P2 findings here for visibility. Both remain present at `95125623`; these are the existing findings, not additional issues.
### [P2] Check selected-file schemes before claiming a shallow clone
The filesystem gate checks only the table's `rootPaths`. A valid Delta shallow clone can have a supported `file:` root while its selected data files still reference `viewfs:`. With the default libhdfs configuration (`hdfs` only), both authority checks accept those files and the contrib claims the scan. Native store preparation then fails with `Unable to recognise URL "viewfs://..."` instead of falling back to Spark.
This was verified with a real Spark 4.0.2 / Delta 4.0.0 shallow clone that Spark successfully reads, plus the exact native store-preparation helper. Please apply the supported-scheme check to the selected data-file URIs before claiming the scan.
[Code](https://github.com/apache/datafusion-comet/blob/951256235efa99c35002d10b64368a2a1520482a/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala#L272-L278) · [Existing discussion and reproduction details](https://github.com/apache/datafusion-comet/pull/5365#discussion_r3831646182)
### [P2] Account for the DV reader's combined-selection allocation
Construction admission and the initial reader clone are now covered. However, DataFusion 54.1 subsequently calls `into_overall_row_selection`, which allocates another selector buffer while the attached original and the consumed clone's backing vector remain live. The reservation has already been reduced to twice the retained selector bytes.
With the default-permitted 1,000,000 alternating deletions across 2,000,000 rows, the current attachment reserves **64,000,000 bytes**, but the attached selectors plus reader-normalization allocations peak at **97,554,457 bytes** and retain **65,554,432 bytes** afterward. Please account for normalization and vector capacity, or avoid the additional allocation through ownership transfer. Simply changing the factor to 3 would still fall below this measured peak.
This was reproduced using the unchanged attachment code and the real locked dependency conversion. These are allocator-requested bytes, not RSS or a reproduced executor OOM. Both findings were checked with focused probes and source tracing, not a full Comet/JNI integration run.
[Code](https://github.com/apache/datafusion-comet/blob/951256235efa99c35002d10b64368a2a1520482a/native/core/src/execution/delta_dv.rs#L504-L507) · [Existing discussion and reproduction details](https://github.com/apache/datafusion-comet/pull/5365#discussion_r3831648406)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3831212662
##########
native/core/src/execution/planner/delta_spark_scan.rs:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! JVM-planned Delta handler for the generic `OpStruct::ContribScan` dispatcher, feature-gated
+//! behind `delta`.
+//!
+//! delta-spark has already done log replay, snapshot resolution, and partition pruning by the
+//! time the scan reaches Comet, so the envelope carries a concrete file list (plus deletion
+//! vector descriptors) and the read path reuses the exact same shared parquet scan builder as
+//! `NativeScan` -- inheriting row-group stats pruning, page-index pruning, and filter pushdown.
+//! Sibling of the kernel-planned handler in `delta_scan.rs`; the two claim different
+//! `type_url`s within the same `ContribScan` envelope.
+
+use std::collections::HashMap;
+
+use datafusion_comet_proto::spark_operator::{
+ ContribScan, DeltaSparkScan, Operator, SparkFilePartition,
+};
+use prost::Message;
+
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::execution::planner::PhysicalPlanner;
+use crate::execution::planner::PlanCreationResult;
+
+/// Type name the JVM-planned Delta contrib claims within the `ContribScan` envelope. The
+/// contrib jar packs a `DeltaSparkScan` with a `type_url` of
+/// `type.googleapis.com/comet.contrib.delta_spark.DeltaSparkScan`; dispatch keys on the
+/// contrib-owned suffix, same convention as the kernel path's `delta_scan.rs`.
+const DELTA_SPARK_SCAN_TYPE_NAME: &str = "comet.contrib.delta_spark.DeltaSparkScan";
+
+/// Contrib entry point for the `OpStruct::ContribScan` dispatcher. Returns `Some(result)` when
+/// the envelope carries a JVM-planned Delta scan, or `None` when the `type_url` belongs to some
+/// other contrib.
+pub(crate) fn try_plan_contrib_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ contrib: &ContribScan,
+) -> Option<PlanCreationResult> {
+ if !contrib.type_url.ends_with(DELTA_SPARK_SCAN_TYPE_NAME) {
+ return None;
+ }
+ Some(
+ DeltaSparkScan::decode(contrib.value.as_slice())
+ .map_err(|e| {
+ GeneralError(format!(
+ "Failed to decode DeltaSparkScan from contrib_scan: {e}"
+ ))
+ })
+ .and_then(|scan| plan_delta_spark_scan(planner, spark_plan, &scan)),
+ )
+}
+
+fn plan_delta_spark_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ scan: &DeltaSparkScan,
+) -> PlanCreationResult {
+ // Delta data files are plain parquet; the read path deliberately reuses
+ // the same shared parquet scan builder as NativeScan so Delta inherits
+ // row-group stats pruning, page-index pruning, and filter pushdown. Only
+ // the file list arrives in Delta-specific form. Note delta_common's
+ // column_mapping_mode is informational in M1: the actual field-id
+ // matching switch is common.use_field_id, same as the Iceberg path.
+ let common = scan
+ .common
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing common data".into()))?;
+
+ let delta_partition = scan
+ .file_partition
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing file_partition".into()))?;
+
+ let spark_partition = SparkFilePartition {
+ partitioned_file: delta_partition
+ .partitioned_file
+ .iter()
+ .map(|f| {
+ f.file
+ .clone()
+ .ok_or_else(|| GeneralError("DeltaSparkPartitionedFile missing inner file".into()))
+ })
+ .collect::<Result<Vec<_>, _>>()?,
+ };
+
+ let (object_store_url, mut files) =
+ planner.prepare_scan_store_and_files(common, &spark_partition)?;
Review Comment:
Added, both sides. At claim time the JVM declines when any two URIs the native side resolves stores for (data files and DV sidecars) share the native store-identity key but differ in userinfo, which is exactly your source/clone container case. The JVM key is a deliberate conservative superset of the native one (it also lowercases host and port and always rewrites s3a to s3), so it can only over-decline, never miss a collision native would hit. The native side got a matching check at store-resolution time, which sees DV paths too; it's deliberately not an extension of the data-file authority check so the legitimate cross-bucket DV shape (data in A, sidecar in B) still claims, which the MinIO suite now proves live. Remaining honest gaps, all narrow: the JVM compares decoded authorities while native parses the url-encoded form, so a userinfo differing only by percent-encoding would pass the gate and hit the native error (not constructible from real container names); the native compar
e uses the username without the password component; and the native check is per partition while the JVM gate is whole-scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4006945812
##########
contrib/delta-spark/README.md:
##########
@@ -0,0 +1,60 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Comet Delta Lake Contrib (experimental)
+
+Native Delta Lake reads for Comet. Delta tables are scanned through Comet's
+existing native Parquet reader, so they get row-group pruning, page-index
+pruning, and filter pushdown, with deletion vectors applied inside the scan.
+
+Support is experimental and explicitly opt-in. Two things are required:
+
+1. This module's jar (`comet-contrib-delta-spark`) on the classpath, alongside
+ `delta-spark`. It is never bundled into `comet-spark`; without it, Comet
+ has no Delta surface at all.
+2. `spark.comet.scan.delta.enabled=true`. The default is `false`, so the jar
+ alone does nothing.
+
+Unsupported tables and features fall back to Spark's reader. See the
+[user guide](https://datafusion.apache.org/comet/user-guide/delta.html)
Review Comment:
> Could the README say the same, or run the full reactor?
Link fixed to `user-guide/latest/delta.html`. The build section now installs `common,spark` from the same checkout immediately before the contrib line, as CI does, and says why.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4125199280
##########
native/core/src/parquet/datetime_rebase.rs:
##########
@@ -0,0 +1,2802 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Per-file datetime calendar-rebase handling for the parquet scan.
+//!
+//! Spark 2.4 and earlier wrote dates and timestamps in the hybrid Julian + Gregorian calendar;
+//! Spark 3.0+ uses the proleptic Gregorian calendar and records the calendar policy of every
+//! file it writes in the parquet footer's key-value metadata (`org.apache.spark.version`,
+//! `org.apache.spark.legacyDateTime`, `org.apache.spark.legacyINT96`,
+//! `org.apache.spark.timeZone`). Spark's reader resolves the rebase policy from EACH FILE's
+//! writer metadata (`DataSourceUtils.datetimeRebaseSpec` / `int96RebaseSpec`) -- the session's
+//! `spark.sql.parquet.datetimeRebaseModeInRead` conf only applies to files whose metadata does
+//! not decide the policy on its own -- so a reader that ignores the metadata silently returns
+//! values shifted by up to ten days for dates before 1582-10-15 (e.g. `1500-01-01` reads as
+//! `1500-01-10`).
+//!
+//! This module mirrors that per-file resolution: [`resolve_file_rebase_policies`] computes the
+//! date / INT64-timestamp / INT96-timestamp policies from a file's arrow schema metadata (the
+//! parquet key-value pairs survive the parquet -> arrow schema conversion), and
+//! [`wrap_datetime_rebase`] wraps the per-file rewritten expressions' column references in a
+//! [`SparkDatetimeRebaseExpr`] that rebases values exactly where that is possible without the
+//! JVM's historical timezone tables (dates always; timestamps for a fixed UTC writer zone) and
+//! refuses -- rather than silently corrupting -- ancient values it cannot rebase. Nested
+//! columns are rebuilt leaf by leaf (struct / list / map / fixed-size list / dictionary), each
+//! leaf under its own policy, with nulls and offsets preserved. Modern values are always the
+//! identity under every policy: from 1582-10-15 onward for dates, and from
+//! [`LAST_SWITCH_JULIAN_TS_SECONDS`] (1900-01-01T00:00:00Z, Spark's
+//! `RebaseDateTime.lastSwitchJulianTs`) onward for timestamps.
+//!
+//! Spark applies `datetimeRebaseSpec` to INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns
+//! and `int96RebaseSpec` to INT96 columns. The two physical types are indistinguishable in the
+//! arrow schema DataFusion hands the expression adapter (both surface as `Timestamp(us, "UTC")`
+//! after INT96 coercion), so Comet's parquet reader factory stamps the file's INT96 leaf
+//! ordinals -- taken from the parquet footer's own `SchemaDescriptor` -- into the key-value
+//! metadata under [`INT96_LEAVES_METADATA_KEY`] before the arrow schema is derived (see
+//! [`stamp_int96_leaves`] and `eager_page_index_reader_factory.rs`), and the adapter attributes
+//! every timestamp leaf to its spec from that stamp. Without a stamp, the two specs are merged:
+//! agreement decides, disagreement degrades to [`RebasePolicy::CheckAncient`].
+//!
+//! The wrapper sits BENEATH the schema adapter's nested narrowing (the struct -> struct convert
+//! that keeps only the requested children), which is what keeps those ordinals physical -- but
+//! it means the wrapper sees every physical child, requested or not. Spark only ever decodes
+//! the requested nested schema, so [`FileRebasePolicies::restrict_to_requested`] marks the
+//! physical leaves the narrowing drops as the identity: an unrequested ancient `s.ts` never
+//! blocks `select s.d`, exactly as in Spark.
+//!
+//! Currently only enabled by the Delta scan arms via
+//! `SparkParquetOptions::rebase_from_file_metadata`, which also carries the session read modes
+//! ([`SessionRebaseModes`], forwarded from the JVM) that decide the policy for files without
+//! Spark writer metadata; the plain NativeScan keeps its documented no-rebase behavior (see
+//! the compatibility guide and issue #5010).
+
+use std::collections::HashMap;
+use std::fmt::{self, Display};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Date32Array, FixedSizeListArray, GenericListArray, MapArray,
+ OffsetSizeTrait, PrimitiveArray, RecordBatch, StructArray,
+};
+use arrow::datatypes::{
+ ArrowPrimitiveType, ArrowTimestampType, DataType, Date32Type, FieldRef, Schema, SchemaRef,
+ TimeUnit, TimestampMicrosecondType, TimestampMillisecondType, TimestampNanosecondType,
+ TimestampSecondType,
+};
+use arrow::error::ArrowError;
+use datafusion::common::tree_node::{Transformed, TreeNode};
+use datafusion::common::{DataFusionError, Result as DataFusionResult};
+use datafusion::physical_expr::expressions::Column;
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::ColumnarValue;
+use parquet::basic::Type as ParquetPhysicalType;
+use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaData};
+use parquet::schema::types::SchemaDescriptor;
+
+use super::name_fold::fold_names;
+use super::parquet_support::field_id;
+
+/// Footer key naming the Spark release that wrote the file; absent for non-Spark writers.
+const SPARK_VERSION_METADATA_KEY: &str = "org.apache.spark.version";
+/// Present (empty value) when the file's dates and INT64 timestamps were written with
+/// `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_DATETIME_KEY: &str = "org.apache.spark.legacyDateTime";
+/// Present (empty value) when the file's INT96 timestamps were written with
+/// `spark.sql.parquet.int96RebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_INT96_KEY: &str = "org.apache.spark.legacyINT96";
+/// The writer session's time zone, stamped alongside either legacy flag.
+const SPARK_TIMEZONE_KEY: &str = "org.apache.spark.timeZone";
+
+/// Key-value metadata entry Comet's parquet reader factory adds to a file's footer metadata
+/// (in memory only, never written back) so the expression adapter can tell INT96 timestamp
+/// columns from INT64 ones after both have been coerced to the same arrow type. Value:
+/// `"<leaf count>:<comma-separated INT96 leaf ordinals>"`, where leaves are the file's
+/// primitive columns in `SchemaDescriptor::columns()` order -- the same depth-first order
+/// parquet-rs assigns arrow leaves, so an arrow-side depth-first walk lines up with it. The
+/// leaf count lets the reader detect a stamp that does not describe the schema it is paired
+/// with (see [`Int96Attribution::from_schema`]).
+pub(crate) const INT96_LEAVES_METADATA_KEY: &str = "comet.int96_leaf_columns";
+
+/// Day of the Gregorian cutover (1582-10-15) as days since the epoch; rebasing is the identity
+/// from this day onward. Same value as Spark's `RebaseDateTime.lastSwitchJulianDay`.
+const LAST_SWITCH_JULIAN_DAY: i32 = -141427;
+
+/// Spark's `RebaseDateTime.lastSwitchJulianTs` (and `lastSwitchGregorianTs`) in seconds since
+/// the epoch: 1900-01-01T00:00:00Z. Spark derives it as the latest switch instant across every
+/// zone in its `julian-gregorian-rebase-micros.json` table (`getLastSwitchTs`, which also
+/// asserts the calendars' difference is zero for every zone from then on): most zones ran on
+/// local mean time before 1900, so the last instant at which rebasing changes a value in ANY
+/// zone is 1900-01-01T00:00:00Z, not the 1582 cutover. `createTimestampRebaseFuncInRead`
+/// under `EXCEPTION` throws exactly for `micros < lastSwitchJulianTs` (after converting
+/// `TIMESTAMP_MILLIS` to micros), and `rebaseJulianToGregorianMicros` is the identity from it
+/// onward in every zone. The value is in seconds so it scales exactly to any timestamp unit.
+pub(crate) const LAST_SWITCH_JULIAN_TS_SECONDS: i64 = -2_208_988_800;
+
+/// The per-century differences between the Julian and proleptic Gregorian calendars, and the
+/// Julian-calendar switch days at which each difference starts to apply. Copied verbatim from
+/// Spark's `RebaseDateTime.julianGregDiffs` / `julianGregDiffSwitchDay` (which Spark generated
+/// from `localRebaseJulianToGregorianDays`); `rebase_julian_to_gregorian_days` must stay
+/// value-for-value equal to Spark's `rebaseJulianToGregorianDays`.
+const JULIAN_GREG_DIFFS: [i32; 14] = [2, 1, 0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, 0];
+const JULIAN_GREG_DIFF_SWITCH_DAY: [i32; 14] = [
+ -719164, -682945, -646420, -609895, -536845, -500320, -463795, -390745, -354220, -317695,
+ -244645, -208120, -171595, -141427,
+];
+
+/// Proleptic-Gregorian days since 1970-01-01 for a nominal civil date, via Howard Hinnant's
+/// `days_from_civil`. `d` may exceed the month's length; the excess rolls into the following
+/// month exactly like `LocalDate.of(y, m, 1).plusDays(d - 1)` in Spark's
+/// `localRebaseJulianToGregorianDays` (how the non-existent proleptic date `1000-02-29`,
+/// valid in the Julian calendar, lands on `1000-03-01`).
+fn days_from_civil(y: i64, m: i64, d: i64) -> i64 {
+ let y = if m <= 2 { y - 1 } else { y };
+ let era = y.div_euclid(400);
+ let yoe = y - era * 400; // [0, 399]
+ let mp = (m + 9) % 12; // [0, 11], March = 0
+ let doy = (153 * mp + 2) / 5 + d - 1;
+ let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
+ era * 146097 + doe - 719468
+}
+
+/// Julian-calendar civil date `(year, month, day)` for a day count since 1970-01-01 that labels
+/// days in the Julian calendar (astronomical year numbering: 1 BCE is year 0). Standard
+/// Julian-day-number conversion (E.G. Richards' algorithm), exact for any day.
+fn julian_day_to_civil(days: i64) -> (i64, i64, i64) {
+ // Integer (noon) Julian Day Number of this civil day: 1970-01-01 is JDN 2440588.
+ let jdn = days + 2_440_588;
+ let f = jdn + 1401;
+ let e = 4 * f + 3;
+ let g = e.rem_euclid(1461) / 4;
+ let h = 5 * g + 2;
+ let day = h.rem_euclid(153) / 5 + 1;
+ let month = (h / 153 + 2).rem_euclid(12) + 1;
+ let year = e.div_euclid(1461) - 4716 + (14 - month) / 12;
+ (year, month, day)
+}
+
+/// Exact port of Spark's `RebaseDateTime.rebaseJulianToGregorianDays`: reinterprets a day count
+/// written in the hybrid Julian + Gregorian calendar as the proleptic Gregorian day count of the
+/// same nominal civil date. Identity for days from 1582-10-15 onward. Days before the tables'
+/// range (before Julian `0001-01-01`) take the calendar-arithmetic path, mirroring Spark's
+/// `localRebaseJulianToGregorianDays` fallback.
+pub(crate) fn rebase_julian_to_gregorian_days(days: i32) -> i32 {
+ if days < JULIAN_GREG_DIFF_SWITCH_DAY[0] {
+ let (y, m, d) = julian_day_to_civil(days as i64);
+ (days_from_civil(y, m, 1) + (d - 1)) as i32
+ } else {
+ // Spark's rebaseDays: linear search from the most recent switch day.
+ let mut i = JULIAN_GREG_DIFF_SWITCH_DAY.len();
+ loop {
+ i -= 1;
+ if i == 0 || days >= JULIAN_GREG_DIFF_SWITCH_DAY[i] {
+ break;
+ }
+ }
+ days + JULIAN_GREG_DIFFS[i]
+ }
+}
+
+/// Timezone strings from `org.apache.spark.timeZone` that denote a fixed zero-offset zone in
+/// both `java.util.TimeZone` and `java.time`. Only for these is timestamp rebasing the pure
+/// nominal-date shift [`SparkDatetimeRebaseExpr::rebase_timestamp_utc`] computes; any other (or
+/// absent) zone needs the JVM's historical timezone tables and stays on the
+/// refuse-ancient-values path.
+const UTC_EQUIVALENT_TIMEZONES: [&str; 6] = ["UTC", "Etc/UTC", "GMT", "Etc/GMT", "Z", "+00:00"];
+
+/// How the writer's session time zone (if recorded) affects timestamp rebasing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum WriterTimeZone {
+ /// A fixed zero-offset zone: rebasing reduces to the exact nominal-date shift.
+ Utc,
+ /// Any other zone, or none recorded (pre-3.0 files): ancient values cannot be rebased
+ /// without the JVM's historical timezone data.
+ OtherOrUnknown,
+}
+
+/// One session-level datetime rebase read mode (a `LegacyBehaviorPolicy` value of
+/// `spark.sql.parquet.datetimeRebaseModeInRead` / `int96RebaseModeInRead`), consulted by
+/// [`resolve_file_rebase_policies`] ONLY for files whose footer metadata does not decide the
+/// policy on its own -- exactly the `getOrElse` fallback in Spark's
+/// `DataSourceUtils.getRebaseSpec`. Files that carry `org.apache.spark.version` ignore these
+/// modes entirely, on every Spark version.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) enum RebaseReadMode {
+ /// Refuse ancient values (Spark raises `SparkUpgradeException`); maps to
+ /// [`RebasePolicy::CheckAncient`]. The default mirrors the conservative posture used
+ /// before the conf was plumbed through (and Spark 3.x's own conf default).
+ #[default]
+ Exception,
+ /// Read values as proleptic Gregorian without rebasing.
+ Corrected,
+ /// Rebase from the hybrid Julian + Gregorian calendar.
+ Legacy,
+}
+
+impl RebaseReadMode {
+ /// Parses a `LegacyBehaviorPolicy` conf value. `SQLConf` validates and upper-cases the
+ /// session conf, but a per-relation `datetimeRebaseMode` option arrives verbatim, so the
+ /// match is case-insensitive. Anything unrecognized -- including the empty string a proto
+ /// producer that predates the field sends -- falls back to [`RebaseReadMode::Exception`],
+ /// which refuses ancient values rather than silently corrupting them.
+ pub(crate) fn from_conf_value(value: &str) -> Self {
+ match value.to_ascii_uppercase().as_str() {
+ "CORRECTED" => RebaseReadMode::Corrected,
+ "LEGACY" => RebaseReadMode::Legacy,
+ _ => RebaseReadMode::Exception,
+ }
+ }
+}
+
+/// The session's effective datetime rebase read modes, one per spec class (INT64
+/// dates/timestamps vs INT96 timestamps), forwarded from the JVM at planning time.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) struct SessionRebaseModes {
+ /// `spark.sql.parquet.datetimeRebaseModeInRead` (or the relation's `datetimeRebaseMode`).
+ pub datetime: RebaseReadMode,
+ /// `spark.sql.parquet.int96RebaseModeInRead` (or the relation's `int96RebaseMode`).
+ pub int96: RebaseReadMode,
+}
+
+/// Calendar policy of one file's date or timestamp columns, resolved from writer metadata the
+/// same way Spark's `DataSourceUtils.getRebaseSpec` resolves it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum RebasePolicy {
+ /// Written in the proleptic Gregorian calendar; values pass through untouched.
+ Corrected,
+ /// Written in the hybrid Julian + Gregorian calendar; values must be rebased.
+ Legacy(WriterTimeZone),
+ /// Policy could not be pinned down (contradictory flags, or a non-Spark writer under the
+ /// `EXCEPTION` read mode): modern values -- identical under either calendar -- pass,
+ /// ancient values raise. Mirrors Spark's `EXCEPTION` behavior (`SparkUpgradeException`).
+ CheckAncient,
+}
+
+/// Which of a file's leaf columns are physically INT96, from the stamp the parquet reader
+/// factory adds under [`INT96_LEAVES_METADATA_KEY`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) enum Int96Attribution {
+ /// No stamp, or a stamp whose leaf count does not match the schema it arrived with: the
+ /// INT64 and INT96 timestamp specs cannot be told apart per column and are merged.
+ Unknown,
+ /// Sorted leaf ordinals (depth-first over the file schema's primitive columns) that are
+ /// INT96; every other timestamp leaf is INT64.
+ Known(Vec<usize>),
+}
+
+impl Int96Attribution {
+ /// Parses the stamp out of `schema`'s metadata and validates its leaf count against the
+ /// schema's own depth-first leaf count, so a stamp that does not describe this schema (a
+ /// crafted footer key, or a cached-metadata mismatch) degrades to [`Self::Unknown`].
+ fn from_schema(schema: &Schema) -> Self {
+ let Some(stamp) = schema.metadata().get(INT96_LEAVES_METADATA_KEY) else {
+ return Int96Attribution::Unknown;
+ };
+ let Some((count, ordinals)) = stamp.split_once(':') else {
+ return Int96Attribution::Unknown;
+ };
+ let schema_leaves: usize = schema
+ .fields()
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ if count.parse::<usize>().ok() != Some(schema_leaves) {
+ return Int96Attribution::Unknown;
+ }
+ let parsed: Option<Vec<usize>> = if ordinals.is_empty() {
+ Some(Vec::new())
+ } else {
+ ordinals
+ .split(',')
+ .map(|o| o.parse::<usize>().ok().filter(|o| *o < schema_leaves))
+ .collect()
+ };
+ match parsed {
+ Some(mut leaves) => {
+ leaves.sort_unstable();
+ Int96Attribution::Known(leaves)
+ }
+ None => Int96Attribution::Unknown,
+ }
+ }
+
+ /// `Some(true)` / `Some(false)` when the leaf is known to be INT96 / INT64, `None` when
+ /// the attribution is unknown.
+ fn is_int96(&self, leaf: usize) -> Option<bool> {
+ match self {
+ Int96Attribution::Unknown => None,
+ Int96Attribution::Known(leaves) => Some(leaves.binary_search(&leaf).is_ok()),
+ }
+ }
+}
+
+/// The [`INT96_LEAVES_METADATA_KEY`] value describing `schema`: its leaf count and the
+/// ordinals of its INT96 primitive columns.
+pub(crate) fn int96_leaf_stamp(schema: &SchemaDescriptor) -> String {
+ let ordinals: Vec<String> = schema
+ .columns()
+ .iter()
+ .enumerate()
+ .filter(|(_, column)| column.physical_type() == ParquetPhysicalType::INT96)
+ .map(|(ordinal, _)| ordinal.to_string())
+ .collect();
+ format!("{}:{}", schema.num_columns(), ordinals.join(","))
+}
+
+/// Returns a copy of `metadata` whose key-value metadata carries the [`int96_leaf_stamp`] of
+/// its own schema, or `None` when it already does (the common case after the first open of a
+/// file, since the caller caches the stamped copy). Any pre-existing entry under the key --
+/// a file cannot legitimately carry one -- is replaced, never trusted. Only the file-level
+/// key-value list changes; row groups and page indexes are carried over as-is. The parquet
+/// API cannot carry a file decryptor, nor `FileMetaData`'s crate-private encryption fields
+/// (encryption algorithm, footer signing key metadata), across this rebuild, so callers must
+/// not stamp opens that supply decryption properties -- and the only consumer, the Delta
+/// scan, declines every encrypted-parquet configuration before planning, so a parquet
+/// modular encryption file never reaches this path with or without those properties.
+pub(crate) fn stamp_int96_leaves(metadata: &ParquetMetaData) -> Option<ParquetMetaData> {
+ let file_metadata = metadata.file_metadata();
+ let stamp = int96_leaf_stamp(file_metadata.schema_descr());
+ let existing = file_metadata
+ .key_value_metadata()
+ .and_then(|kvs| kvs.iter().find(|kv| kv.key == INT96_LEAVES_METADATA_KEY))
+ .and_then(|kv| kv.value.as_deref());
+ if existing == Some(stamp.as_str()) {
+ return None;
+ }
+ let mut key_values: Vec<KeyValue> = file_metadata
+ .key_value_metadata()
+ .map(|kvs| {
+ kvs.iter()
+ .filter(|kv| kv.key != INT96_LEAVES_METADATA_KEY)
+ .cloned()
+ .collect()
+ })
+ .unwrap_or_default();
+ key_values.push(KeyValue::new(INT96_LEAVES_METADATA_KEY.to_string(), stamp));
+ let stamped_file_metadata = FileMetaData::new(
+ file_metadata.version(),
+ file_metadata.num_rows(),
+ file_metadata.created_by().map(str::to_string),
+ Some(key_values),
+ file_metadata.schema_descr_ptr(),
+ file_metadata.column_orders().cloned(),
+ );
+ Some(
+ ParquetMetaData::new(stamped_file_metadata, metadata.row_groups().to_vec())
+ .into_builder()
+ .set_column_index(metadata.column_index().cloned())
+ .set_offset_index(metadata.offset_index().cloned())
+ .build(),
+ )
+}
+
+/// Per-file rebase policies for the three affected column classes, plus the INT96
+/// attribution that selects between the two timestamp specs per leaf.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct FileRebasePolicies {
+ /// `DATE` columns, governed by `org.apache.spark.legacyDateTime` alone.
+ pub date: RebasePolicy,
+ /// INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns: the datetime spec (same
+ /// resolution as `date`), as Spark's `ParquetVectorUpdaterFactory` selects for INT64.
+ pub int64_timestamp: RebasePolicy,
+ /// INT96 columns: the INT96 spec (`org.apache.spark.legacyINT96`, min version 3.1.0).
+ pub int96_timestamp: RebasePolicy,
+ /// Which timestamp leaves are INT96. See [`Int96Attribution`].
+ pub int96_leaves: Int96Attribution,
+ /// Sorted depth-first leaf ordinals -- over the physical file schema, the same ordinals
+ /// `int96_leaves` uses -- that the query does not read: nested children the schema
+ /// adapter's struct narrowing drops before any value leaves the scan. Spark never decodes
+ /// them either, so their policy is the identity whatever the file's calendar. Empty until
+ /// [`Self::restrict_to_requested`] runs (every leaf requested).
+ pub unrequested_leaves: Vec<usize>,
+}
+
+impl FileRebasePolicies {
+ /// True when some policy is not the plain proleptic-Gregorian pass-through, i.e. when the
+ /// per-column wrap in [`wrap_datetime_rebase`] can install anything at all.
+ pub(crate) fn any_rebase_needed(&self) -> bool {
+ self.date != RebasePolicy::Corrected
+ || self.int64_timestamp != RebasePolicy::Corrected
+ || self.int96_timestamp != RebasePolicy::Corrected
+ }
+
+ fn is_requested(&self, leaf: usize) -> bool {
+ self.unrequested_leaves.binary_search(&leaf).is_err()
+ }
+
+ /// The policy of the `Date32` leaf at depth-first ordinal `leaf`: the file's date policy,
+ /// or the identity when the query does not read that leaf.
+ fn date_policy(&self, leaf: usize) -> RebasePolicy {
+ if self.is_requested(leaf) {
+ self.date
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+
+ /// The policy of the timezone-carrying timestamp leaf at depth-first ordinal `leaf`: the
+ /// identity when the query does not read it; otherwise its physical type's spec when the
+ /// attribution is known, or else the two specs merged -- agreement decides, disagreement
+ /// degrades to [`RebasePolicy::CheckAncient`], which still passes every modern value and
+ /// refuses only ancient ones.
+ fn timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ Some(false) => self.int64_timestamp,
+ None if self.int64_timestamp == self.int96_timestamp => self.int64_timestamp,
+ None => RebasePolicy::CheckAncient,
+ }
+ }
+
+ /// These policies with every physical leaf the query does not read marked the identity.
+ /// `requested` pairs each top-level field of `physical_schema` (by position) with the type
+ /// of the logical field the schema adapter narrows it to -- `None` for a column without a
+ /// logical counterpart, whose leaves are left as they are (no expression reads it anyway).
+ /// Nested children pair the way the adapter's struct convert selects them (see
+ /// [`push_unrequested_leaves`]); the INT96 attribution is untouched, since the ordinals
+ /// stay physical. `requested` is parallel to the schema's fields; should a caller pass a
+ /// shorter slice, the trailing columns simply keep every leaf (the safe direction).
+ pub(crate) fn restrict_to_requested(
+ mut self,
+ physical_schema: &Schema,
+ requested: &[Option<&DataType>],
+ case_sensitive: bool,
+ use_field_id: bool,
+ ) -> Self {
+ debug_assert_eq!(requested.len(), physical_schema.fields().len());
+ let matching = FieldMatching {
+ case_sensitive,
+ use_field_id,
+ };
+ let mut next_leaf = 0;
+ let mut unrequested = Vec::new();
+ for (field, requested) in physical_schema.fields().iter().zip(requested) {
+ match requested {
+ Some(logical) => push_unrequested_leaves(
+ field.data_type(),
+ logical,
+ &mut next_leaf,
+ matching,
+ &mut unrequested,
+ ),
+ None => next_leaf += leaf_count(field.data_type()),
+ }
+ }
+ // Emitted in depth-first order, so already sorted for `is_requested`'s binary search.
+ self.unrequested_leaves = unrequested;
+ self
+ }
+}
+
+/// The field-matching rules of the schema adapter's nested narrowing
+/// (`parquet_convert_struct_to_struct`): names fold per `case_sensitive`, and Parquet field ids
+/// select fields when `use_field_id` is set.
+#[derive(Debug, Clone, Copy)]
+struct FieldMatching {
+ case_sensitive: bool,
+ use_field_id: bool,
+}
+
+/// Appends to `out` the depth-first leaf ordinals of `physical` (counting from `next_leaf`,
+/// which advances past every leaf of `physical`) that reading it as `requested` drops.
+///
+/// Recurses through exactly the pairings `parquet_convert_array` narrows, and no others: a
+/// struct child is dropped only when NO requested child selects it by either rule the struct
+/// convert uses -- folded name, or Parquet field id when ids are in play -- and an ambiguous
+/// child (several requested children select it) is kept; `List` pairs with `List` by element
+/// type, and `Map` with a `Map` of the same key ordering by its entries, positionally. Any
+/// other pairing -- a leaf, a `LargeList` / `FixedSizeList` / dictionary, a map whose ordering
+/// differs, or a shape mismatch -- is handed to arrow's cast or passed through whole by the
+/// convert, so it keeps every leaf. Keeping a superset of what the narrowing reads is always
+/// safe (a spurious check at worst); dropping a leaf the narrowing reads would skip its
+/// rebase, so every doubt resolves to "requested".
+fn push_unrequested_leaves(
+ physical: &DataType,
+ requested: &DataType,
+ next_leaf: &mut usize,
+ matching: FieldMatching,
+ out: &mut Vec<usize>,
+) {
+ match (physical, requested) {
+ (DataType::Struct(physical_fields), DataType::Struct(requested_fields)) => {
+ let names: Vec<&str> = physical_fields
+ .iter()
+ .chain(requested_fields.iter())
+ .map(|f| f.name().as_str())
+ .collect();
+ // A fold failure means the names could not be compared at all; keeping every leaf
+ // requested is the safe superset, the same as the pass-through pairings below.
+ let Ok(folded) = fold_names(&names, matching.case_sensitive) else {
+ *next_leaf += leaf_count(physical);
+ return;
+ };
+ let (physical_folded, requested_folded) = folded.split_at(physical_fields.len());
+ for (i, child) in physical_fields.iter().enumerate() {
+ let child_id = if matching.use_field_id {
+ field_id(child)
+ } else {
+ None
+ };
+ let mut selectors = requested_fields.iter().enumerate().filter(|(j, r)| {
+ requested_folded[*j] == physical_folded[i]
+ || (child_id.is_some() && field_id(r) == child_id)
+ });
+ match (selectors.next(), selectors.next()) {
+ (None, _) => {
+ let n = leaf_count(child.data_type());
+ out.extend(*next_leaf..*next_leaf + n);
+ *next_leaf += n;
+ }
+ (Some((_, requested_child)), None) => push_unrequested_leaves(
+ child.data_type(),
+ requested_child.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (Some(_), Some(_)) => *next_leaf += leaf_count(child.data_type()),
+ }
+ }
+ }
+ (DataType::List(physical_item), DataType::List(requested_item)) => push_unrequested_leaves(
+ physical_item.data_type(),
+ requested_item.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (
+ DataType::Map(physical_entries, physical_sorted),
+ DataType::Map(requested_entries, requested_sorted),
+ ) if physical_sorted == requested_sorted => {
+ match (physical_entries.data_type(), requested_entries.data_type()) {
+ (DataType::Struct(physical_kv), DataType::Struct(requested_kv))
+ if physical_kv.len() == requested_kv.len() =>
+ {
+ for (p, r) in physical_kv.iter().zip(requested_kv.iter()) {
+ push_unrequested_leaves(
+ p.data_type(),
+ r.data_type(),
+ next_leaf,
+ matching,
+ out,
+ );
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+}
+
+/// The writer time zone recorded in `metadata`, classified for timestamp rebasing. Mirrors the
+/// `Option(lookupFileMeta(SPARK_TIMEZONE_METADATA_KEY))` lookup Spark's `getRebaseSpec` performs
+/// for every LEGACY resolution, conf-fallback included; Spark substitutes the JVM default zone
+/// when the key is absent (`RebaseSpec.timeZone`), which is unavailable natively, so an absent or
+/// non-UTC zone classifies as [`WriterTimeZone::OtherOrUnknown`] (dates still rebase fully --
+/// the day rebase is zone-free -- while ancient timestamps refuse rather than guess).
+fn writer_time_zone(metadata: &HashMap<String, String>) -> WriterTimeZone {
+ match metadata.get(SPARK_TIMEZONE_KEY) {
+ Some(tz) if UTC_EQUIVALENT_TIMEZONES.contains(&tz.as_str()) => WriterTimeZone::Utc,
+ _ => WriterTimeZone::OtherOrUnknown,
+ }
+}
+
+/// One spec resolution, mirroring Spark's `DataSourceUtils.getRebaseSpec` exactly: a Spark
+/// version below `min_version` (lexicographic comparison, same as the Scala `String.<`) or a
+/// present legacy flag means LEGACY; a Spark version at/after `min_version` without the flag
+/// means CORRECTED; no Spark version at all falls back to `conf_mode`, the session read conf
+/// forwarded from the JVM (`getRebaseSpec`'s `modeByConfig` fallback, its ONLY use of the
+/// conf): CORRECTED passes values through, LEGACY rebases (with the writer zone from the
+/// file's `org.apache.spark.timeZone` key, same lookup as the metadata-driven LEGACY path),
+/// and EXCEPTION refuses ancient values as [`RebasePolicy::CheckAncient`].
+fn resolve_spec(
+ metadata: &HashMap<String, String>,
+ min_version: &str,
+ legacy_key: &str,
+ conf_mode: RebaseReadMode,
+) -> RebasePolicy {
+ match metadata.get(SPARK_VERSION_METADATA_KEY) {
+ None => match conf_mode {
+ RebaseReadMode::Corrected => RebasePolicy::Corrected,
+ RebaseReadMode::Legacy => RebasePolicy::Legacy(writer_time_zone(metadata)),
+ RebaseReadMode::Exception => RebasePolicy::CheckAncient,
+ },
+ Some(version) => {
+ if version.as_str() < min_version || metadata.contains_key(legacy_key) {
+ RebasePolicy::Legacy(writer_time_zone(metadata))
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+ }
+}
+
+/// Resolves the per-file rebase policies from a file's arrow schema: the parquet footer's
+/// key-value pairs in its metadata decide the specs (the datetime spec uses min version
+/// `3.0.0` and the INT96 spec `3.1.0`, matching `DataSourceUtils.datetimeRebaseSpec` /
+/// `int96RebaseSpec`; `session_modes` supplies the per-spec conf fallback for files without
+/// Spark writer metadata), and the reader factory's INT96 stamp -- validated against the
+/// schema's leaf structure -- attributes each timestamp leaf to its spec.
+pub(crate) fn resolve_file_rebase_policies(
+ physical_file_schema: &Schema,
+ session_modes: SessionRebaseModes,
+) -> FileRebasePolicies {
+ let metadata = physical_file_schema.metadata();
+ let datetime_spec = resolve_spec(
+ metadata,
+ "3.0.0",
+ SPARK_LEGACY_DATETIME_KEY,
+ session_modes.datetime,
+ );
+ let int96_spec = resolve_spec(
+ metadata,
+ "3.1.0",
+ SPARK_LEGACY_INT96_KEY,
+ session_modes.int96,
+ );
+ FileRebasePolicies {
+ date: datetime_spec,
+ int64_timestamp: datetime_spec,
+ int96_timestamp: int96_spec,
+ int96_leaves: Int96Attribution::from_schema(physical_file_schema),
+ unrequested_leaves: Vec::new(),
+ }
+}
+
+/// Number of primitive leaves `dt` contains in a depth-first walk -- the same count and order
+/// parquet-rs uses when it maps the file's `SchemaDescriptor` columns onto the arrow schema, so
+/// arrow-side leaf ordinals line up with [`int96_leaf_stamp`]'s.
+fn leaf_count(dt: &DataType) -> usize {
+ match dt {
+ DataType::Struct(fields) => fields.iter().map(|f| leaf_count(f.data_type())).sum(),
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_count(f.data_type()),
+ DataType::Dictionary(_, value) => leaf_count(value),
+ DataType::RunEndEncoded(_, value) => leaf_count(value.data_type()),
+ DataType::Union(fields, _) => fields.iter().map(|(_, f)| leaf_count(f.data_type())).sum(),
+ _ => 1,
+ }
+}
+
+/// Appends the policy of every leaf of `dt`, in depth-first order, to `out`, consuming leaf
+/// ordinals from `next_leaf` (exactly [`leaf_count`] of them). Only `Date32` and
+/// timezone-carrying timestamps have a policy to apply, and only when the query reads the
+/// leaf; timezone-free timestamps are `TIMESTAMP_NTZ`, which Spark never rebases, and every
+/// other leaf is the identity ([`RebasePolicy::Corrected`]).
+fn leaf_policies(
+ dt: &DataType,
+ next_leaf: &mut usize,
+ policies: &FileRebasePolicies,
+ out: &mut Vec<RebasePolicy>,
+) {
+ match dt {
+ DataType::Date32 => {
+ out.push(policies.date_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Timestamp(_, Some(_)) => {
Review Comment:
[P2] Respect the requested `TIMESTAMP_NTZ` type when selecting timestamp rebase policies. On Spark 4.0/4.1, an opted-in Delta scan requesting `TIMESTAMP_NTZ` from an INT96 file containing `1800-01-01`, with `spark.sql.parquet.int96RebaseModeInRead=EXCEPTION`, should return the value unchanged. Spark explicitly skips rebasing for requested NTZ timestamps, and Delta’s protocol permits INT96 storage for them. Here, INT96 is physically represented as `Timestamp(_, Some("UTC"))`, so this branch applies the ancient-value check beneath the NTZ conversion and fails the query. Please use the physical-to-requested leaf pairing to suppress timestamp rebasing for NTZ requests, while preserving DATE-to-NTZ rebasing, and add this regression case.
Evidence: Executed a disposable test appended to an otherwise unchanged copy of the head’s `parquet_exec.rs`: `review_ntz_int96_ignores_rebase_mode` in `/tmp/comet5365-ntz-review-l5jqmu1i/native/core/src/parquet/parquet_exec.rs`. It writes metadata-free `message m { required int96 ts; }` with INT96 words `(0, 0, 2378497)`, then calls the current scan builder with requested `Timestamp(Microsecond, None)`, LTZ-to-NTZ conversion enabled and metadata rebasing enabled. `CORRECTED` returns `1800-01-01T00:00:00`; `EXCEPTION` returns `Native scan cannot rebase ancient values in column 'ts'`. Stock Spark 4.1.3, reading the same `/tmp/comet5365-review-int96-ntz.parquet` with `.schema("ts TIMESTAMP_NTZ")` and the EXCEPTION setting, returned `[1800-01-01T00:00]`. Spark 4.0.4/4.1.3 `ParquetVectorUpdaterFactory` selects `BinaryToSQLTimestampUpdater` without rebasing for this requested type. Delta 4.3.1’s `DeltaParquetFileFormat` forwards the requested schema and options to that reader. Full
Comet/Delta/JNI execution was not run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-6110659758
Thanks for your patience @dwsmith1983. I am reviewing this one now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4246904919
##########
docs/source/user-guide/latest/delta.md:
##########
@@ -0,0 +1,72 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Delta Lake (experimental)
+
+Comet can execute DSv1 Delta Lake table scans natively. Reads planned by
+delta-spark run through Comet's native Parquet scan, inheriting row-group
+pruning, page-index pruning, and filter pushdown, with deletion vectors
+applied inside the scan.
+
+Support is experimental and explicitly opt-in. Two things are required:
+
+1. The `comet-contrib-delta-spark` contrib jar on the classpath, alongside
Review Comment:
> Could this line say the jar has to be built from source for now and point at the README's build steps?
Done in 28d0ce726. It says no release publishes the jar yet and links the README's build section, and the toctree entry reads "Delta Lake (experimental)".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-6125028931
The Delta S3 job failure (https://github.com/apache/datafusion-comet/actions/runs/38233957666/job/114789734253) is the MinIO image, not this PR: Docker Hub no longer serves `minio/minio`, which `CometS3TestBase` pulls (#6884). #6887 moves the test base to another image and fixes two region gaps the S3 suites hit once the image pulls. That needs to merge first, and I'll merge main into this branch after it does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3826290502
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,587 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::ParquetAccessPlan;
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::ObjectStoreExt;
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::PageIndexPolicy;
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::parquet::parquet_support::prepare_object_store_with_configs;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
Review Comment:
Two layers now. At planning, `spark.comet.scan.deltaNative.dv.maxDeletedRowsPerFile` (default 1M) declines pathological DVs before they reach native; the bound is cardinality-based so it's deliberately pessimistic, a contiguous delete declines the same as an alternating one, but run structure isn't knowable without decoding bitmaps on the driver, and the knob makes it recoverable. Natively, the exact selector bytes (derived from `size_of::<RowSelector>()`) are reserved against the memory pool, released with the plan, with multi-file accumulation and release-on-failure covered by tests. I looked at building the access plan at file-open time as you suggested, but DataFusion 54.1's ParquetOpener reads the plan out of the file extensions in `create_initial_plan` with no hook to compute it lazily, so that needs an upstream API first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3826302523
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,587 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::ParquetAccessPlan;
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::ObjectStoreExt;
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::PageIndexPolicy;
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::parquet::parquet_support::prepare_object_store_with_configs;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
+ }
+ cursor = row + 1;
+ }
+ if deleted_in_group == num_rows && num_rows > 0 {
+ plan.skip(idx);
+ } else if deleted_in_group > 0 {
+ if group_end > cursor {
+ selectors.push(RowSelector::select((group_end - cursor) as usize));
+ }
+ plan.scan_selection(idx, RowSelection::from(selectors));
+ }
+ group_start = group_end;
+ }
+ // A deleted index beyond the file's total row count means the DV does not
+ // belong to this file (stale or corrupted metadata); silently dropping it
+ // would under-apply deletions.
+ if let Some(&row) = deleted_iter.peek() {
+ return Err(GeneralError(format!(
+ "Deletion vector marks row {row} but the file only has {group_start} rows"
+ )));
+ }
+ Ok(plan)
+}
+
+/// One data file plus everything needed to apply its deletion vector. The
+/// file's size comes from `file.object_meta.size` (built by the planner from
+/// the proto's `file_size`).
+pub struct DvScanFile {
+ pub file: PartitionedFile,
+ /// Full URL of the data file (proto `file_path`).
+ pub file_path: String,
+ pub dv: Option<DeltaSparkDvDescriptor>,
+}
+
+/// Upper bound on concurrent DV-blob and footer fetches per partition. Both
+/// are small ranged reads, so a modest fan-out hides object-store latency
+/// without flooding the store client.
+const DV_FETCH_CONCURRENCY: usize = 8;
+
+/// Called via `block_on` at plan-creation time on the executor task: DV blobs
+/// are small ranged reads and footers are needed to learn row-group
+/// boundaries. Files are fetched concurrently (bounded by
+/// [`DV_FETCH_CONCURRENCY`]) with input order preserved. Footer fetches go
+/// through the scan's shared FileMetadataCache, so the scan's subsequent open
+/// of the same file is served from cache. That reuse relies on each input
+/// [`PartitionedFile`] being returned as-is (only `with_extension` applied),
+/// never rebuilt: the cache entry is keyed by this exact `object_meta` and the
+/// scan later looks it up through the same struct.
+pub async fn attach_access_plans(
+ runtime_env: Arc<RuntimeEnv>,
+ object_store_options: &HashMap<String, String>,
+ files: Vec<DvScanFile>,
+) -> Result<Vec<PartitionedFile>, ExecutionError> {
+ futures::stream::iter(files)
+ .map(|scan_file| {
+ attach_access_plan(Arc::clone(&runtime_env), object_store_options, scan_file)
+ })
+ .buffered(DV_FETCH_CONCURRENCY)
+ .try_collect()
+ .await
+}
+
+/// Resolve one file's deletion vector into an attached [`ParquetAccessPlan`];
+/// files without a DV pass through untouched.
+async fn attach_access_plan(
+ runtime_env: Arc<RuntimeEnv>,
+ object_store_options: &HashMap<String, String>,
+ scan_file: DvScanFile,
+) -> Result<PartitionedFile, ExecutionError> {
+ let DvScanFile {
+ file,
+ file_path,
+ dv,
+ } = scan_file;
+ let dv = match dv {
+ Some(dv) => dv,
+ None => return Ok(file),
+ };
Review Comment:
Fixed: the canonical zero-cardinality, zero-size descriptor now short-circuits to a pass-through file before any bitmap read or footer fetch. Added to the existing `attach_access_plans` test as a fourth file asserting no access plan, preserved order, and no footer fetched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5379622807
Agreed on keeping it simple. The conf is now spark.comet.scan.delta.enabled (plus spark.comet.scan.delta.dv.maxDeletedRowsPerFile), so there's one flag to enable Delta scans, and the kernel path can add its own experimental key later. Docs updated. Fixes for the three open threads are pushed as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5389049677
### [P2] Also handle plain long-form S3 bucket credentials
The long-form JCEKS cases are fixed, but the same compatibility gap remains for **plain configuration values** at `514f71e0`.
With `fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` and only these plain keys configured:
```text
fs.s3a.bucket.mybucket.fs.s3a.access.key
fs.s3a.bucket.mybucket.fs.s3a.secret.key
```
[`bucketCredentialAliasReason`](https://github.com/apache/datafusion-comet/blob/514f71e0d09f4b0e85964014afae17363fa2dba1/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala#L712-L713) returns `None` because no credential-provider path is set. Hadoop resolves these keys, but `NativeConfig` forwards them unchanged and the [native lookup](https://github.com/apache/datafusion-comet/blob/514f71e0d09f4b0e85964014afae17363fa2dba1/native/core/src/parquet/objectstore/s3.rs#L299-L318) reads only short bucket keys and globals. The native Simple provider therefore has no credentials. If different plain global credentials are also configured, Hadoop uses the long bucket aliases while native uses those globals instead. Claiming the Delta scan exposes this mismatch to reads that previously used Hadoop successfully.
I verified both cases with actual Hadoop 3.4.1 `SimpleAWSCredentialsProvider` and the unchanged current admission/extraction methods, before and after `S3AUtils.propagateBucketOptions`, plus exact-source Rust lookup tests. Short-only and global-only controls agree. This was not a full Comet/JNI scan or a live S3 request.
Could we conservatively decline these configurations, or normalize the effective credentials using Hadoop's precedence, and add plain-long-only and global-shadowing regressions?
This is a remaining case of the [existing S3-authentication P2](https://github.com/apache/datafusion-comet/pull/5365#discussion_r3835076674), posted here for visibility rather than as another inline thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3886368860
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.ListBuffer
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable. Only called when
+ * [[isDeltaScan]] is true. `scanHelper` is the [[CometScanExec]] built to drive
+ * [[CometDeltaNativeScan.convert]] on a claim, reused for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`: it runs at most
+ // once per claim attempt, and not at all for the common non-DV-shape case.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for SSE-C
+ // before the credential-divergence gates below, which do not otherwise notice this table is
+ // readable through Hadoop only because Hadoop's request factory attaches the customer key
+ // native never learns about.
+ val sseCReason = sseCustomerKeyReason(hadoopConf, dataFileUris ++ dvUris)
+ if (sseCReason.isDefined) {
+ return sseCReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason = s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The three per-bucket base keys where Hadoop's own effective value CAN differ from
+ * short-then-global: `S3AUtils#lookupPassword` additionally honors the long bucket form
+ * (`fs.s3a.bucket.B.fs.s3a.<key>`) and Hadoop credential providers (JCEKS et al.) for exactly
+ * these three, and only these three -- every other `fs.s3a` option instead flows through
+ * `S3AUtils#propagateBucketOptions`, which folds the long form into an unread key
+ * (`fs.s3a.fs.s3a.<key>`) and leaves Hadoop's own effective value at short-then-global already,
+ * identical to native's.
+ */
+ private val CredentialValueKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ /**
+ * Every OTHER per-bucket `fs.s3a.*` base key native's S3 client's `get_config` (s3.rs)
+ * resolves, verified directly against its call sites: `extract_s3_config_options`
+ * (endpoint.region, path.style.access, endpoint, requester.pays.enabled),
+ * `lookup_provider_class` (the Comet-specific credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name). Every one of
+ * these resolves short-bucket-then-global inside `get_config` -- s3.rs never reads the long
+ * bucket form. Literal strings, not the [[AwsCredentialsProviderKey]] /
+ * [[AssumedRoleCredentialsProviderKey]] vals declared below, purely to avoid a forward
+ * reference inside this `object` body; kept textually identical to those two constants.
+ */
+ private val OtherS3ConfigKeys: Seq[String] = Seq(
+ "fs.s3a.aws.credentials.provider",
+ "fs.s3a.assumed.role.arn",
+ "fs.s3a.assumed.role.session.name",
+ "fs.s3a.assumed.role.credentials.provider",
+ "fs.s3a.endpoint",
+ "fs.s3a.endpoint.region",
+ "fs.s3a.path.style.access",
+ "fs.s3a.requester.pays.enabled",
+ "fs.s3a.comet.credential.provider.class")
+
+ private val AllS3ConfigKeys: Seq[String] = CredentialValueKeys ++ OtherS3ConfigKeys
+
+ /**
+ * The short-bucket-then-global value resolved for `baseKey` under `bucket` from `hadoopConf`,
+ * skipping an empty value at either alias exactly like [[plainValue]]. Used on a
+ * [[propagateBucketOptions]] result, this is (for the non-credential keys in
+ * [[OtherS3ConfigKeys]]) Hadoop's own effective value, since real Hadoop reads those options as
+ * plain `Configuration#get(baseKey)` calls against its propagated conf, which already folded
+ * any per-bucket override into `baseKey` itself -- so re-checking the (untouched) short bucket
+ * key here first is a no-op, not a second resolution mechanism, and gives the identical value
+ * `Configuration#get(baseKey)` alone would. Also used by [[shortThenGlobalOrReason]] to read
+ * provider-CLASS strings from the original conf for name-support validation; the earlier
+ * [[s3ConfigDivergenceReason]] gate (which uses [[nativeShortThenGlobal]], not this function,
+ * to model native's actual resolution) already runs ahead of that check and would catch any
+ * divergence this skip-empty read models imprecisely. NEVER used to compute native's own
+ * effective value any more -- see [[nativeShortThenGlobal]] for that.
+ */
+ private def shortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket` from the ORIGINAL, unpropagated `hadoopConf` -- `NativeConfig
+ * .extractObjectStoreOptions` forwards `Configuration#get`'s substituted value for every
+ * `fs.s3a.*` entry with no bucket-option propagation step of its own, so the original conf is
+ * the right input here. Unlike [[shortThenGlobal]]/[[plainValue]], this mirrors `get_config`
+ * faithfully: PRESENCE of the short-bucket key alone -- never its emptiness -- decides whether
+ * native falls back to the global key (`get_config` is a plain `HashMap::get`, which returns
+ * `Some` for a key explicitly set to `""`), so an explicitly empty or whitespace-only
+ * short-bucket value resolves to `Some("")` here and never falls through to global -- the
+ * OPPOSITE of Hadoop's own `getPassword`/`lookupPassword` semantics (see
+ * [[hadoopCredentialEffective]]), which treat empty as absent and keep trying the next alias.
+ * The ONLY function used to compute native's effective value in [[s3KeyDivergenceReason]].
+ *
+ * Deliberately does NOT apply `get_config_trimmed`'s `.trim()` here: [[s3KeyDivergenceReason]]
+ * trims both this value and Hadoop's effective value together, symmetrically, at the point they
+ * are compared, rather than one-sidedly here -- trimming only the native side would flag a
+ * spurious divergence for a value neither side's whitespace actually changes the behavior of
+ * once each side's own downstream parsing normalizes it (e.g. Hadoop's own multi-line
+ * `fs.s3a.aws.credentials.provider` default, which both Hadoop and native additionally trim per
+ * comma-separated entry after splitting), while a one-sided trim would make an
+ * otherwise-identical default value look diverged for every bucket, never claiming natively at
+ * all.
+ */
+ private def nativeShortThenGlobal(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ Option(hadoopConf.get(shortKey)).orElse(Option(hadoopConf.get(baseKey)))
+ }
+
+ /**
+ * Faithful in-memory replica of `S3AUtils#propagateBucketOptions` (`hadoop-aws`), which
+ * `S3AFileSystem#initialize` calls FIRST, before any option or credential is read:
+ * `Configuration conf = propagateBucketOptions(originalConf, bucket); ...; setConf(conf);` --
+ * every subsequent `conf.get`/`getPassword` call in that filesystem instance, including
+ * `${...}` variable substitution, resolves against this propagated view, not the original conf.
+ * `hadoop-aws` is not on this module's runtime classpath (see the string-literal-keys note
+ * above), so `S3AUtils#propagateBucketOptions` cannot be called directly; this reproduces its
+ * logic verbatim using only `hadoop-common`'s `Configuration`:
+ * {{{
+ * public static Configuration propagateBucketOptions(Configuration source, String bucket) {
+ * final String bucketPrefix = FS_S3A_BUCKET_PREFIX + bucket + '.';
+ * final Configuration dest = new Configuration(source);
+ * for (Map.Entry<String, String> entry : source) {
+ * final String key = entry.getKey();
+ * final String value = entry.getValue(); // the (unexpanded) value
+ * if (!key.startsWith(bucketPrefix) || bucketPrefix.equals(key)) continue;
+ * final String stripped = key.substring(bucketPrefix.length());
+ * if (stripped.startsWith("bucket.") || "impl".equals(stripped)) {
+ * // ignored
+ * } else {
+ * final String generic = FS_S3A_PREFIX + stripped;
+ * dest.set(generic, value, ...); // overwrites any existing global value
+ * }
+ * }
+ * return dest;
+ * }
+ * }}}
+ * Note the LONG bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) folds to an unread
+ * `fs.s3a.fs.s3a.<key>` key here too, exactly like the real method -- `stripped` already starts
+ * with `fs.s3a.` in that case, so prepending `fs.s3a.` again produces a key nothing ever reads.
+ */
+ private def propagateBucketOptions(hadoopConf: Configuration, bucket: String): Configuration = {
+ val bucketPrefix = s"fs.s3a.bucket.$bucket."
+ val dest = new Configuration(hadoopConf)
+ hadoopConf.iterator().asScala.foreach { entry =>
+ val key = entry.getKey
+ if (key.startsWith(bucketPrefix) && key != bucketPrefix) {
+ val stripped = key.substring(bucketPrefix.length)
+ if (!stripped.startsWith("bucket.") && stripped != "impl") {
+ dest.set(s"fs.s3a.$stripped", entry.getValue)
+ }
+ }
+ }
+ dest
+ }
+
+ /**
+ * Canonical and deprecated Hadoop S3A encryption-algorithm config keys, verified via `javap`
+ * against `hadoop-aws` 3.3.4's `org.apache.hadoop.fs.s3a.Constants`: `S3_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.encryption.algorithm"` (canonical) and `SERVER_SIDE_ENCRYPTION_ALGORITHM =
+ * "fs.s3a.server-side-encryption-algorithm"` (DEPRECATED -- note the hyphen before "algorithm",
+ * unlike the corresponding `*.key` constants below, which both use a `.key` suffix).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, same
+ * rationale as [[HadoopCredentialProviderPathKey]].
+ */
+ private val S3EncryptionAlgorithmKey = "fs.s3a.encryption.algorithm"
+ private val DeprecatedS3EncryptionAlgorithmKey = "fs.s3a.server-side-encryption-algorithm"
+
+ /**
+ * The literal string `S3AEncryptionMethods.SSE_C.getMethod()` returns, verified via `javap`
+ * against `hadoop-aws` 3.3.4: `S3AEncryptionMethods#getMethod(String)` parses the configured
+ * algorithm with `values().find(_.getMethod.equalsIgnoreCase(algorithm))`, so this gate's
+ * comparison is case-insensitive too, matching Hadoop's own parsing rather than being stricter
+ * or looser than it.
+ */
+ private val SseCustomerKeyAlgorithm = "SSE-C"
+
+ /**
+ * `bucket`'s effective encryption-algorithm key and value under `hadoopConf`, or `None` when
+ * neither the canonical nor deprecated key is set anywhere consulted. Mirrors
+ * `S3AUtils#buildEncryptionSecrets`'s real resolution order (verified via `javap`): it first
+ * tries `lookupBucketSecret`, which checks ONLY the bucket-scoped SHORT form of each key
+ * (canonical before deprecated), then falls back to a BUCKET-AGNOSTIC global `lookupPassword`
+ * call for each key (canonical before deprecated) -- confirmed from the bytecode, which passes
+ * a literal `null` bucket argument at that second tier. Unlike [[s3KeyDivergenceReason]]'s
+ * [[CredentialValueKeys]] handling, the LONG bucket form
+ * (`fs.s3a.bucket.B.fs.s3a.encryption.algorithm`) is genuinely never consulted for this key:
+ * running it through [[propagateBucketOptions]] would only fold it into the unread
+ * `fs.s3a.fs.s3a.encryption.algorithm` key documented there, matching Hadoop's own behavior, so
+ * the global tier here reads `hadoopConf` directly rather than a propagated replica.
+ *
+ * The canonical-vs-deprecated distinction below is frequently moot in practice: `hadoop-aws`'s
+ * `S3AFileSystem.addDeprecatedKeys()` statically registers `fs.s3a.server-side-encryption-*` as
+ * `Configuration`-level deprecated aliases of `fs.s3a.encryption.*` (verified via `javap`), a
+ * registration that lives in a static field on Hadoop's `Configuration` class -- process-wide
+ * once `S3AFileSystem`'s class has loaded anywhere in the JVM, which a real scan has always
+ * already done by the time this gate runs, since reading the S3 table at all requires loading
+ * that class. Once active, `Configuration#get` resolves either literal key to the identical
+ * value transparently, making the two `plainValue`/`bucketShort` reads below redundant (but
+ * harmless) for that case; they remain the operative path only when nothing else in the process
+ * has loaded `S3AFileSystem` yet.
+ *
+ * Deliberately does NOT walk the Hadoop-credential-provider (JCEKS) path
+ * `buildEncryptionSecrets` falls back to when neither key is set in plain conf: unlike the
+ * actual key MATERIAL (`fs.s3a.encryption.key`/`fs.s3a.server-side-encryption.key`, never read
+ * here at all -- this gate only needs the ALGORITHM to decide), the algorithm NAME is not
+ * credential-sensitive data, so storing it in a keystore is not a realistic Hadoop deployment
+ * pattern. Documented, narrow limitation: a configuration that stores the algorithm name ONLY
+ * in a credential provider would under-decline here.
+ */
+ private def effectiveEncryptionAlgorithm(
+ hadoopConf: Configuration,
+ bucket: String): Option[(String, String)] = {
+ def bucketShort(baseKey: String): Option[(String, String)] =
+ plainValue(hadoopConf, s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a."))
+ .map(baseKey -> _)
+
+ bucketShort(S3EncryptionAlgorithmKey)
+ .orElse(bucketShort(DeprecatedS3EncryptionAlgorithmKey))
+ .orElse(plainValue(hadoopConf, S3EncryptionAlgorithmKey).map(S3EncryptionAlgorithmKey -> _))
+ .orElse(plainValue(hadoopConf, DeprecatedS3EncryptionAlgorithmKey)
+ .map(DeprecatedS3EncryptionAlgorithmKey -> _))
+ }
+
+ private def sseCustomerKeyDeclineReason(bucket: String, algorithmKey: String): String =
+ s"Native Delta scan does not support $algorithmKey=$SseCustomerKeyAlgorithm for $bucket " +
+ "(SSE-C requires the customer-provided encryption key on every S3 GET/HEAD request; " +
+ "the native S3 client's extract_s3_config_options forwards none of Hadoop's " +
+ "fs.s3a.encryption.*/fs.s3a.server-side-encryption* options, so requests would go out " +
+ "without the required x-amz-server-side-encryption-customer-key* headers and fail, where " +
+ "Hadoop's own reader -- whose request factory attaches the key -- succeeds)"
+
+ /**
+ * First reason any bucket among `uris` is configured for SSE-C (Server-Side Encryption with
+ * Customer-Provided Keys), or `None` when claimable. Scoped to SSE-C only: SSE-S3, SSE-KMS, and
+ * DSSE-KMS are all decrypted transparently by S3 on GET/HEAD once the caller has read/decrypt
+ * permission -- no request header is required, so native's silent failure to forward those
+ * options costs nothing on the read path this gate protects. SSE-C is different in kind, not
+ * degree: S3 rejects a GET/HEAD for an SSE-C object outright (400 Bad Request) unless the
+ * customer key is resent as a request header on every call, so a native scan that never learns
+ * the key cannot succeed at all, where Hadoop's own reader (which does attach it) would.
+ * Client-side encryption (CSE-KMS/CSE-CUSTOM) is a different mechanism entirely -- it
+ * encrypts/decrypts object bytes locally rather than gating the S3 request itself -- and is out
+ * of scope for this HTTP-header gate. Never interpolates a resolved key value, only key names,
+ * the bucket, and the (non-secret) algorithm name.
+ */
+ private[delta] def sseCustomerKeyReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ effectiveEncryptionAlgorithm(hadoopConf, bucket).collect {
+ case (key, value) if value.equalsIgnoreCase(SseCustomerKeyAlgorithm) =>
Review Comment:
Covered by the allowlist change: CSE-KMS, CSE-CUSTOM, and any unknown algorithm decline, server-side ones stay claimable. Two adjacent things got picked up in the same pass: the algorithm can live in a JCEKS keystore so it resolves through the provider machinery now, and there's a proxy gate since native has no proxy support at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5376691586
@schenksj Let’s see how it goes. For now, I see the `delta-spark`-based implementation as the most practical approach: it builds on mature Delta planning while allowing Comet to reuse its optimized native Parquet reader. Longer term, I’m also excited about `delta-kernel-rs` as a shared foundation for native Delta integrations, and I've also heard that the Delta community is also converging on the Rust implementation.
In terms of your concern, I think we should aim to keep the user-facing configuration simple, perhaps with one flag to enable Delta scans and another to opt into an experimental Rust-kernel-backed path. Ideally, both approaches would share as much integration and testing infrastructure as possible.
Really appreciate all your work on this! We’re planning to move quickly with the current `delta-spark` integration and evaluate it against some very large-scale production workloads. We also plan to evaluate the `delta-kernel-rs`-based approach in the future, and I’d love to collaborate on your series and take on some work to move the Rust-based reader forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3827550396
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ val common = commonBuilder.build()
+ val tableRoot = relation.location.rootPaths.head.toString
+ val deltaCommon = OperatorOuterClass.DeltaSparkScanCommon
+ .newBuilder()
+ .setTableRoot(tableRoot)
+ .setColumnMappingMode(columnMappingMode(scanExec))
+ .setSourceKey(DeltaPlanDataInjector.sourceKey(tableRoot, common))
+ .build()
+ val deltaScan = OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaCommon)
+ Operator
+ .newBuilder()
+ .setPlanId(scanExec.id)
+ .setContribScan(DeltaSparkScanEnvelope.pack(deltaScan.build()))
+ .build()
+ }
+ }
+
+ /**
+ * Harvest subquery-bearing predicates for this scan from its covering FilterExec. Spark 3.x
+ * strips them from a scan's `dataFilters` at planning (`FileSourceStrategy` routes them to the
+ * post-scan filter only), while Spark 4.x keeps them in `dataFilters`. Collecting them here at
+ * claim time gives the execution-time resolve-and-push path the same inputs on every Spark
+ * version; the dedup keeps Spark 4.x from carrying duplicates.
+ */
+ def subqueryFiltersFromParent(
+ plan: org.apache.spark.sql.execution.SparkPlan,
+ scanExec: FileSourceScanExec): Seq[org.apache.spark.sql.catalyst.expressions.Expression] = {
+ import org.apache.spark.sql.catalyst.expressions.{PlanExpression, SubqueryExpression}
+ // Nearest FilterExec above the scan (the DV shape interposes nodes between them, so do
+ // not require a direct parent-child edge). Safety comes from the reference guard, not the
+ // plan walk: only conjuncts expressed directly over the scan's own output attributes
+ // survive, so filters from other branches or over aliased projections contribute nothing.
+ val filtersAboveScan = plan.collect {
+ case f: org.apache.spark.sql.execution.FilterExec if f.find(_ eq scanExec).isDefined => f
+ }
+ filtersAboveScan.lastOption
+ .map { f =>
+ splitConjunctivePredicates(f.condition)
+ .filter(_.references.subsetOf(scanExec.outputSet))
Review Comment:
**[P1] Also stop at nondeterministic projections**
The LIMIT/TopN case is fixed at `7e09e04f`, but `spineToScan` still crosses every `ProjectExec`. A deterministic conjunct does not commute with a nondeterministic projection. For a single-file Delta table `t` containing IDs 0–4:
```sql
SELECT id
FROM (SELECT id, monotonically_increasing_id() AS seq FROM t) q
WHERE id > (SELECT max(id) FROM range(1)) AND seq = 1
```
An isolated probe using the unchanged current-head harvesting method on a real Spark 4.0.2 / Delta 4.0.0 physical plan harvests `id > scalar-subquery`; applying that resolved predicate immediately above the scan changes the result from ID 1 to ID 2. With `spark.comet.parquet.rowFilterPushdown.enabled=true`, the native path can perform that same early filtering. Keeping the covering filter does not restore the sequence values assigned to the surviving rows. This was an exact-method/physical-plan probe, not a full Comet/JNI run.
Could we require deterministic projection expressions before crossing a `ProjectExec`, as Spark's predicate-pushdown rule does, and add this regression?
##########
native/core/src/execution/planner/delta_spark_scan.rs:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! JVM-planned Delta handler for the generic `OpStruct::ContribScan` dispatcher, feature-gated
+//! behind `delta`.
+//!
+//! delta-spark has already done log replay, snapshot resolution, and partition pruning by the
+//! time the scan reaches Comet, so the envelope carries a concrete file list (plus deletion
+//! vector descriptors) and the read path reuses the exact same shared parquet scan builder as
+//! `NativeScan` -- inheriting row-group stats pruning, page-index pruning, and filter pushdown.
+//! Sibling of the kernel-planned handler in `delta_scan.rs`; the two claim different
+//! `type_url`s within the same `ContribScan` envelope.
+
+use std::collections::HashMap;
+
+use datafusion_comet_proto::spark_operator::{
+ ContribScan, DeltaSparkScan, Operator, SparkFilePartition,
+};
+use prost::Message;
+
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::execution::planner::PhysicalPlanner;
+use crate::execution::planner::PlanCreationResult;
+
+/// Type name the JVM-planned Delta contrib claims within the `ContribScan` envelope. The
+/// contrib jar packs a `DeltaSparkScan` with a `type_url` of
+/// `type.googleapis.com/comet.contrib.delta_spark.DeltaSparkScan`; dispatch keys on the
+/// contrib-owned suffix, same convention as the kernel path's `delta_scan.rs`.
+const DELTA_SPARK_SCAN_TYPE_NAME: &str = "comet.contrib.delta_spark.DeltaSparkScan";
+
+/// Contrib entry point for the `OpStruct::ContribScan` dispatcher. Returns `Some(result)` when
+/// the envelope carries a JVM-planned Delta scan, or `None` when the `type_url` belongs to some
+/// other contrib.
+pub(crate) fn try_plan_contrib_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ contrib: &ContribScan,
+) -> Option<PlanCreationResult> {
+ if !contrib.type_url.ends_with(DELTA_SPARK_SCAN_TYPE_NAME) {
+ return None;
+ }
+ Some(
+ DeltaSparkScan::decode(contrib.value.as_slice())
+ .map_err(|e| {
+ GeneralError(format!(
+ "Failed to decode DeltaSparkScan from contrib_scan: {e}"
+ ))
+ })
+ .and_then(|scan| plan_delta_spark_scan(planner, spark_plan, &scan)),
+ )
+}
+
+fn plan_delta_spark_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ scan: &DeltaSparkScan,
+) -> PlanCreationResult {
+ // Delta data files are plain parquet; the read path deliberately reuses
+ // the same shared parquet scan builder as NativeScan so Delta inherits
+ // row-group stats pruning, page-index pruning, and filter pushdown. Only
+ // the file list arrives in Delta-specific form. Note delta_common's
+ // column_mapping_mode is informational in M1: the actual field-id
+ // matching switch is common.use_field_id, same as the Iceberg path.
+ let common = scan
+ .common
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing common data".into()))?;
+
+ let delta_partition = scan
+ .file_partition
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing file_partition".into()))?;
+
+ let spark_partition = SparkFilePartition {
+ partitioned_file: delta_partition
+ .partitioned_file
+ .iter()
+ .map(|f| {
+ f.file
+ .clone()
+ .ok_or_else(|| GeneralError("DeltaSparkPartitionedFile missing inner file".into()))
+ })
+ .collect::<Result<Vec<_>, _>>()?,
+ };
+
+ let (object_store_url, mut files) =
+ planner.prepare_scan_store_and_files(common, &spark_partition)?;
Review Comment:
**[P1] Decline the residual cross-container DV case before claiming the scan**
The mixed-data-file guard fixes the original case. For the residual you identified, could we add a Delta-side decline guard until store identity is fixed? A shallow clone from `abfss://source@account.dfs.core.windows.net/...` to `abfss://clone@account.dfs.core.windows.net/...`, followed by DELETE, can keep all data files in `source` while putting the new DV in `clone`. Both current authority gates accept it because they inspect only data files.
At `7e09e04f`, an offline probe using the exact cache/prepare functions, new resolver, and real Azure store builder confirms that both URLs map to the same cache/registry identity: the DV resolves to `MicrosoftAzure { container: source }`, and its handle is pointer-identical to the data store. The DV read therefore requests the clone-relative path from the wrong container. Normally that fails with a missing object; a matching, well-formed object could instead supply the wrong deletion bitmap.
The helper predates this PR, but the new single-scan data-plus-DV path makes this reachable. Declining sidecars whose full authority differs but collides in the current store identity would contain it without a broad cache refactor. Fixing only the local `resolved_stores` key is insufficient because the shared cache and DataFusion registry also collapse the container. No live Azure request was used in the probe.
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,587 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::ParquetAccessPlan;
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::ObjectStoreExt;
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::PageIndexPolicy;
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::parquet::parquet_support::prepare_object_store_with_configs;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
Review Comment:
**[P2] Reserve construction memory and account for the reader's copy**
The cardinality cap and retained-plan reservations improve the original issue, but two allocations still bypass the pool at `7e09e04f`. `build_access_plan` finishes before `try_grow`, and Parquet's `RowSelection::from(Vec)` allocates a second vector during construction. Later, DataFusion 54.1's `create_initial_plan` deep-clones the attached access plan while the original remains in the file extensions, without an additional DV reservation.
A counting-allocator probe using the unchanged `build_access_plan`/`total_selectors` functions and the locked dependencies tested 2,000,000 rows with exactly 1,000,000 alternating deletions, which the default cap admits. It measured 65,554,457 bytes of peak construction allocation before a 1-byte pool rejected the reservation. With a 32,000,000-byte reservation accepted, the opener-equivalent clone allocated another 32,000,025 bytes while the pool stayed at 32,000,000. These are allocator-requested bytes, not RSS or a reproduced executor OOM; reservation release on drop works correctly.
Could we pre-reserve a conservative construction bound and then shrink it, and either transfer ownership or account for the active reader's copy? The current check can reject after memory pressure has already occurred. I would treat this as a remaining P2 under the new cap, rather than the original unbounded P1.
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.net.URI
+import java.util.Locale
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.CometScanRule
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands. Anything else on the protocol declines the
+ * table. Note `deletionVectors` and `columnMapping` are declined separately (below) so their
+ * fallback reasons are specific.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format (not a further subclass)? Compared by class name,
+ * not `classOf`, deliberately: this is the first gate on every V1 scan, and it must stay inert
+ * when the contrib jar is deployed without delta-spark on the classpath:
+ * `classOf[DeltaParquetFileFormat]` here raises NoClassDefFoundError inside CometScanRule and
+ * takes down every parquet scan in the session. When the name matches, delta-spark is
+ * necessarily present (the instance exists), so the Delta types past this gate are safe.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Returns the first reason this Delta scan cannot go native, or None when it is claimable. Only
+ * called when [[isDeltaScan]] is true. `scanHelper` is the same [[CometScanExec]] the caller
+ * builds to drive [[CometDeltaNativeScan.convert]] on a claim, reused here (rather than listed
+ * separately) to resolve the scan's selected files for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles with useMetadataRowIndex=false) injects a
+ // generated row-index column directly into the data schema and disables reader
+ // optimizations; its values must come from Spark's reader. Claiming such a scan would
+ // feed NULL row indexes into deletion-vector construction, silently corrupting DML.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ // Name mode is supported by serializing physical-name schemas (the parquet reader then
+ // matches file columns by name natively). Id mode needs the field-id path and stays
+ // declined until validated.
+ val cmMode = metadata.columnMappingMode.name
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping
+ // EXISTS_DEFAULT; decline any column defaults under column mapping rather than
+ // return nulls where a default belongs.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names (not just top-level column
+ // names) to their physical, column-mapped form, and the shared native builder uses the
+ // required schema verbatim as the scan's output schema: struct fields below the top level
+ // would carry physical names. Ordinal access (GetStructField) is unaffected, but
+ // name-sensitive native expressions (e.g. to_json) read the Arrow struct field names
+ // directly and would leak physical names into query results. Restoring logical names
+ // natively needs a rename adapter/proto field for the logical schema (follow-up); decline
+ // until then.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and not
+ // supported, except Delta's DV bookkeeping columns, which the native path emits as
+ // constants (correct by construction once the DV is applied in the reader).
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is not a DV read: it is Delta DML
+ // bookkeeping (findTouchedFiles building deletion bitmaps from REAL row indexes).
+ // Claiming it with constant row indexes would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // The internal columns must form a suffix of the read schema so data-column
+ // positions agree between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // The row-index column's real values are consumed inside the reader when Spark applies
+ // the DV; native applies the DV itself and emits a dead constant instead, so the value
+ // must be provably unused above the scan (beyond the _metadata reassembly that gets
+ // discarded).
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet; decline rather
+ // than silently return nulls for backfilled columns in old files.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bound the memory the native side will retain for expanded DV row selectors before
+ // committing to native execution: applying a deletion vector expands it into per-row
+ // RowSelectors that are reserved against the execution memory pool at scan time (see
+ // delta_dv.rs), and an alternating deleted/retained bitmap produces one non-coalescing
+ // selector per row. A row group's selector count is bounded above by
+ // 2*cardinality + #row-groups (each deleted row splits at most one run into a
+ // select/skip pair, plus one selector per row-group boundary), so the descriptor's
+ // cardinality -- deserialized at planning time via selectedDvDescriptors, no bitmap
+ // decode needed -- is a sound, pessimistic upper bound on the native reservation.
+ // Pessimistic by design: a large but CONTIGUOUS deletion is declined the same as a
+ // large alternating one, even though it would retain far fewer selectors natively; the
+ // conf below makes that recoverable.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = selectedDvDescriptors(scanHelper, tableRoot)
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from InputFileBlockHolder, a thread-local set by Spark's
+ // FileScanRDD; the native scan does not populate it. Delta's own DELETE/UPDATE/MERGE
+ // find-touched-files scans use input_file_name, so this gate is load-bearing for DML
+ // correctness (mirrors core's check in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core).
+ // The DV shape's trailing row-index column is exempt: the gates above already proved its
+ // values are dead and the native path emits a constant for it.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does
+ // not yet wire the decryption-key broadcast to executors, so claiming even a
+ // supported-encryption scan would fail at execution.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults (schema-evolution backfill of map/struct/array columns)
+ // cannot be serialized; a silently-dropped default would misalign the value/index lists
+ // consumed positionally on the native side. Mirrors core's transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read; otherwise a custom Hadoop FileSystem would fail at execution instead of
+ // falling back gracefully. Mirrors core's unsupportedFsSchemes gate.
+ val libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+ val unsupportedFsSchemes = scanExec.relation.location.rootPaths
+ .map(_.toUri)
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfsSchemes.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ if (unsupportedFsSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedFsSchemes.mkString(", ")}")
+ }
+
+ // A Delta shallow clone across buckets followed by an append is a valid table whose data
+ // files span multiple object-store authorities. The shared native scan builder resolves
+ // the whole scan's ObjectStoreUrl from the FIRST selected file only and then strips every
+ // other file down to its bare object-store path, so a later file under a different store
+ // would silently read through the first file's store handle -- normally a NoSuchKey, but
+ // the wrong data if a same-named key happens to exist in both stores. Force file listing
+ // here (scanHelper is already built for the claim path, so this is not extra work) and
+ // decline rather than risk it.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles,
+ // AQE DPP on Spark 3.4, exec enabled). This tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, deserialized once at
+ * planning time and normalized to absolute on-disk paths via `copyWithAbsolutePath` (a no-op
+ * for inline and already-absolute descriptors), so callers never need `tableRoot` again to
+ * resolve a UUID-relative sidecar. Returns `Seq.empty` for the plain shape
+ * ([[CometDeltaNativeScan.isDvShape]] false on the wrapped scan): only DV reads carry the
+ * row-index-filter metadata this deserializes.
+ *
+ * Shared plumbing: finding 8's cross-authority object-store option merge
+ * ([[CometDeltaNativeScan.convert]]) and finding 3's DV cardinality decline gate both need
+ * every selected file's descriptor; this is the one planning-time deserialization pass for both
+ * consumers.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * Returns a decline reason when `uris` span more than one object-store authority (scheme plus
+ * the URI's raw authority component -- userinfo, host, and port together -- all lowercased so
+ * e.g. `S3A://Bucket:1234` and `s3a://bucket:1234` collapse to the same authority), or `None`
+ * when every URI shares a single authority. `file://` paths never carry an authority, so purely
+ * local scans across any number of distinct directories are unaffected. Factored out of
+ * [[declineReason]] so it is directly unit-testable without a Spark session.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the URI's raw
+ * `getAuthority` rather than the individually-parsed host/port/userinfo fields. Two pitfalls
+ * that motivate this:
+ * - `getAuthority` already includes userinfo (e.g. the container in
+ * `abfss://container@account.dfs.core.windows.net`), so two containers on the same storage
+ * account no longer collapse into one authority the way `getHost` alone would.
+ * - `getHost` (and `getUserInfo`/`getPort`) return `null` for the *entire* authority when it
+ * doesn't conform to RFC 3986's `reg-name` syntax -- e.g. an underscore in a GCS bucket
+ * name (`gs://my_bucket`) -- silently collapsing distinct buckets into the same empty-host
+ * key. `getAuthority` returns the raw authority text regardless of RFC conformance, so it
+ * stays accurate for exactly the URIs where the structured getters fail.
+ *
+ * A `null` authority (schemes with no authority component, e.g. `file:///tmp/x`) normalizes to
+ * the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * True when `dataType` is, or structurally contains (through array elements or map keys/
+ * values), a [[StructType]]. Array and map are structural container types whose own
+ * "element"/"key"/"value" labels are never column-mapped; only the [[StructType]] fields
+ * reachable through them carry Delta's physical, column-mapped names.
+ */
+ private def containsNestedStruct(dataType: DataType): Boolean = dataType match {
+ case _: StructType => true
+ case ArrayType(elementType, _) => containsNestedStruct(elementType)
+ case MapType(keyType, valueType, _) =>
+ containsNestedStruct(keyType) || containsNestedStruct(valueType)
+ case _ => false
+ }
+
+ /**
+ * True when the scan's row-index column value is provably dead above the scan. The standard DV
+ * plan shape routes it only into a `named_struct(... row_index ...) AS _metadata` projection
+ * whose result the final projection discards; anything else (a query actually selecting
+ * `_metadata.row_index`) makes the value live and must decline. Conservative: any unrecognized
+ * consumption pattern returns false.
+ */
+ private def rowIndexUnusedAbove(plan: SparkPlan, scanExec: FileSourceScanExec): Boolean = {
+ val rowIndexAttrs = scanExec.output
+ .filter(_.name == CometDeltaNativeScan.RowIndexColumn)
+ .map(_.exprId)
+ .toSet
+ if (rowIndexAttrs.isEmpty) {
+ return true
+ }
+ // Transitive taint analysis: everything derived (via Project aliases) from the
+ // row-index attribute within the VISIBLE plan. The plan handed to this rule may be
+ // an AQE stage fragment, so anything tainted that reaches the fragment's own output
+ // escapes to invisible consumers and must decline. Non-Project consumption of any
+ // tainted attribute (a Filter, Aggregate, Join key, ...) declines outright.
+ var tainted = rowIndexAttrs
+ var changed = true
+ while (changed) {
+ changed = false
+ plan.foreach {
+ case p: ProjectExec =>
+ p.projectList.foreach {
+ case a: Alias
+ if !tainted.contains(a.exprId) &&
+ a.references.exists(r => tainted.contains(r.exprId)) =>
+ tainted += a.exprId
+ changed = true
+ case _ =>
+ }
+ case _ =>
+ }
+ }
+ val nonProjectConsumer = plan.exists {
+ case _: ProjectExec => false
+ case n if n ne scanExec =>
+ n.expressions.exists(_.references.exists(r => tainted.contains(r.exprId)))
Review Comment:
**[P2] Track live row-index values across Union output remapping**
Could we propagate row-index dependencies through `UnionExec`'s positional outputs, or conservatively decline this shape? The current analysis only propagates through `ProjectExec` aliases. `UnionExec.expressions` is empty and its output uses the first child's expression IDs, so a live row-index alias in the second branch disappears from both checks here. With native Delta scans enabled and AQE disabled, this valid query over DV-backed tables reaches the gap:
```sql
SELECT id, _metadata.row_index AS ri FROM delta.`t1`
UNION ALL
SELECT id, _metadata.row_index AS ri FROM delta.`t2`
```
I ran the unchanged `rowIndexUnusedAbove` method on actual Spark 4.0.2 / Delta 4.0.0 plans. Each table had one file containing IDs 0–4, with ID 2 deleted from the first and ID 3 from the second. The first scan correctly returns `false`, but the second returns `true`; its other admission inputs pass. The DV serializer then supplies `0L` for that supposedly unused row index. Modeling exactly that substitution changes the second branch's row indexes to zero and `SUM(ri)` over the union from 15 to 8. This is an exact-method/physical-plan probe, not a full Comet/JNI execution. Please add second-branch selection and aggregate-over-union regression coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5593772322
@andygrove the restructure you asked for is complete: #5653 merged, #5654 split out, and the case folding comes from #5602. The changes-requested predates that; could you take another look when you get a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5529054218
@dwsmith1983 @ErikBPF we have a Slack channel in the ASF workspace. Could you share your email address? I can add both of you there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5519402237
@andygrove @sunchao @ErikBPF do you all have a Slack channel we can use to plan?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3861279048
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so
+ * `CometPlanAdaptiveDynamicPruningFilters`'s rewrite (via [[CometScanWithPlanData]]) survives
+ * plan copies, the lesson from CometIcebergNativeScanExec (a transient field is dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node, our
+ // own withDynamicPruningFilters, or a generic Catalyst expression rewrite going through
+ // TreeNode.makeCopy, automatically gets a helper consistent with ITS runtimeFilters. A
+ // stored helper field would desync from rewritten filters (the #3510 class of bug). The
+ // cost is that file listing runs once per executed instance (planning listed separately in
+ // the rule extension); correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below (via
+ // `scanHelper.metrics`), and AQE's UI plan-walk calls `.metrics` on every node MID-PLANNING --
+ // including while a DPP subquery is still an adaptive placeholder or a partition filter holds
+ // an unresolved ScalarSubquery (see `hasUnevaluableSubqueryFilter` just below). Constructing
+ // `scanHelper` here, and reading `CometScanExec.metrics` off it, is safe ONLY because that
+ // construction is a cheap case-class build with no file listing, and core's
+ // `CometScanExec.metrics` (spark/.../CometScanExec.scala) touches only `wrapped.driverMetrics`
+ // -- populated by Spark's own planning, not by this scan -- plus a static metric-node
+ // constructor. Neither does file listing (`selectedPartitions`/`getFilePartitions`) or
+ // subquery resolution. If core's `metrics` is ever changed to touch either, forcing
+ // `scanHelper` from this `metrics` override would resurrect the two AQE mid-planning crashes
+ // this invariant was written to prevent.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element (planningHelper passes
+ // partitionFilters = runtimeFilters into CometScanExec's plain constructor field below), so
+ // checking runtimeFilters here avoids constructing/forcing the derived scanHelper just to
+ // read partitioning. The InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep these two
+ // sets in sync; if that rule learns to unwrap a new wrapper form, mirror it here too. The
+ // ScalarSubquery case (e.g. `p = (SELECT max(p) FROM dim ...)`, a partition-column filter,
+ // so it lands in runtimeFilters rather than dataFilters) is presence-based, unlike the
+ // InSubqueryExec check above: Spark exposes no public finished/updated probe on
+ // ExecSubqueryExpression, so any partition filter containing one forces UnknownPartitioning(0)
+ // even once it has actually resolved. That is safe one-directionally only -- worst case an
+ // extra shuffle from an overly conservative partitioning, never a wrong answer, since
+ // execution reads perPartitionData directly and never goes through this getter.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case _: ScalarSubquery => true
Review Comment:
Fixed. The guard now probes resolution instead of presence: eval on a resolved scalar subquery is a pure cached read (verified across the 3.5 through 4.2 jars), so once prepare resolves it the getter returns the real partition count, and only the unresolved case reports UnknownPartitioning(0). Your fused-parent shape reproduced the per-partition length failure locally and is now a regression that requires the native scan beneath a native aggregate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3861307562
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,539 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = toPhysical(scanExec, relation.partitionSchema),
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme.
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
+
+ val common = commonBuilder.build()
+ val deltaCommon = OperatorOuterClass.DeltaSparkScanCommon
+ .newBuilder()
+ .setTableRoot(tableRoot)
+ .setColumnMappingMode(columnMappingMode(scanExec))
+ .setSourceKey(DeltaPlanDataInjector.sourceKey(tableRoot, common))
+ .build()
+ val deltaScan = OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaCommon)
+ Operator
+ .newBuilder()
+ .setPlanId(scanExec.id)
+ .setContribScan(DeltaSparkScanEnvelope.pack(deltaScan.build()))
+ .build()
+ }
+ }
+
+ /**
+ * One representative store URI per distinct object-store authority this scan's partitions may
+ * need options for: the data-file authority (`firstFileUri`), the table root unconditionally
+ * (UUID-relative DV sidecars resolve against it, and it is cheap to include even when absent),
+ * and every distinct on-disk DV authority from `descriptors`. Inline DVs are filtered out --
+ * they carry no external URI, only embedded bytes. Deduping by authority (rather than by full
+ * URI) keeps this O(distinct authorities) instead of O(files): a table with N deletion-vector
+ * files on the same external store previously produced ~N distinct URIs here, each
+ * independently fed into `mergedObjectStoreOptions`'s `extractObjectStoreOptions` walk over
+ * `hadoopConf`. Candidates are deduped keeping the FIRST URI seen per authority, so callers can
+ * rely on `firstFileUri`/the table root winning over any DV path that happens to share their
+ * authority. Factored out of [[convert]] so the URI-assembly logic (in particular the
+ * `storageType` filter and `absolutePath` resolution) is directly unit-testable with hand-built
+ * [[DeletionVectorDescriptor]] fixtures, without a Spark session or real selected files (a
+ * `file://` scan alone can't exercise a foreign-authority DV).
+ */
+ private[delta] def storeUris(
+ descriptors: Seq[DeletionVectorDescriptor],
+ tableRootPath: Path,
+ firstFileUri: Option[java.net.URI]): Seq[java.net.URI] = {
+ val dvAuthorityUris = descriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(tableRootPath).toUri)
+ val candidates = firstFileUri.toSeq ++ Seq(tableRootPath.toUri) ++ dvAuthorityUris
+ val byAuthority = scala.collection.mutable.LinkedHashMap.empty[String, java.net.URI]
+ candidates.foreach(uri =>
+ byAuthority.getOrElseUpdate(DeltaScanSupport.uriAuthority(uri), uri))
+ byAuthority.values.toSeq
+ }
+
+ /**
+ * Unions `NativeConfig.extractObjectStoreOptions` over every `uris` authority. Safe to simply
+ * union rather than pick one: the extracted keys are scheme-disjoint prefixes (`fs.s3a.*` vs
+ * `fs.azure.*`, ...), so options for different schemes never collide, and re-extracting the
+ * same scheme from two URIs is idempotent. Factored out of [[convert]] so it is directly
+ * unit-testable without a Spark session.
+ */
+ private[delta] def mergedObjectStoreOptions(
+ hadoopConf: org.apache.hadoop.conf.Configuration,
+ uris: Seq[java.net.URI]): Map[String, String] =
+ uris.foldLeft(Map.empty[String, String]) { (merged, uri) =>
+ merged ++ NativeConfig.extractObjectStoreOptions(hadoopConf, uri)
+ }
+
+ /**
+ * Harvest subquery-bearing predicates for this scan from its covering FilterExec. Spark 3.x
+ * strips them from a scan's `dataFilters` at planning (`FileSourceStrategy` routes them to the
+ * post-scan filter only), while Spark 4.x keeps them in `dataFilters`. Collecting them here at
+ * claim time gives the execution-time resolve-and-push path the same inputs on every Spark
+ * version; the dedup keeps Spark 4.x from carrying duplicates.
+ *
+ * Safety comes from the plan walk, not just the reference guard: a filter is only harvested
+ * when every operator between it and the scan commutes with pushing the predicate into the scan
+ * (see `spineToScan`). Reference containment alone proves the predicate is expressible over the
+ * scan's output, not that moving it there is semantics-preserving -- an intervening LIMIT/TopN
+ * (or Sort, Aggregate, Window, join, ...) can change which rows the predicate would have
+ * applied to, so those stop the walk and the filter is left where Spark placed it.
+ */
+ def subqueryFiltersFromParent(
+ plan: org.apache.spark.sql.execution.SparkPlan,
+ scanExec: FileSourceScanExec): Seq[org.apache.spark.sql.catalyst.expressions.Expression] = {
+ import org.apache.spark.sql.catalyst.expressions.{PlanExpression, SubqueryExpression}
+ import org.apache.spark.sql.execution.{FilterExec, ProjectExec, SparkPlan}
+
+ // Whether every node on the path from `node` down to `scanExec` is one that pushdown can
+ // safely cross: a deterministic ProjectExec is 1:1 on rows (an Alias mints a new exprId, so
+ // it cannot alias over the scan's own output attributes, which is what the reference guard
+ // below requires) and a deterministic FilterExec only removes rows, so moving a predicate
+ // expressed over the scan's output through either preserves the query's semantics. A
+ // nondeterministic projection breaks that: a deterministic conjunct does not commute with it,
+ // because pushing the predicate into the scan changes which rows survive to have
+ // nondeterministic expressions (e.g. monotonically_increasing_id()) evaluated over them,
+ // changing the result -- so both guards require `.deterministic`, mirroring Spark's own
+ // PushPredicateThroughNonJoin/CollapseProject rules. Anything else (LIMIT/TopN, Sort,
+ // Aggregate, Window, joins, Union, Sample, ...) can reorder or drop rows in ways that make
+ // "push the predicate down to the scan" change the result, so an unrecognized node stops the
+ // walk: the filter is left uncollected (missed pruning only, never a correctness issue).
+ def spineToScan(node: SparkPlan): Boolean = node match {
+ case n if n eq scanExec => true
+ case p: ProjectExec if p.projectList.forall(_.deterministic) => spineToScan(p.child)
+ case f: FilterExec if f.condition.deterministic => spineToScan(f.child)
+ case _ => false
+ }
+
+ // Nearest FilterExec whose spine down to the scan is Project/Filter-only (the DV shape
+ // interposes such nodes between them, so do not require a direct parent-child edge).
+ val filtersAboveScan = plan.collect {
+ case f: FilterExec if spineToScan(f.child) => f
+ }
+ filtersAboveScan.lastOption
+ .map { f =>
+ splitConjunctivePredicates(f.condition)
+ .filter(_.deterministic)
+ .filter(_.references.subsetOf(scanExec.outputSet))
+ .filter(p =>
+ SubqueryExpression.hasSubquery(p) || p.exists(_.isInstanceOf[PlanExpression[_]]))
+ .filterNot(p => scanExec.dataFilters.exists(_.semanticEquals(p)))
+ }
+ .getOrElse(Seq.empty)
+ }
+
+ /**
+ * Resolve scalar-subquery data filters at execution time and serialize them for native
+ * pushdown, mirroring `CometNativeScanExec.serializedPartitionData`. `supportedDataFilters`
+ * excludes PlanExpressions at planning time (subquery results do not exist yet), so these
+ * bounds reach the native reader only through this path. Filters that fail to serialize are
+ * skipped: Spark keeps a covering FilterExec above the scan, so this is missed pruning only,
+ * never a correctness issue.
+ *
+ * Known core-parity limitation: when the scan is fused under a parent native operator,
+ * `ensureSubqueriesResolved` has already called `updateResult()` on these subqueries and this
+ * path calls it again (Spark's ScalarSubquery.updateResult re-executes unconditionally). Core's
+ * CometNativeScanExec has the identical double-execution; benign for Delta (the subquery's
+ * snapshot is pinned at analysis), wasteful for expensive subqueries. Fix belongs in core.
+ */
+ def resolvedSubqueryFilters(
+ dataFilters: Seq[org.apache.spark.sql.catalyst.expressions.Expression],
+ output: Seq[org.apache.spark.sql.catalyst.expressions.Attribute],
+ requiredSchema: StructType,
+ conf: org.apache.spark.sql.internal.SQLConf)
+ : Seq[org.apache.comet.serde.ExprOuterClass.Expr] = {
+ if (!conf.getConf(org.apache.spark.sql.internal.SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ return Seq.empty
+ }
+ val subqueryFilters = dataFilters.filter(_.exists(_.isInstanceOf[ExecScalarSubquery]))
+ if (subqueryFilters.isEmpty) {
+ return Seq.empty
+ }
+ // Same binding guard as the DV shape's plan-time filters: references limited to the
+ // data-column prefix of the output, where positions agree between the output and the
+ // native read schema. For the plain shape strippedLen is the full required schema.
+ // Guard BEFORE updateResult so discarded filters never execute their subqueries.
+ val strippedLen = requiredSchema.count(f => !internalColumnNames.contains(f.name))
+ val dataColIds = output.take(strippedLen).map(_.exprId).toSet
+ val pushableFilters =
+ subqueryFilters.filter(_.references.forall(r => dataColIds.contains(r.exprId)))
+ pushableFilters.foreach(_.foreach {
+ case s: ExecScalarSubquery => s.updateResult()
+ case _ =>
+ })
+ pushableFilters
+ .flatMap { filter =>
+ // MergeScalarSubqueries can fuse several scalar subqueries into one struct-returning
+ // subquery accessed via GetStructField; fold that whole subtree to a literal (a bare
+ // GetStructField-over-Literal would not serialize).
+ val resolved = filter.transform {
+ case g @ org.apache.spark.sql.catalyst.expressions
+ .GetStructField(_: ExecScalarSubquery, _, _) =>
+ Literal.create(g.eval(null), g.dataType)
+ case s: ExecScalarSubquery =>
+ Literal.create(s.eval(null), s.dataType)
+ }
+ val proto = exprToProto(resolved, output)
+ if (proto.isEmpty) {
+ logWarning(s"Could not serialize resolved scalar subquery filter: $resolved")
+ }
+ proto
+ }
+ }
+
+ /**
+ * DV shape common builder. Layout invariants (declined by DeltaScanSupport when violated): scan
+ * output = requiredSchema attrs (data columns, then the internal columns as a suffix) followed
+ * by partition and constant-metadata columns. The parquet read schema strips the internal
+ * columns; they are appended to the partition schema as per-file constants, so the projection
+ * vector routes them from the constants block.
+ */
+ private def buildDvScanCommon(
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ firstFileUri: Option[java.net.URI],
+ hadoopConf: org.apache.hadoop.conf.Configuration)
+ : Option[OperatorOuterClass.NativeScanCommon.Builder] = {
+ val relation = scanExec.relation
+ val output = scanExec.output
+ val requiredSchema = scanExec.requiredSchema
+
+ val commonBuilder = OperatorOuterClass.NativeScanCommon.newBuilder()
+ commonBuilder.setSource(scanExec.simpleStringWithNodeId())
+
+ val scanTypes = output.flatMap(attr => serializeDataType(attr.dataType))
+ if (scanTypes.length != output.length) {
+ return None
+ }
+ commonBuilder.addAllFields(scanTypes.asJava)
+
+ val strippedRequired =
+ StructType(requiredSchema.filterNot(f => internalColumnNames.contains(f.name)))
+ val strippedLen = strippedRequired.length
+ val requiredLen = requiredSchema.length
+
+ // Keep only data filters that bind identically in the output and the native
+ // (strippedRequired ++ partitionFields) index spaces: references limited to the first
+ // strippedLen output attributes. Internal-column filters (is_row_deleted = 0) are
+ // trivially true after native DV application, and Spark's Filter above the scan
+ // re-evaluates everything anyway.
+ if (scanExec.conf.getConf(
+ org.apache.spark.sql.internal.SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ val dataColIds = output.take(strippedLen).map(_.exprId).toSet
+ val filterProtos = scanHelper.supportedDataFilters
+ .filter(_.references.forall(r => dataColIds.contains(r.exprId)))
+ .flatMap(f => exprToProto(f, output))
+ commonBuilder.addAllDataFilters(filterProtos.asJava)
+ }
+
+ // Constant metadata columns and real partition columns follow the required schema in the
+ // output, exactly like the plain shape.
+ val constantMetadataFields = scanExec.fileConstantMetadataColumns.map(attr =>
+ StructField(
+ s"${CometNativeScan.constantMetadataFieldPrefix}${attr.name}",
+ attr.dataType,
+ attr.nullable))
+ val internalFields = requiredSchema.fields.toSeq
+ .filter(f => internalColumnNames.contains(f.name))
+ .map(f => StructField(s"$deltaConstFieldPrefix${f.name}", f.dataType, f.nullable))
+ // The real partition columns must carry physical names in the proto, same as the data/
+ // required schemas: a retained physical data name can otherwise collide with a partition
+ // column's LOGICAL name after a rename history (e.g. a->b then p->a), and DataFusion's
+ // name-based partition rewrite would then replace the data projection with the partition
+ // constant. constantMetadataFields/internalFields are synthetic slots the
+ // native side allocates for this scan, not table columns, so they are not physicalized.
+ val partitionSchemaFields = toPhysical(scanExec, relation.partitionSchema).fields.toSeq ++
+ constantMetadataFields ++ internalFields
+
+ // Protos carry physical names (column mapping); index math below stays logical.
+ val partitionSchemaProto = schema2Proto(partitionSchemaFields)
+ val requiredSchemaProto = schema2Proto(toPhysical(scanExec, strippedRequired))
+ val dataSchemaProto = schema2Proto(toPhysical(scanExec, relation.dataSchema))
+
+ // Projection: data columns from the (stripped) read schema; internal columns from their
+ // constants slots at the END of the partition fields; the output tail (real partitions +
+ // constant metadata) positionally from the head of the partition fields.
+ val dataSchema = relation.dataSchema
+ val internalBase = dataSchema.length + partitionSchemaFields.length - internalFields.length
+ val internalIndexByName = requiredSchema.fields.toSeq
+ .filter(f => internalColumnNames.contains(f.name))
+ .zipWithIndex
+ .map { case (f, i) => f.name -> (internalBase + i) }
+ .toMap
+ val projectionVector = output.zipWithIndex.map { case (attr, i) =>
+ val idx = if (internalColumnNames.contains(attr.name)) {
+ internalIndexByName(attr.name)
+ } else if (i < requiredLen) {
+ dataSchema.fieldIndex(attr.name)
+ } else {
+ dataSchema.length + (i - requiredLen)
+ }
+ idx.toLong.asInstanceOf[java.lang.Long]
+ }
+ commonBuilder.addAllProjectionVector(projectionVector.asJava)
+
+ commonBuilder.addAllDataSchema(dataSchemaProto.asJava)
+ commonBuilder.addAllRequiredSchema(requiredSchemaProto.asJava)
+ commonBuilder.addAllPartitionSchema(partitionSchemaProto.asJava)
+
+ CometNativeScan.populateScanConfFlags(
+ commonBuilder,
+ strippedRequired,
+ firstFileUri,
+ hadoopConf,
+ scanExec.conf)
+
+ Some(commonBuilder)
+ }
+
+ /** Serialize one file partition into a DeltaSparkScan proto with per-file DV descriptors. */
+ def serializePartition(
+ filePartition: FilePartition,
+ scanExec: FileSourceScanExec,
+ tableRoot: String): Array[Byte] = {
+ val relation = scanExec.relation
+ val sparkPartition = partition2Proto(
+ filePartition,
+ relation.partitionSchema,
+ scanExec.fileConstantMetadataColumns,
+ ShimFileFormat.fileConstantMetadataExtractors(relation.fileFormat))
+
+ val dvShape = isDvShape(scanExec)
+
+ val deltaPartition = OperatorOuterClass.DeltaSparkFilePartition.newBuilder()
+ sparkPartition.getPartitionedFileList.asScala.zip(filePartition.files.toSeq).foreach {
+ case (fileProto, file) =>
+ val fileBuilder = fileProto.toBuilder
+ if (dvShape) {
+ // Append the internal-constant values after the real partition/constant-metadata
+ // values, matching the order of the appended partition-schema fields.
+ scanExec.requiredSchema.fields
+ .filter(f => internalColumnNames.contains(f.name))
+ .foreach { f =>
+ val lit = f.dataType match {
+ case ByteType => Literal(0.toByte, ByteType)
+ case LongType => Literal(0L, LongType)
+ case other =>
+ // Fixed internal invariant (observed Delta 3.3 types); fail loudly on
+ // drift rather than emit a plausible-looking constant.
+ throw new IllegalStateException(
+ s"Unexpected type $other for Delta internal column ${f.name}")
+ }
+ fileBuilder.addPartitionValues(
+ literalToProto(lit, s"delta internal constant ${f.name}"))
+ }
+ }
+ val dfb = OperatorOuterClass.DeltaSparkPartitionedFile
+ .newBuilder()
+ .setFile(fileBuilder.build())
+ extractDvDescriptor(file, tableRoot).foreach(dfb.setDv)
+ deltaPartition.addPartitionedFile(dfb.build())
+ }
+
+ OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setFilePartition(deltaPartition.build())
+ .build()
+ .toByteArray
+ }
+
+ /**
+ * Pull the DV descriptor Delta attached to this file (base64 under
+ * `row_index_filter_id_encoded`), resolving UUID-relative paths to absolute URLs and
+ * Z85-decoding inline bitmaps here on the JVM where delta-spark's codecs live.
+ */
+ private def extractDvDescriptor(
+ file: PartitionedFile,
+ tableRoot: String): Option[OperatorOuterClass.DeltaSparkDvDescriptor] = {
+ val encoded = file.otherConstantMetadataColumnValues
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ val filterType = file.otherConstantMetadataColumnValues
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_TYPE)
+ encoded.map { enc =>
+ filterType match {
+ case Some(RowIndexFilterType.IF_CONTAINED) | None =>
+ case other =>
+ // DeltaScanSupport declines CDF reads, the only source of inverted filters;
+ // reaching here means a gate was bypassed -- fail loudly rather than corrupt.
+ throw new IllegalStateException(
+ s"Native Delta scan cannot apply row index filter type $other")
+ }
+ val desc = DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String])
+ val builder = OperatorOuterClass.DeltaSparkDvDescriptor
+ .newBuilder()
+ .setStorageType(desc.storageType)
+ .setSizeInBytes(desc.sizeInBytes)
+ .setCardinality(desc.cardinality)
+ if (desc.storageType == DeletionVectorDescriptor.INLINE_DV_MARKER) {
+ // The comet-spark jar relocates protobuf, so use the shaded ByteString.
+ builder.setInlineData(
+ org.apache.comet.shaded.protobuf.ByteString.copyFrom(desc.inlineData))
Review Comment:
Fixed for both classpaths. The contrib no longer names the protobuf package at all: a small helper in comet-spark takes the builder and raw bytes and sets the field internally. You were right that just changing the import would invert the problem, and it's worse than that: a helper returning ByteString also breaks the packaged side, because the shade plugin rewrites bytecode descriptors but not Scala pickled signatures. Keeping the type inside the helper's body sidesteps both. Your clean root compile repro now passes, and the packaged path plus full suite stay green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3861318116
##########
contrib/delta-spark/dev/run-delta-regression.sh:
##########
@@ -0,0 +1,163 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Run Delta Lake's own Spark test suites against a Comet build with the
+# native Delta scan enabled. Clones delta at $DELTA_VERSION into $WORKDIR,
+# injects Comet into the test SparkSession (DeltaSQLCommandTest) and the
+# test classpath (unmanagedJars), then runs the given testOnly selectors.
+#
+# Usage:
+# COMET_JARS=/path/comet-spark.jar,/path/comet-contrib-delta.jar,/path/flatbuffers.jar \
+# ./run-delta-regression.sh <workdir> 'org.apache.spark.sql.delta.DeletionVectorsSuite' [...]
+#
+# Env:
+# DELTA_VERSION delta tag to test against (default 3.3.2)
+# COMET_JARS comma-separated jars added to the test classpath (required)
+# JAVA_HOME JDK for sbt (17 recommended)
+set -euo pipefail
+
+DELTA_VERSION="${DELTA_VERSION:-3.3.2}"
+WORKDIR="${1:?usage: run-delta-regression.sh <workdir> <suite> [...suites]}"
+shift
+[ $# -ge 1 ] || { echo "no suites given" >&2; exit 2; }
+: "${COMET_JARS:?COMET_JARS must list the comet jars}"
+
+IFS=',' read -ra _jars <<< "$COMET_JARS"
+for j in "${_jars[@]}"; do
+ [ -f "$j" ] || { echo "COMET_JARS entry not found: $j" >&2; exit 2; }
+done
+
+DELTA_DIR="$WORKDIR/delta-$DELTA_VERSION"
+if [ ! -d "$DELTA_DIR" ]; then
+ git clone --depth 1 --branch "v$DELTA_VERSION" https://github.com/delta-io/delta.git "$DELTA_DIR"
+elif [ ! -d "$DELTA_DIR/.git" ]; then
+ echo "stale/partial checkout at $DELTA_DIR; remove it (rm -rf) and rerun" >&2
+ exit 2
+fi
+cd "$DELTA_DIR"
+
+# Add COMET_EXTRA_JARS to every project's test classpath, plus the JDK-17
+# module-access flags Spark needs (both for forked test JVMs and sbt's own JVM).
+if ! grep -q "COMET_EXTRA_JARS" build.sbt; then
+ python3 - <<'EOF'
+s = open('build.sbt').read()
+marker = 'lazy val commonSettings = Seq('
+opens = [
+ "--add-opens=java.base/java.lang=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED",
+ "--add-opens=java.base/java.io=ALL-UNNAMED",
+ "--add-opens=java.base/java.net=ALL-UNNAMED",
+ "--add-opens=java.base/java.nio=ALL-UNNAMED",
+ "--add-opens=java.base/java.util=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED",
+ "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED",
+ "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
+ "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED",
+ "--add-opens=java.base/sun.security.action=ALL-UNNAMED",
+ "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED",
+ "--add-exports=java.base/sun.nio.ch=ALL-UNNAMED",
+]
+opts = ", ".join('"%s"' % o for o in opens)
+inject = (
+ 'lazy val commonSettings = Seq(\n'
+ ' Test / unmanagedJars ++= sys.env.get("COMET_EXTRA_JARS").toSeq\n'
+ ' .flatMap(_.split(",")).map(p => Attributed.blank(file(p))),\n'
+ ' Test / fork := true,\n'
+ ' Test / javaOptions ++= Seq(%s),\n' % opts
+)
+assert marker in s, 'commonSettings marker not found'
+open('build.sbt', 'w').write(s.replace(marker, inject, 1))
+EOF
+fi
+
+# Inject Comet into the shared test SparkSession when COMET_EXTRA_JARS is set.
+TEST_BASE=spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala
+if ! grep -q "CometSparkSessionExtensions" "$TEST_BASE"; then
+ python3 - "$TEST_BASE" <<'EOF'
+import sys
+p = sys.argv[1]
+s = open(p).read()
+old = ''' override protected def sparkConf: SparkConf = {
+ super.sparkConf
+ .set(StaticSQLConf.SPARK_SESSION_EXTENSIONS.key,
+ classOf[DeltaSparkSessionExtension].getName)
+ .set(SQLConf.V2_SESSION_CATALOG_IMPLEMENTATION.key,
+ classOf[DeltaCatalog].getName)
+ }'''
+new = ''' override protected def sparkConf: SparkConf = {
+ val conf = super.sparkConf
+ .set(StaticSQLConf.SPARK_SESSION_EXTENSIONS.key,
+ classOf[DeltaSparkSessionExtension].getName)
+ .set(SQLConf.V2_SESSION_CATALOG_IMPLEMENTATION.key,
+ classOf[DeltaCatalog].getName)
+ if (sys.env.contains("COMET_EXTRA_JARS")) {
+ conf
+ .set(StaticSQLConf.SPARK_SESSION_EXTENSIONS.key,
+ classOf[DeltaSparkSessionExtension].getName +
+ ",org.apache.comet.CometSparkSessionExtensions")
+ .set("spark.comet.enabled", "true")
+ .set("spark.comet.exec.enabled", "true")
+ .set("spark.comet.exec.shuffle.enabled", "true")
+ .set("spark.shuffle.manager",
+ "org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager")
+ .set("spark.memory.offHeap.enabled", "true")
+ .set("spark.memory.offHeap.size", "2g")
+ .set("spark.comet.scan.delta.enabled", "true")
+ } else conf
+ }'''
+assert old in s, 'sparkConf block not found'
+open(p, 'w').write(s.replace(old, new))
+EOF
+fi
+
+# ScanReportHelper is a test-only trait that counts scans by pattern-matching
+# FileSourceScanExec in the executed plan. The Comet Delta scan replaces those
+# nodes, so claimed scans would go uncounted ("0 did not equal 2" in
+# MergeIntoSuiteBase's insert-only data-skipping test). Map the Comet node back
+# to the FileSourceScanExec it was built from: originalPlan carries the same
+# PreparedDeltaFileIndex, so the reported paths and skipping stats are identical.
+SCAN_HELPER=spark/src/test/scala/org/apache/spark/sql/delta/test/ScanReportHelper.scala
+if [ -f "$SCAN_HELPER" ] && ! grep -q "CometDeltaNativeScanExec" "$SCAN_HELPER"; then
+ python3 - "$SCAN_HELPER" <<'EOF'
+import sys
+p = sys.argv[1]
+s = open(p).read()
+old = " case fs: FileSourceScanExec => Seq(fs)\n"
+new = (" case fs: FileSourceScanExec => Seq(fs)\n"
+ " case c: org.apache.spark.sql.comet.CometDeltaNativeScanExec =>\n"
+ " Seq(c.originalPlan)\n")
+assert s.count(old) == 1, s.count(old)
+open(p, 'w').write(s.replace(old, new))
+EOF
+fi
+
+export COMET_EXTRA_JARS="$COMET_JARS"
+export SPARK_LOCAL_IP=127.0.0.1
+export RUST_BACKTRACE=1
+
+cmds=()
+for sel in "$@"; do
+ cmds+=("spark/testOnly $sel")
+done
+
+LOG="$WORKDIR/delta-regression-$(date +%Y%m%d-%H%M%S).log"
+echo "==> logging to $LOG"
+build/sbt "${cmds[@]}" 2>&1 | tee "$LOG" | grep -E "^\[info\] (Tests:|Suites:|All tests|.*\*\*\* FAILED| - )" | tail -80
Review Comment:
Fixed, the work directory is normalized to an absolute path before changing into the Delta checkout. A relative argument now exits 0 with the log in the expected place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5419693450
On scalar partition subqueries in outputPartitioning:
Fixed. The guard now treats a ScalarSubquery in any runtime filter as unevaluable too, so outputPartitioning returns UnknownPartitioning(0) without forcing perPartitionData. Added your mixed UNION ALL shape as a regression; it routes native and answers match. Same caveat as the earlier one: pre-fix it didn't crash on my Spark 3.5 profile, so the repro likely needs your 4.0 harness. One thing the audit turned up: core's CometNativeScanExec resolves scalar subqueries only in data filters and its outputPartitioning has no equivalent guard, so the plain parquet path may share this shape. Happy to file a follow-up issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3814824901
##########
.github/workflows/ci.yml:
##########
@@ -232,8 +233,18 @@
~ with:
~ spark-short: '3.5'
spark-full: '3.5.9'
java: 17
+ delta_contrib:
+ name: Delta Contrib Tests
+ needs: changes
+ if: |
+ needs.changes.outputs.delta == 'true' &&
+ (github.event_name == 'push' ||
+ github.event_name == 'workflow_dispatch' ||
+ github.event_name == 'pull_request')
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{}}
[Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/126)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3876137719
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,548 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so both internal
+ * columns are emitted as per-file constants (0), the parquet read schema is stripped to the
+ * real data columns, and the DV descriptor ships per file for native to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ // [[allocateUniqueInternalFields]] additionally suffixes on collision with a real column.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so output binding and projection are unaffected. The scan's internal DV columns
+ * are not part of the table schema and must be stripped before calling this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. Strip the parquet.field.id metadata
+ // createPhysicalSchema also stamps: files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations.
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
Review Comment:
**[P1] Current-head reader reproduction of the Unicode mismatch**
Follow-up on `4ca3207af385`: the current native library reads a real Parquet value `42` as follows with case sensitivity disabled. The reference column is Spark 4.1.3's actual `ParquetReadSupport.clipParquetSchema` on JDK21:
| Physical / requested name | Spark footer lookup | Native value |
| --- | --- | --- |
| `A1Σ` / `a1σ` | Missing field | `42` |
| `A1Σ` / `a1ς` | Match | `NULL` |
| U+A7C0 / U+A7C1 | Match | `NULL` |
The inverse sigma case therefore reads a field Spark considers missing, in addition to the previously reported lost values. A covering `IS NOT NULL` filter retains or drops the wrong row. Latin and case-sensitive controls pass.
Could the [matcher](https://github.com/apache/datafusion-comet/blob/4ca3207af3853f8c6572098e14b53f79ab1d6fea/native/core/src/parquet/schema_adapter.rs#L242-L248) account for the executing JVM's Unicode version and contextual sigma behavior? This is native Parquet execution plus Spark footer-method execution, not a full Spark/Delta/JNI query. The unchanged JVM parity suite also fails its full sweep on JDK21 (2 passed, 1 failed; only the test repository locator was stubbed).
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme (finding 8).
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
Review Comment:
**[P2] Empty short bucket aliases still bypass the credential comparison**
One remaining case at `4ca3207af385`: with `SimpleAWSCredentialsProvider`, set global and long bucket access/secret aliases (`fs.s3a.bucket.b.fs.s3a.access.key` / `secret.key`) to the same nonempty synthetic pair, and set the short aliases (`fs.s3a.bucket.b.access.key` / `secret.key`) to empty strings.
Both admission gates return `None`. Actual Hadoop 3.4.2 `S3AUtils.propagateBucketOptions` plus `SimpleAWSCredentialsProvider` resolves the populated long aliases. Current `NativeConfig` forwards the empty short aliases, and the unchanged Rust `get_config_trimmed` returns `Some("")` for both credentials. A protected S3 read is therefore claimed with different credentials and fails instead of falling back. Removing the empty short aliases is a passing control.
Could the native side of the comparator preserve empty overrides? [plainValue](https://github.com/apache/datafusion-comet/blob/4ca3207af3853f8c6572098e14b53f79ab1d6fea/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala#L509-L510) filters them out, unlike [native lookup](https://github.com/apache/datafusion-comet/blob/4ca3207af3853f8c6572098e14b53f79ab1d6fea/native/core/src/parquet/objectstore/s3.rs#L300-L317). These are executed Hadoop/admission/extraction/Rust component probes, not a live S3 or full Spark/JNI scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4084951687
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,2192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::mem::size_of;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::{ParquetAccessPlan, RowGroupAccess};
+use datafusion::execution::memory_pool::{MemoryConsumer, MemoryReservation};
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::path::Path;
+use object_store::{ObjectStore, ObjectStoreExt};
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::{PageIndexPolicy, ParquetMetaData};
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+/// An inline payload carries no framing, so its length is checked against the descriptor here,
+/// the way `unframe_dv_blob` checks an on-disk blob's size header.
+fn check_inline_payload_size(
+ file_path: &str,
+ payload: &[u8],
+ size_in_bytes: i32,
+) -> Result<(), ExecutionError> {
+ if payload.len() as i64 != i64::from(size_in_bytes) {
+ return Err(GeneralError(format!(
+ "Inline deletion vector for {file_path} has {} bytes but its descriptor says {size_in_bytes}",
+ payload.len()
+ )));
+ }
+ Ok(())
+}
+
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ // A corrupt footer can report a negative row count. `num_rows as u64` would otherwise
+ // wrap it into a huge positive value, silently corrupting every row-group boundary
+ // computed from `group_start`/`group_end` below (and therefore which deleted row indexes
+ // land in which row group) instead of failing loudly.
+ if num_rows < 0 {
+ return Err(GeneralError(format!(
+ "Parquet footer reports a negative row count ({num_rows}) for row group {idx}"
+ )));
+ }
+ let num_rows = num_rows as u64;
+ let group_end = group_start.checked_add(num_rows).ok_or_else(|| {
+ GeneralError(format!(
+ "Parquet footer row counts overflow at row group {idx} ({group_start} + {num_rows})"
+ ))
+ })?;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
+ }
+ cursor = row + 1;
+ }
+ if deleted_in_group == num_rows && num_rows > 0 {
+ plan.skip(idx);
+ } else if deleted_in_group > 0 {
+ if group_end > cursor {
+ selectors.push(RowSelector::select((group_end - cursor) as usize));
+ }
+ plan.scan_selection(idx, RowSelection::from(selectors));
+ }
+ group_start = group_end;
+ }
+ // A deleted index beyond the file's total row count means the DV does not
+ // belong to this file (stale or corrupted metadata); silently dropping it
+ // would under-apply deletions.
+ if let Some(&row) = deleted_iter.peek() {
+ return Err(GeneralError(format!(
+ "Deletion vector marks row {row} but the file only has {group_start} rows"
+ )));
+ }
+ Ok(plan)
+}
+
+/// Verify a decoded deletion vector's row count matches the descriptor's
+/// declared `cardinality`, mirroring Delta's JVM reader
+/// (`StoredBitmap.validateCardinality`). The CRC and framing checks catch
+/// corruption but not a stale, otherwise well-formed bitmap whose row count
+/// no longer matches the descriptor -- that would silently under- or
+/// over-delete rows.
+fn validate_cardinality(
+ file_path: &str,
+ expected: i64,
+ deleted: &RoaringTreemap,
+) -> Result<(), ExecutionError> {
+ let actual = deleted.len();
+ if actual != expected as u64 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has cardinality mismatch: descriptor says {expected}, decoded bitmap has {actual} deleted rows"
+ )));
+ }
+ Ok(())
+}
+
+/// One data file plus everything needed to apply its deletion vector. The
+/// file's size comes from `file.object_meta.size` (built by the planner from
+/// the proto's `file_size`).
+///
+/// `data_store` and `dv_store` are resolved by the caller *before* entering
+/// the async `attach_access_plans` runtime (see its doc comment): building an
+/// object store is sync I/O that, for a cold S3 authority, internally issues
+/// its own `Handle::block_on` calls, which panics if nested inside another
+/// `block_on`. Resolving up front means this module never constructs a
+/// store itself.
+pub struct DvScanFile {
+ pub file: PartitionedFile,
+ /// Full URL of the data file (proto `file_path`).
+ pub file_path: String,
+ pub dv: Option<DeltaSparkDvDescriptor>,
+ /// Object store for `file_path`, pre-resolved by the caller. Only read
+ /// when `dv` is `Some` (files without a deletion vector never open their
+ /// footer here), but every file carries one so the struct's shape
+ /// doesn't depend on whether a deletion vector is present.
+ pub data_store: Arc<dyn ObjectStore>,
+ /// Store and within-store path for an on-disk deletion vector's absolute
+ /// path, pre-resolved by the caller. `None` when the file has no
+ /// deletion vector or the deletion vector is stored inline.
+ pub dv_store: Option<(Arc<dyn ObjectStore>, Path)>,
+}
+
+/// Execution-memory-pool reservation covering one file's expanded DV row selectors across
+/// their *entire* lifetime attached to a scan -- from `build_access_plan`'s construction
+/// through DataFusion 54.1's reader normalizing the attached [`ParquetAccessPlan`]
+/// (`create_initial_plan`'s deep clone plus `into_overall_row_selection`'s combined
+/// `RowSelection`; see [`reader_peak_bytes`]) -- attached to the file's [`PartitionedFile`]
+/// extensions alongside its [`ParquetAccessPlan`]. The reservation's lifetime is tied to the
+/// `PartitionedFile` it is attached to, so it is released back to the pool exactly when the
+/// plan is dropped (query completion or an early-terminated scan), never held open longer.
+/// Newtype-wrapped so it occupies its own slot in the multi-slot, type-keyed `extensions` map
+/// (`datafusion_common::extensions::Extensions`) alongside the plan, rather than a bare
+/// `MemoryReservation` colliding with one some other extension might attach.
+pub struct DvAccessPlanReservation(pub MemoryReservation);
+
+/// Total number of [`RowSelector`]s materialized across `plan`'s per-row-group
+/// selections (`RowGroupAccess::Selection`); `Scan`/`Skip` row groups
+/// contribute none. An alternating deleted/retained bitmap produces one
+/// non-coalescing selector per row (see [`reader_peak_bytes`]'s doc comment
+/// for the worst-case accounting), so this count -- not the deletion
+/// vector's cardinality -- is the thing that must be bounded and reserved
+/// against the execution memory pool.
+fn total_selectors(plan: &ParquetAccessPlan) -> usize {
+ plan.inner()
+ .iter()
+ .map(|access| match access {
+ RowGroupAccess::Selection(selection) => selection.iter().count(),
+ _ => 0,
+ })
+ .sum()
+}
+
+/// Multiplier bounding the peak allocation live *during construction* of one
+/// file's [`RowSelection`]s, relative to the conservative selector-count
+/// bound `S = 2 * cardinality + num_row_groups` (one non-coalescing selector
+/// per deleted row in the worst-case alternating pattern, doubled, plus up to
+/// one extra boundary selector per row group). Split `S` into `r`, the
+/// selectors already retained from row groups `build_access_plan` has
+/// finished, and `c`, the selectors accumulated so far in the current row
+/// group's source `Vec`; `r` and `c` partition the selectors counted toward
+/// `S`, so `r + c <= S` always. While the current group is being built, the
+/// `Vec`'s doubling growth strategy can leave its backing allocation at up to
+/// `2 * c` (the next power-of-two capacity above `c`). Once the group
+/// finishes, `RowSelection::from(Vec)` (parquet's `FromIterator` impl,
+/// `with_capacity` + copy) builds a second, separate `Vec` of size `c` from
+/// that source while the source is still alive, so at the moment the copy
+/// begins, the retained selectors, the current group's doubled source `Vec`,
+/// and the copy are all live simultaneously: `r + 2c + c = r + 3c`. Since
+/// `r >= 0`, `r + 3c <= 3r + 3c = 3(r + c) <= 3S`. 3x covers that peak.
+const CONSTRUCTION_PEAK_FACTOR: usize = 3;
+
+/// Upper bound on how much larger a `Vec`'s backing allocation can be than its element count
+/// after being built by repeated pushes: `std`'s doubling growth strategy never leaves a `Vec`
+/// of `n` elements with a backing allocation larger than the next power of two above `n`, which
+/// is at most `2 * n` for any `n >= 1`.
+const VEC_GROWTH_CAPACITY_FACTOR: usize = 2;
+
+/// `RawVec`'s minimum non-zero capacity for element sizes `<= 1024` bytes ([`RowSelector`] is
+/// 16 bytes on 64-bit platforms: a `usize` row count plus a padded `bool`). Applied once per
+/// row group (or per contiguous run of row groups) a fresh `from_fn`/`FlatMap`-driven `Vec`
+/// gets built for (see [`reader_peak_bytes`]), so even a group or run whose true selector count
+/// is tiny still pays this floor.
+const MIN_VEC_CAPACITY_SELECTORS: usize = 4;
+
+/// Conservative upper bound, in bytes, on the peak allocation live while DataFusion 54.1's
+/// reader normalizes one file's attached [`ParquetAccessPlan`] -- the allocation this module's
+/// steady-state reservation must cover, not merely the plan's own retained selector bytes.
+/// THREE allocations can be live simultaneously by the time `into_overall_row_selection`
+/// returns, not two -- the clone is only exact when page-index pruning never touches it:
+///
+/// 1. **Attached original** (`selectors`, exact): `create_initial_plan` deep-clones the
+/// attached plan while the original remains reachable from the file's `extensions` until
+/// the scan consumes it. The ORIGINAL's own selector `Vec`s are exact -- a coalesced
+/// [`RowSelection`] built via `RowSelection::from(Vec<RowSelector>)` (what
+/// `build_access_plan` uses) has no excess capacity, because that conversion is a plain
+/// `with_capacity(len)` copy, not a `size_hint`-blind fold.
+/// 2. **The clone, possibly capacity-inflated** (`<= VEC_GROWTH_CAPACITY_FACTOR * selectors +
+/// MIN_VEC_CAPACITY_SELECTORS * num_row_groups`): if page-index pruning fires
+/// (`PagePruningAccessPlanFilter`; `access_plan.rs`'s `scan_selection` on a row group that
+/// already carries a `RowGroupAccess::Selection` calls `existing.intersection(&page_derived)`
+/// -- `RowSelection::intersection` -> `intersect_row_selections`), it replaces the CLONE's
+/// per-row-group selection with that intersection's output. `intersect_row_selections` is
+/// ANOTHER `from_fn` generator with `size_hint() == (0, None)`, so each intersected row
+/// group's backing `Vec` starts at `with_capacity(0)` and doubles as it grows, independent
+/// of whatever capacity the pre-intersection selection had. This inflated clone is still
+/// live when `into_overall_row_selection` later moves its buffer. Term 1's exactness
+/// guarantee holds for the ORIGINAL always, and for the clone only when page-index pruning
+/// never fires against it -- once it does, the clone must be charged at the SAME
+/// growth-capped bound as a fresh combined-selection `Vec` (term 3), summed once per row
+/// group rather than once per run, since each row group's `Selection` is intersected
+/// independently.
+/// 3. **Per-run combined-selection allocation** (`<= VEC_GROWTH_CAPACITY_FACTOR * (selectors +
+/// num_row_groups) + MIN_VEC_CAPACITY_SELECTORS * num_row_groups`): `into_overall_row_selection`
+/// collects each contiguous run of row groups' selectors into a *new* `RowSelection` via a
+/// `FlatMap` whose `size_hint().0 == 0`, so that run's `Vec` starts at `with_capacity(0)`
+/// and doubles as it grows -- capping its backing allocation at
+/// `max(MIN_VEC_CAPACITY_SELECTORS, next_power_of_two(len))`, which is at most
+/// `MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR * len` for a run of `len`
+/// selectors. `len` is at most that run's share of `selectors` plus one boundary selector
+/// per `RowGroupAccess::Scan` row group in the run (`Scan` always contributes exactly one
+/// `RowSelector::select(num_rows)`; see `access_plan.rs`'s `into_overall_row_selection`).
+/// Summing across at most `num_row_groups` runs (each spans >= 1 row group) bounds the total
+/// at `VEC_GROWTH_CAPACITY_FACTOR * selectors + (MIN_VEC_CAPACITY_SELECTORS +
+/// VEC_GROWTH_CAPACITY_FACTOR) * num_row_groups`.
+///
+/// Summing all three terms and converting to bytes: `((1 + 2 * VEC_GROWTH_CAPACITY_FACTOR) *
+/// selectors + (2 * MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR) * num_row_groups)
+/// * size_of::<RowSelector>()` -- with the constants above, `(5 * selectors + 10 *
+/// num_row_groups) * size_of::<RowSelector>()`. Checked against two measured worst cases:
+///
+/// - No page-index pruning (the original P2 report; term 2 stays exact): one 2,000,000-row
+/// group, 1,000,000 alternating deletions, `selectors = 2,000,000`. Measured allocator peak
+/// 97,554,457 B; the byte-for-byte accounting for the attached original plus the (here,
+/// exact) clone plus the inflated combined selection explains 97,554,432 B of that, a 25 B
+/// residue we did not attribute. This bound gives 160,000,160 B -- much looser here because
+/// it must also cover the next case, where the clone is NOT exact.
+/// - Page-index pruning fires against the clone: one 1,048,577-row group, `selectors =
+/// 1,048,577`. Measured peak 83,886,096 B; this bound gives 83,886,320 B (a 224 B, <1%
+/// margin -- deliberately tight, since this is the case that drives the bound).
+///
+/// Uses checked arithmetic throughout: a selector or row-group count large enough to overflow
+/// `usize` indicates a corrupted or malicious input, reported as a clean error rather than
+/// panicking.
+fn reader_peak_bytes(selectors: usize, num_row_groups: usize) -> Result<usize, ExecutionError> {
+ let overflow = || {
+ GeneralError(format!(
+ "Deletion vector reader-peak bound overflowed for {selectors} selectors and \
+ {num_row_groups} row groups"
+ ))
+ };
+ // Term 1: the attached original -- exact, untouched by page-index pruning (only the clone
+ // is ever intersected; see the doc comment above).
+ let attached_term = selectors;
+ // Term 2: the clone, bounded as if page-index pruning DID fire against every row group
+ // (safe even when it doesn't: term 2's bound is always >= `selectors`, so it never
+ // undershoots the exact case either).
+ let clone_growth = selectors
+ .checked_mul(VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let clone_floor = num_row_groups
+ .checked_mul(MIN_VEC_CAPACITY_SELECTORS)
+ .ok_or_else(overflow)?;
+ let clone_term = clone_growth.checked_add(clone_floor).ok_or_else(overflow)?;
+ // Term 3: into_overall_row_selection's per-run combined-selection allocation.
+ let combined_growth = selectors
+ .checked_mul(VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let combined_floor = num_row_groups
+ .checked_mul(MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let combined_term = combined_growth
+ .checked_add(combined_floor)
+ .ok_or_else(overflow)?;
+
+ let selector_bound = attached_term
+ .checked_add(clone_term)
+ .and_then(|sum| sum.checked_add(combined_term))
+ .ok_or_else(overflow)?;
+ selector_bound
+ .checked_mul(size_of::<RowSelector>())
+ .ok_or_else(overflow)
+}
+
+/// Upper bound, in [`RowSelector`]s, on how many extra selectors the parquet reader's
+/// page-index pruning can add on top of the deletion vector's own selection when normalizing
+/// one file, from that file's already-fetched [`ParquetMetaData`].
+///
+/// `intersect_row_selections` (parquet's `selection.rs`), which combines a page-pruning
+/// selection with the deletion vector's selection, is a `from_fn` generator whose
+/// `size_hint()` is `(0, None)`: for inputs of length `a` and `b`, its output can have up to
+/// `a + b` selectors -- longer than either input. Bounding the page-pruning side of that sum
+/// requires knowing how many selectors a page-index-derived selection could produce: at most
+/// two per data page (one skip, one select, in the worst case of alternating page-level
+/// pruning decisions), summed over every column of every row group.
+///
+/// Returns `0` when `metadata` carries no offset index (`metadata.offset_index()` is `None`).
+/// This is provably safe, not merely a convenient default: page-index pruning cannot produce a
+/// page-level selection without the offset index to locate pages by, so there are no
+/// page-pruning selectors to bound. The offset index is fetched with
+/// `PageIndexPolicy::Optional` from the same `FileMetadataCache` entry the scan's reader later
+/// reopens (see [`attach_access_plan`]'s footer-fetch comment), so this function observes
+/// exactly what the reader will see.
+///
+/// Uses checked arithmetic throughout for the same reason as [`admission_bound_bytes`].
+fn page_selection_bound_selectors(metadata: &ParquetMetaData) -> Result<usize, ExecutionError> {
+ let Some(offset_index) = metadata.offset_index() else {
+ return Ok(0);
+ };
+ let overflow = || {
+ GeneralError(
+ "Deletion vector page-selection bound overflowed while summing offset-index page \
+ locations"
+ .to_string(),
+ )
+ };
+ let mut total_page_locations = 0usize;
+ for row_group in offset_index {
+ for column in row_group {
+ total_page_locations = total_page_locations
+ .checked_add(column.page_locations().len())
+ .ok_or_else(overflow)?;
+ }
+ }
+ total_page_locations.checked_mul(2).ok_or_else(overflow)
+}
+
+/// Execution-memory-pool admission bound, in bytes, for one file's deletion-vector access
+/// plan -- reserved *before* calling `build_access_plan` (see [`attach_access_plan`]'s
+/// pre-reserve call site) to cover the larger of two peaks live at different points in the
+/// plan's lifetime. In practice the reader-normalization peak below dominates the construction
+/// peak unconditionally for any non-trivial input (`reader_peak_bytes(S, G) = (5S + 10G) *
+/// size_of::<RowSelector>()` always exceeds `CONSTRUCTION_PEAK_FACTOR * S *
+/// size_of::<RowSelector>() = 3S * size_of::<RowSelector>()` once `S >= 1`, since the `5S` term
+/// alone already exceeds `3S`); the construction term is retained as a documented floor rather
+/// than dropped, since it is cheap to compute and keeps this bound correct even if the reader's
+/// growth factors ever shrink below construction's.
+///
+/// - **Construction peak** (`CONSTRUCTION_PEAK_FACTOR * S`, see that constant's doc comment):
+/// live while `build_access_plan` builds the plan's `RowSelection`s. Construction's
+/// transient allocations fully unwind before `build_access_plan` returns, so this peak never
+/// overlaps the reader-normalization peak below.
+/// - **Reader-normalization peak** (`reader_peak_bytes(S + page_bound_selectors,
+/// num_row_groups)`, see that function): live later, once DataFusion's reader normalizes the
+/// attached plan. `S = 2 * cardinality + num_row_groups` is the same conservative bound on
+/// the plan's final retained selector count used for the construction peak -- it provably
+/// bounds `R = total_selectors(&plan)` (`R <= S`, from `build_access_plan`'s
+/// one-non-coalescing-selector-per-deleted-row worst case plus one boundary selector per row
+/// group), so `S + page_bound_selectors` bounds `R` after page-index inflation the same way
+/// `S` bounds `R` before it.
+///
+/// These two peaks never overlap in time, so `max` -- not `sum` -- is the correct combinator:
+/// reserving their sum would over-reserve for no safety benefit.
+///
+/// Deliberately not clamped by the file's total row count here, unlike the reader-peak target
+/// `attach_access_plan` resizes down to after construction (see that call site): `S`'s
+/// `+ num_row_groups` boundary term is a worst-case padding margin that can legitimately exceed
+/// the total row count for a small, heavily-deleted file, and admission sizing has no actual
+/// retained-selector count yet to clamp against -- only after construction, once `R` is known,
+/// is clamping to the total row count both meaningful and strictly tighter. Leaving this bound
+/// unclamped only ever makes admission more conservative, never less safe.
+///
+/// Uses checked arithmetic throughout: a cardinality, row-group count, or page bound large
+/// enough to overflow `usize` while computing this bound indicates a corrupted or malicious
+/// descriptor, reported as a clean error rather than panicking.
+fn admission_bound_bytes(
+ cardinality: i64,
+ num_row_groups: usize,
+ page_bound_selectors: usize,
+) -> Result<usize, ExecutionError> {
+ let overflow = || {
+ GeneralError(format!(
+ "Deletion vector admission bound overflowed for cardinality {cardinality}, \
+ {num_row_groups} row groups, and page bound {page_bound_selectors} selectors"
+ ))
+ };
+ let cardinality_usize = usize::try_from(cardinality).map_err(|_| overflow())?;
+ // S: the conservative bound on the plan's final *retained* selector count (what
+ // `total_selectors(&plan)` cannot exceed) -- unchanged from the pre-existing
+ // construction-only bound this function replaces.
+ let s = cardinality_usize
+ .checked_mul(2)
+ .and_then(|doubled| doubled.checked_add(num_row_groups))
+ .ok_or_else(overflow)?;
+
+ let construction_bytes = s
+ .checked_mul(size_of::<RowSelector>())
+ .and_then(|bytes| bytes.checked_mul(CONSTRUCTION_PEAK_FACTOR))
+ .ok_or_else(overflow)?;
+
+ let s_plus_page = s.checked_add(page_bound_selectors).ok_or_else(overflow)?;
+ let reader_bytes = reader_peak_bytes(s_plus_page, num_row_groups)?;
+
+ Ok(construction_bytes.max(reader_bytes))
+}
+
+/// Upper bound on concurrent DV-blob and footer fetches per partition. Both
+/// are small ranged reads, so a modest fan-out hides object-store latency
+/// without flooding the store client.
+const DV_FETCH_CONCURRENCY: usize = 8;
+
+/// Called via `block_on` at plan-creation time on the executor task: DV blobs
+/// are small ranged reads and footers are needed to learn row-group
+/// boundaries. Files are fetched concurrently (bounded by
+/// [`DV_FETCH_CONCURRENCY`]) with input order preserved. Footer fetches go
+/// through the scan's shared FileMetadataCache, so the scan's subsequent open
+/// of the same file is served from cache. That reuse relies on each input
+/// [`PartitionedFile`] being returned as-is (only `with_extension` applied),
+/// never rebuilt: the cache entry is keyed by this exact `object_meta` and the
+/// scan later looks it up through the same struct.
+///
+/// Deliberately takes no object-store options map and imports no
+/// store-construction helper: every [`DvScanFile`] arrives with its stores
+/// already resolved by the caller (see its doc comment), so this async path
+/// structurally cannot build an object store -- only `runtime_env` is still
+/// threaded through, for the shared `FileMetadataCache` and (per file) the
+/// execution `MemoryPool` each expanded access plan's row selectors are
+/// reserved against -- see [`DvAccessPlanReservation`].
+pub async fn attach_access_plans(
+ runtime_env: Arc<RuntimeEnv>,
+ files: Vec<DvScanFile>,
+) -> Result<Vec<PartitionedFile>, ExecutionError> {
+ futures::stream::iter(files)
+ .map(|scan_file| attach_access_plan(Arc::clone(&runtime_env), scan_file))
+ .buffered(DV_FETCH_CONCURRENCY)
+ .try_collect()
+ .await
+}
+
+/// Resolve one file's deletion vector into an attached [`ParquetAccessPlan`];
+/// files without a DV pass through untouched.
+async fn attach_access_plan(
+ runtime_env: Arc<RuntimeEnv>,
+ scan_file: DvScanFile,
+) -> Result<PartitionedFile, ExecutionError> {
+ let DvScanFile {
+ file,
+ file_path,
+ dv,
+ data_store,
+ dv_store,
+ } = scan_file;
+ let dv = match dv {
+ Some(dv) => dv,
+ None => return Ok(file),
+ };
+ // Delta's canonical `DeletionVectorDescriptor.EMPTY`: inline storage, empty
+ // payload, size 0, cardinality 0. Spark's reader returns all rows for it;
+ // decoding would fail (the empty payload is too short for a magic
+ // number), so pass the file through unchanged before attempting to read it.
+ if dv.cardinality == 0 && dv.size_in_bytes == 0 {
+ return Ok(file);
+ }
+ if dv.size_in_bytes < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative size {}",
+ dv.size_in_bytes
+ )));
+ }
+ if dv.cardinality < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative cardinality {}",
+ dv.cardinality
+ )));
+ }
+
+ let data: Vec<u8> = if let Some(inline) = dv.inline_data {
+ check_inline_payload_size(&file_path, &inline, dv.size_in_bytes)?;
+ inline
+ } else if let Some(dv_path) = &dv.absolute_path {
+ let offset = dv
+ .offset
+ .ok_or_else(|| GeneralError("On-disk deletion vector missing offset".into()))?;
+ if offset < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative offset {offset}"
+ )));
+ }
+ let offset = offset as u64;
+ // [i32 BE size][data: size_in_bytes][i32 BE crc]
+ let framed_len = 4 + dv.size_in_bytes as u64 + 4;
+ let (store, dv_store_path) = dv_store.ok_or_else(|| {
+ GeneralError(format!(
+ "Deletion vector for {file_path} has an absolute path but no pre-resolved object store"
+ ))
+ })?;
+ let blob = store
+ .get_range(&dv_store_path, offset..offset + framed_len)
+ .await
+ .map_err(|e| GeneralError(format!("Failed to read deletion vector {dv_path}: {e}")))?;
+ unframe_dv_blob(&blob, dv.size_in_bytes as usize)?.to_vec()
+ } else {
+ return Err(GeneralError(
+ "Deletion vector descriptor has neither inline data nor a path".into(),
+ ));
+ };
+ let deleted = deserialize_dv_bitmap(&data)
+ .map_err(|e| GeneralError(format!("Invalid deletion vector for {file_path}: {e}")))?;
+ validate_cardinality(&file_path, dv.cardinality, &deleted)?;
+
+ // Row-group boundaries come from the data file's footer, fetched through the scan's
+ // shared FileMetadataCache with the page index loaded eagerly and the scan's metadata
+ // size hint (mirroring EagerPageIndexReaderFactory): the one fetch here also serves the
+ // subsequent data-file open, so DV files pay no extra footer round-trip. Keyed by
+ // `file.object_meta`, the exact ObjectMeta the scan's reader factory will look up.
+ let metadata_cache = runtime_env.cache_manager.get_file_metadata_cache();
+ let metadata = DFParquetMetadata::new(data_store.as_ref(), &file.object_meta)
+ .with_file_metadata_cache(Some(metadata_cache))
+ .with_page_index_policy(Some(PageIndexPolicy::Optional))
+ .with_metadata_size_hint(Some(crate::parquet::parquet_exec::METADATA_SIZE_HINT))
+ .fetch_metadata()
+ .await
+ .map_err(|e| GeneralError(format!("Failed to read parquet footer of {file_path}: {e}")))?;
+ let row_counts: Vec<i64> = metadata
+ .row_groups()
+ .iter()
+ .map(|rg| rg.num_rows())
+ .collect();
+
+ // Pre-reserve the admission bound *before* calling build_access_plan: this bound covers
+ // both construction's own transient peak AND the larger peak DataFusion's reader hits
+ // later while normalizing the attached plan (`create_initial_plan`'s deep clone plus
+ // `into_overall_row_selection`'s combined RowSelection) -- see admission_bound_bytes and
+ // reader_peak_bytes. Reserving first means a rejection happens before any large `Vec` is
+ // allocated, not after -- see reader_peak_bytes's doc comment for the measured worst
+ // cases. The error message names this as a construction-phase rejection (contains
+ // "construct"), textually distinct from the steady-state message below, so callers/logs
+ // can tell which phase failed.
+ let page_bound_selectors = page_selection_bound_selectors(&metadata)?;
+ let admission_bytes =
+ admission_bound_bytes(dv.cardinality, row_counts.len(), page_bound_selectors)?;
+ let reservation =
+ MemoryConsumer::new("DeltaDeletionVectorAccessPlan").register(&runtime_env.memory_pool);
Review Comment:
Confirmed. `attach_access_plan` registered a consumer per file and the reservation rode in the file's extensions until the task ended, so every DV'd file in the partition added one to the fair pool's consumer count for the rest of the task.
`attach_access_plans` now registers one consumer per call and each DV'd file takes `new_empty()` from it. The per-file grow, resize and release are unchanged. Each reservation keeps its own byte count, so dropping one file returns only that file's bytes, and the single registration goes away when the last file holding a reservation from it is dropped. When no file in the batch carries a DV the registration ends when the call returns.
The new test uses a pool that copies `CometFairMemoryPool`'s check (limit is pool size over registered consumers) and counts register calls. It attaches three DV'd files in one call, asserts one registration, then registers a second consumer and grows it by a third of the pool, which fits with two consumers registered and would not with four. It also checks that dropping one file releases only that file's bytes and that the registration is gone once all files are dropped. That test fails on the old code with three registrations.
The `maxDeletedRowsPerFile` doc now says the cap bounds one file's selectors, and that the selectors for every file in a partition stay held until the task finishes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3881836013
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.ListBuffer
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable. Only called when
+ * [[isDeltaScan]] is true. `scanHelper` is the [[CometScanExec]] built to drive
+ * [[CometDeltaNativeScan.convert]] on a claim, reused for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`: it runs at most
+ // once per claim attempt, and not at all for the common non-DV-shape case.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
Review Comment:
Added the decline: any s3 scan whose effective encryption algorithm resolves to SSE-C falls back, covering the canonical, deprecated, and per-bucket spellings (Hadoop registers the deprecated ones as process-wide aliases, which one test learned the hard way). SSE-S3 and KMS stay claimable since decryption is server-side on read. The customer key never appears in the reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5723216445
Merged main; the `planner.rs` conflict was the shuffle test that #5916 removed next to the Delta ones. `run-delta-tests` does not exist as a label yet, so I cannot add it; the other `run-*` labels are there. Could you create it, or add it once it exists? Runner minutes for the CI-cost thread will come from that run. I resolved the three threads you listed. The rebase pass-through, the S3 hard-failure switch and the config-table guard are pushed, with replies inline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4006924646
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1838 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ // Populated now (rather than only at the very end) so it is available even though several
+ // early-return gates below still lie ahead: cheap to set, and every one of those gates
+ // declines the scan anyway, so `memo` is simply never read by `convert` in that case.
+ memo.hadoopConf = hadoopConf
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // An opted-in S3-compliant alias scheme (fs.comet.s3Compliant.schemes) is declined before the
+ // generic scheme gate below so the reason says why: core's native scan reads it through the
+ // S3 client, but the S3 divergence gates further down model Hadoop's S3AFileSystem only.
+ val rootUris = scanExec.relation.location.rootPaths.map(_.toUri)
+ val aliasReason = s3CompliantAliasSchemeReason(hadoopConf, rootUris)
+ if (aliasReason.isDefined) {
+ return aliasReason
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes = unsupportedSchemes(rootUris, libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A recognized scheme can still carry a path object_store rejects (a directory name with a
+ // newline surfaces as `%0A`), which native planning hard-fails on while Spark's reader opens
+ // it. Mirrors core's root-path gate; the complete selected paths are probed below.
+ val rejectedRoot = objectStoreRejectedPathReason(rootUris, libhdfs)
+ if (rejectedRoot.isDefined) {
+ return rejectedRoot
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Same path probe for every complete selected path, not just its directory: a shallow
+ // clone's source can sit outside this root, and CONVERT TO DELTA keeps the source Parquet
+ // basenames, so the rejected character can be in the file name itself. The probe is a
+ // native URL parse with no I/O, so once per distinct URI costs less than the scan's own
+ // per-file parse.
+ val rejectedSelected = objectStoreRejectedPathReason(dataFileUris ++ dvUris, libhdfs)
+ if (rejectedSelected.isDefined) {
+ return rejectedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Zero-I/O, conf-only, like the GCS gate above: decline any bucket configured for an
+ // encryption algorithm outside the allowlist (SSE-C, CSE-KMS, CSE-CUSTOM, or unknown) before
+ // the credential-divergence gates below, which do not otherwise notice this table is readable
+ // through Hadoop only because Hadoop's request factory (SSE-C) or SDK-level decryption layer
+ // (CSE-*) does something native never learns about.
+ val encryptionReason =
+ unsupportedEncryptionAlgorithmReason(hadoopConf, dataFileUris ++ dvUris)
+ if (encryptionReason.isDefined) {
+ return encryptionReason
+ }
+
+ // Shared across the two gates below: propagateBucketOptions is a full Configuration deep
+ // copy, and both gates would otherwise recompute it independently for the same bucket(s)
+ // (once here, then again per-key inside s3ConfigDivergenceReason). One cache, populated
+ // lazily per bucket on first use, makes it a single copy total per bucket across both gates.
+ val propagatedConfCache = MutableMap.empty[String, Configuration]
+
+ // Always zero-I/O (plain propagated-conf read, no keystore): native's S3 client has no
+ // HTTP proxy support at all (no fs.s3a.proxy.* key is read anywhere in s3.rs), so a bucket
+ // requiring a proxy for S3 egress must decline here rather than claim and then connect
+ // directly, bypassing whatever network-segmentation/firewall policy required the proxy.
+ val proxyReason = proxyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (proxyReason.isDefined) {
+ return proxyReason
+ }
+
+ // Zero-I/O, conf-only, like the proxy gate above: Hadoop's AssumedRoleCredentialProvider
+ // sends fs.s3a.assumed.role.policy as the session policy of its STS AssumeRole request,
+ // while native's assumed-role provider never reads the key -- a claimed scan would assume
+ // the role WITHOUT the configured session restriction, silently widening permissions.
+ val rolePolicyReason =
+ assumedRolePolicyGateReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (rolePolicyReason.isDefined) {
+ return rolePolicyReason
+ }
+
+ // Every fs.s3a.* option native's get_config (s3.rs) resolves must agree between what Hadoop
+ // itself would use and what native would read from the forwarded, substituted conf (covers
+ // long-form bucket credentials, JCEKS/credential-provider shadowing, and any other
+ // short-vs-effective divergence in one mechanism); reuses hadoopConf from the encryption gate
+ // above.
+ val s3Reason =
+ s3ConfigDivergenceReason(hadoopConf, dataFileUris ++ dvUris, propagatedConfCache)
+ if (s3Reason.isDefined) {
+ return s3Reason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ // Claimable: hand the already-forced descriptors to `convert` via `memo` so it does not
+ // deserialize them a second time.
+ memo.dvDescriptors = dvDescriptors
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * parsed exactly like core's scan gate (`NativeConfig.parseSchemeSet`: split on commas,
+ * trimmed, lowercased) and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) => NativeConfig.parseSchemeSet(s)
+ case None => Set("hdfs")
+ }
+
+ /**
+ * Decline reason when any of `uris` uses a scheme opted in as an S3-compliant alias through
+ * `fs.comet.s3Compliant.schemes` (e.g. `blob`), or `None`. Core's native Parquet scan admits
+ * such a scheme and reads it through its S3 client, with `NativeConfig` translating the vendor
+ * `fs.<scheme>.<authority>.*` keys into `fs.s3a.bucket.*` options. Spark, however, reads the
+ * same table through the vendor's own Hadoop FileSystem, not `S3AFileSystem`, and every S3
+ * divergence gate in this object ([[s3ConfigDivergenceReason]] and its siblings) is verified
+ * against `S3AFileSystem`'s consumers only. With no model of how the vendor filesystem resolves
+ * its configuration, whether native and Spark would agree cannot be decided, so the scan is
+ * declined rather than claimed on a guess. Selected data-file and deletion-vector URIs under an
+ * alias scheme are declined by the generic scheme gates, which never admit an alias (see
+ * [[unsupportedSchemes]]).
+ */
+ private[delta] def s3CompliantAliasSchemeReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val aliases = NativeConfig.resolveS3CompliantSchemes(hadoopConf)
+ if (aliases.isEmpty) {
+ return None
+ }
+ val found = uris
+ .flatMap(uri => Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)))
+ .filter(aliases.contains)
+ .distinct
+ if (found.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support S3-compliant alias filesystem scheme(s) " +
+ s"${found.sorted.mkString(", ")} (${CometConf.COMET_S3_COMPLIANT_SCHEMES_KEY}): " +
+ "Spark reads them through a vendor filesystem whose S3 configuration resolution the " +
+ "native scan's S3AFileSystem divergence model cannot verify")
+ }
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source. The alias
+ * set handed to core's gate is deliberately empty: an `fs.comet.s3Compliant.schemes` alias is
+ * never admitted here (see [[s3CompliantAliasSchemeReason]]), even though core admits it.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri, Set.empty)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason naming the first of `uris` whose path object_store rejects even though it
+ * recognizes the scheme ([[CometScanRule.objectStoreAcceptsPath]], e.g. a directory name
+ * containing a newline, `%0A` in the URI), or `None`. Schemes in `libhdfs` never reach
+ * object_store's path parser and are skipped, as is a `null` scheme (see
+ * [[unsupportedSchemes]]); an S3-compliant alias is declined before this gate runs. The probe
+ * is uncached but is a plain native URL parse with no I/O
+ * ([[CometScanRule.objectStoreAcceptsPath]]), so callers pass every complete selected path
+ * (root paths, data files and deletion vectors) once per distinct URI; a converted table can
+ * carry the rejected character in a file basename. The reason masks any userinfo in the named
+ * URI ([[redactedAuthority]]).
+ */
+ private[delta] def objectStoreRejectedPathReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ uris.distinct
+ .find { uri =>
+ val sch = uri.getScheme
+ sch != null && !libhdfs.contains(sch.toLowerCase(Locale.ROOT)) &&
+ !CometScanRule.objectStoreAcceptsPath(uri)
+ }
+ .map { uri =>
+ // Mask userinfo (see redactedAuthority); the raw path keeps its percent encoding so the
+ // reason shows the rejected sequence as written.
+ val shown =
+ if (uriUserInfo(uri).isEmpty) uri.toString
+ else s"${redactedAuthority(uri)}${Option(uri.getRawPath).getOrElse("")}"
+ s"Native Delta scan cannot open path '$shown': object_store rejects it " +
+ "(e.g. an unsupported character in the path)"
+ }
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted below. `hadoop-aws` is NOT on this module's runtime
+ * classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be referenced here (would raise
+ * `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ /**
+ * `CommonConfigurationKeysPublic.HADOOP_SECURITY_CREDENTIAL_CLEAR_TEXT_FALLBACK`, default
+ * `true`, verified via `javap` against `hadoop-common` 3.3.4's
+ * `Configuration#getPasswordFromConfig`: `getPassword` only falls back to reading a plaintext
+ * conf value once `getBoolean(<this key>, true)` holds -- with the flag off, a plaintext value
+ * is invisible to every `getPassword`-based resolver, even when no credential provider is
+ * configured at all.
+ */
+ private val ClearTextFallbackKey = "hadoop.security.credential.clear-text-fallback"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * How Hadoop's OWN consumer reads one of the keys compared by [[s3ConfigDivergenceReason]],
+ * which decides how [[s3KeyDivergenceReason]] computes the Hadoop-effective side of its
+ * equality check. Exactly two consumer families exist among [[AllS3ConfigKeys]] in `hadoop-aws`
+ * 3.3.4, each verified via `javap`/CFR against the real call sites (cited per key on
+ * [[S3ConfigKeyConsumers]]). The tier must mirror the key's ACTUAL consumer: resolving a
+ * [[PropagatedOptionConsumer]] key through the wider `lookupPassword` cascade is NOT fail-safe
+ * for a value-EQUALITY comparator -- a long-form alias value Hadoop itself never reads can
+ * EQUAL native's resolution while Hadoop's true propagate-then-plain-get value differs, turning
+ * a real divergence into a wrongly-claimed scan (the endpoint `${...}`-redirect shape pinned in
+ * `DeltaScanContribSuite`).
+ */
+ private[delta] sealed trait S3ConfigConsumer
+
+ /**
+ * Read via `S3AUtils#lookupPassword(bucket, conf, baseKey)`, verified via `javap` against
+ * `hadoop-aws` 3.3.4: builds `longBucketKey = "fs.s3a.bucket." + bucket + "." + baseKey` (the
+ * FULL, already-`fs.s3a`-prefixed base key appended after the bucket segment) and reads it via
+ * `Configuration#getPassword` BEFORE the short-bucket key, keeping the long value whenever
+ * `getPassword` returns non-empty and only falling through to short-then-global otherwise.
+ * `getPassword` is Hadoop-credential-provider-aware and skips plaintext conf entirely when
+ * [[ClearTextFallbackKey]] is false. Modeled by [[hadoopLookupPasswordEffective]].
+ */
+ private[delta] case object LookupPasswordConsumer extends S3ConfigConsumer
+
+ /**
+ * Read via `S3AUtils#propagateBucketOptions` followed by a plain `Configuration#get`-family
+ * call (`getTrimmed`/`getBoolean`/`getClasses`) against the propagated view: the short bucket
+ * form wins only by having overwritten the global key during propagation, the long bucket form
+ * folds into an unread `fs.s3a.fs.s3a.*` key, and neither a credential provider nor
+ * [[ClearTextFallbackKey]] is ever consulted. Modeled as a plain `Configuration#get` on the
+ * [[propagateBucketOptions]] result, which also expands `${...}` references under that
+ * propagated view exactly like the real consumer.
+ */
+ private[delta] case object PropagatedOptionConsumer extends S3ConfigConsumer
+
+ /**
+ * Every `fs.s3a.*` base key that governs whether a claimed native scan actually behaves like
+ * Hadoop's own reader would, paired with the consumer family Hadoop resolves it through -- ONE
+ * list, with each key's resolution tier declared beside it, so a key can never sit in the
+ * comparator without a deliberate classification (adding one without picking a tier does not
+ * compile). The entries are every per-bucket `fs.s3a.*` base key native's S3 client's
+ * `get_config` (s3.rs) resolves, verified directly against its call sites:
+ * `extract_s3_config_options` (endpoint.region, path.style.access, endpoint,
+ * requester.pays.enabled), `lookup_provider_class` (the Comet-specific
+ * credential-provider-class activation key), and
+ * `build_credential_provider`/`build_aws_credential_provider_metadata`/
+ * `build_assume_role_credential_provider_metadata` (aws.credentials.provider,
+ * assumed.role.credentials.provider, assumed.role.arn, assumed.role.session.name).
+ *
+ * Tier assignments, each verified via `javap`/CFR against `hadoop-aws` 3.3.4:
+ * - access.key/secret.key/session.token: `S3AUtils#getAWSAccessKeys` and
+ * `MarshalledCredentialBinding#fromFileSystem` (reached from
+ * `TemporaryAWSCredentialsProvider`) resolve all three via `S3AUtils#lookupPassword` --
+ * [[LookupPasswordConsumer]].
+ * - aws.credentials.provider and assumed.role.credentials.provider:
+ * `S3AUtils#buildAWSProviderList` -> `loadAWSProviderClasses` -> plain
+ * `Configuration#getClasses` -- [[PropagatedOptionConsumer]].
+ * - assumed.role.arn/session.name: `AssumedRoleCredentialProvider`'s constructor reads both
+ * via plain `Configuration#getTrimmed` -- [[PropagatedOptionConsumer]].
+ * - endpoint (`S3AFileSystem`: `getTrimmed`), endpoint.region (`DefaultS3ClientFactory`:
+ * `getTrimmed`), path.style.access (`S3AFileSystem`: `getBoolean`) --
+ * [[PropagatedOptionConsumer]].
+ * - requester.pays.enabled: not read anywhere in `hadoop-aws` 3.3.4 (the constant does not
+ * even exist in its `Constants` class); later releases read it via plain `getBoolean`
+ * against the propagated conf, so the plain tier is both the faithful forward model and
+ * inert on 3.3.4 -- [[PropagatedOptionConsumer]].
+ * - comet.credential.provider.class: Comet's own activation key, plain conf read on both
+ * sides, never a Hadoop key at all -- [[PropagatedOptionConsumer]].
+ *
+ * SYNC NOTE: the key list must stay a superset of native's `NATIVE_S3A_CONFIG_PROPERTIES`
+ * constant (`native/core/src/parquet/objectstore/s3.rs`, property suffixes without the
+ * `fs.s3a.` prefix) -- `DeltaScanContribSuite`'s discovery-harness test asserts this
+ * mechanically against [[AllS3ConfigKeys]]. Literal strings, not the
+ * [[AwsCredentialsProviderKey]] / [[AssumedRoleCredentialsProviderKey]] vals declared below,
+ * purely to avoid a forward reference inside this `object` body; kept textually identical to
+ * those two constants.
+ */
+ private[delta] val S3ConfigKeyConsumers: Seq[(String, S3ConfigConsumer)] = Seq(
Review Comment:
> Does the discovery harness in `DeltaScanContribSuite` catch either of these?
No, by design: the harness bounds the keys native reads, and both of these are keys Hadoop reads that native ignores. Added `hadoopOnlyEndpointGateReason` next to the proxy and session-policy gates: a scheme-less `fs.s3a.endpoint` with `fs.s3a.connection.ssl.enabled=false` declines, and either assumed-role STS endpoint key declines whenever set, both resolved on the propagated conf so per-bucket overrides apply. Six tests cover the decline, the SSL default, an endpoint carrying its own scheme, the per-bucket SSL override leaving another bucket claimable, both STS keys, and the empty-conf control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4006913895
##########
.github/workflows/delta_contrib_test.yml:
##########
@@ -0,0 +1,169 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Delta Contrib Tests
+
+# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
+# live in the umbrella workflow.
+on:
+ workflow_call:
+
+permissions:
+ contents: read
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: rust-lld cannot resolve -ljvm against the Zulu JDK
+ # layout installed by setup-java (same rationale as pr_build_linux.yml).
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+ # The container's default locale is POSIX, which makes the JVM's file-path encoder
+ # reject non-ASCII partition directory names the suites create.
+ LANG: "C.UTF-8"
+ LC_ALL: "C.UTF-8"
+
+jobs:
+
+ contrib-delta:
+ name: Delta contrib (Spark ${{ matrix.profile.spark }})
+ runs-on: ubuntu-24.04
+ container:
Review Comment:
> Could you either mount `/var/run/docker.sock` into this job (or run that one suite outside the container), or state in the workflow that the S3 suite is manual-only and drop it from the description's CI coverage claim?
Added a plain-runner job, `contrib-delta-s3`, that builds the same way and runs only `CometDeltaS3Suite` on Spark 3.5, where Testcontainers can start MinIO. The container matrix keeps the rest. I cannot exercise the workflow locally, so the first run here is the proof.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4006904426
##########
spark/src/main/scala/org/apache/comet/rules/CometScanContrib.scala:
##########
@@ -147,6 +154,21 @@ object CometScanContrib extends Logging {
"declining it and continuing with Comet's built-in handling",
e)
None
+ case e: LinkageError =>
+ // A version-skewed contrib jar (compiled against a Comet internal that has since
Review Comment:
> Could the discovery loop get the same `LinkageError` arm (log and skip), with a test alongside `FatalScanContrib` that drives discovery against a provider whose interface cannot load?
Added in `ContribServices.loadFrom`: a `LinkageError` from `hasNext` or `next` is logged with its subtype and skipped, so the lazy `contribs` initializer completes and every scan keeps working. The test lists a provider name behind a classloader that throws `NoClassDefFoundError` for it and asserts the linkable provider is still discovered and the warning names the subtype.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4007012049
##########
native/core/Cargo.toml:
##########
@@ -103,12 +106,25 @@ datafusion-functions-nested = { version = "55.1.0" }
[features]
backtrace = ["datafusion/backtrace"]
-default = ["hdfs-opendal"]
+default = ["hdfs-opendal", "delta"]
hdfs-opendal = ["opendal", "object_store_opendal", "hdfs-sys"]
jemalloc = ["tikv-jemallocator", "tikv-jemalloc-ctl"]
-# Delta Lake integration. When enabled, links the `comet-contrib-delta` crate
-# into `libcomet` and activates the `OpStruct::DeltaScan` dispatcher arm.
-# Default builds carry zero Delta surface.
+# Native Delta Lake scan support for the JVM-planned path (contrib/delta-spark).
+# In the default set: inert at runtime unless the contrib jar is on the
+# classpath (ServiceLoader) AND spark.comet.scan.delta.enabled is set, so it
+# cannot affect non-Delta scans. Opt out with --no-default-features for slim
Review Comment:
> Could this state the rationale directly rather than pointing at the PR review?
Rewritten: in the default set so trying the contrib needs only the jar and the config, inert without both, no new crates since `roaring` and `crc32fast` are already in the tree. The footprint number now lives only in `dev/verify-contrib-delta-gate.sh`, which the comment points at.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-6112359277
I added the `run-delta-tests` label, but #6059 has since landed and this now conflicts with `main` in `pom.xml` and `CometNativeScan.scala`, so the Delta jobs didn't start. Once you merge `main`, the next run will include them and I'll approve it. The `CometNativeScan` one needs a decision on your side, since `buildNativeScanCommon` and `populateScanConfFlags` still take `firstFileUri` while `convert` now uses `extractScanObjectStoreOptions` over every file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-6134975980
> @dwsmith1983 could you fix conflicts?
nm, I took the liberty of fixing them so we can get this merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5418790494
Consolidating the three outstanding issues from the review of `ef46cb8dd45891b7c2ab892ee9c35adaf05d4ce5` in one top-level comment:
1. **[P2] Column-mapping name collisions can silently return wrong data.** Write a Delta table `(a BIGINT, p BIGINT)` partitioned by `p`, enable name mapping, rename data column `a` to `b`, then rename partition column `p` to `a`. The scan mixes physical data names with logical partition names, so DataFusion replaces the data column with the partition constant. With an asserted native Delta scan, `SELECT b, a` returned `(100,100), (100,100)` instead of Spark's `(1,100), (2,100)`. Use consistent physical partition names, or fall back for ambiguous scans.
2. **[P2] Scalar-subquery partition filters can fail in a fused native operator.** After the scalar subquery resolves, `outputPartitioning` still reports zero partitions. A native parent reads that count and fails with `All per-partition arrays must have length 0`. Reproduced with AQE disabled using `SELECT id + 1, p FROM fact WHERE p = (SELECT max(p) FROM dim WHERE country = 'yes')`, with both a native Delta scan and native projection present. The partition count must reflect the resolved execution data.
3. **[P2] Hadoop credential references are forwarded unresolved.** With `SimpleAWSCredentialsProvider` and access/secret keys referencing `${review.access}` / `${review.secret}`, Hadoop resolves the configured values, but Delta admission accepts the scan and native option extraction retains the literal placeholders. Native credential lookup does not expand them. Resolve the values consistently with Hadoop, or fall back. This was verified against the actual Hadoop provider and current admission/extraction helpers plus the production call path; this credential probe did not make a live S3 request.
The two query failures above were reproduced on `chao-reviews-1` with Spark 4.0.4 / Delta 4.0.1. These are bounded P2 correctness/compatibility issues, not broad P0/P1 failures; the silent wrong-result case is the highest priority. The existing 43 native tests and 100 contrib tests passed, but do not cover these cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5419684276
On the long bucket credential precedence:
You're right, long wins. S3AUtils.getPassword keeps an already non-empty value, so the long form resolves first and the short form only fills in when it's unset. Flipped the comparison to long, short, global and added your exact conflicting fixture as a regression (distinct long and short pairs now decline). Two related hardenings went in alongside: the gate now declines when fs.s3a.aws.credentials.provider resolves to a class the native reader doesn't implement, so those scans fall back instead of failing mid query, and for non-credential options (endpoint etc.) we verified the long form is inert in Hadoop (propagateBucketOptions folds it into an unread key), so no gate is needed there; that's documented with a control test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3835075936
##########
native/core/src/execution/planner/delta_spark_scan.rs:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! JVM-planned Delta handler for the generic `OpStruct::ContribScan` dispatcher, feature-gated
+//! behind `delta`.
+//!
+//! delta-spark has already done log replay, snapshot resolution, and partition pruning by the
+//! time the scan reaches Comet, so the envelope carries a concrete file list (plus deletion
+//! vector descriptors) and the read path reuses the exact same shared parquet scan builder as
+//! `NativeScan` -- inheriting row-group stats pruning, page-index pruning, and filter pushdown.
+//! Sibling of the kernel-planned handler in `delta_scan.rs`; the two claim different
+//! `type_url`s within the same `ContribScan` envelope.
+
+use std::collections::HashMap;
+
+use datafusion_comet_proto::spark_operator::{
+ ContribScan, DeltaSparkScan, Operator, SparkFilePartition,
+};
+use prost::Message;
+
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::execution::planner::PhysicalPlanner;
+use crate::execution::planner::PlanCreationResult;
+
+/// Type name the JVM-planned Delta contrib claims within the `ContribScan` envelope. The
+/// contrib jar packs a `DeltaSparkScan` with a `type_url` of
+/// `type.googleapis.com/comet.contrib.delta_spark.DeltaSparkScan`; dispatch keys on the
+/// contrib-owned suffix, same convention as the kernel path's `delta_scan.rs`.
+const DELTA_SPARK_SCAN_TYPE_NAME: &str = "comet.contrib.delta_spark.DeltaSparkScan";
+
+/// Contrib entry point for the `OpStruct::ContribScan` dispatcher. Returns `Some(result)` when
+/// the envelope carries a JVM-planned Delta scan, or `None` when the `type_url` belongs to some
+/// other contrib.
+pub(crate) fn try_plan_contrib_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ contrib: &ContribScan,
+) -> Option<PlanCreationResult> {
+ if !contrib.type_url.ends_with(DELTA_SPARK_SCAN_TYPE_NAME) {
+ return None;
+ }
+ Some(
+ DeltaSparkScan::decode(contrib.value.as_slice())
+ .map_err(|e| {
+ GeneralError(format!(
+ "Failed to decode DeltaSparkScan from contrib_scan: {e}"
+ ))
+ })
+ .and_then(|scan| plan_delta_spark_scan(planner, spark_plan, &scan)),
+ )
+}
+
+fn plan_delta_spark_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ scan: &DeltaSparkScan,
+) -> PlanCreationResult {
+ // Delta data files are plain parquet; the read path deliberately reuses
+ // the same shared parquet scan builder as NativeScan so Delta inherits
+ // row-group stats pruning, page-index pruning, and filter pushdown. Only
+ // the file list arrives in Delta-specific form. Note delta_common's
+ // column_mapping_mode is informational in M1: the actual field-id
+ // matching switch is common.use_field_id, same as the Iceberg path.
+ let common = scan
+ .common
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing common data".into()))?;
+
+ let delta_partition = scan
+ .file_partition
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing file_partition".into()))?;
+
+ let spark_partition = SparkFilePartition {
+ partitioned_file: delta_partition
+ .partitioned_file
+ .iter()
+ .map(|f| {
+ f.file
+ .clone()
+ .ok_or_else(|| GeneralError("DeltaSparkPartitionedFile missing inner file".into()))
+ })
+ .collect::<Result<Vec<_>, _>>()?,
+ };
+
+ let (object_store_url, mut files) =
+ planner.prepare_scan_store_and_files(common, &spark_partition)?;
Review Comment:
**[P1] Also contain cross-container reuse across separate scans**
The new guard fixes the within-scan collision, but the process-wide cache can still reuse the wrong container across separate scans. Two scans of `abfss://source@account/...` and `abfss://target@account/...` each pass the single-authority check, while the cache key omits the container.
An offline probe using the current cache and Azure builder confirmed that the second scan receives the first container's store, even with a separate `RuntimeEnv`. This can fail with a missing object or read the wrong same-named object. No live Azure request was made.
Could we conservatively keep these Azure scans on Spark until the shared cache and registry preserve container identity? That would contain this remaining case without requiring a broader object-store refactor here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5306420769
HI @andygrove,
Can you review this as it adds Delta functionality?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5371510656
CI notes for this push: the S3 test base built its client without a region, which aborted CometDeltaS3Suite in CI's empty AWS environment before any test ran; fixed, and since GitHub mounts the Docker socket into job containers the MinIO scenarios will actually execute in CI now. They've been run live locally with all AWS env vars unset (first executions ever, both pass on Spark 3.5 and 4.1; that surfaced a missing spark-hadoop-cloud test dependency, also fixed). Heads up that inside the job container the MinIO endpoint may resolve as unreachable sibling-container networking; the suite now fails soft to canceled rather than aborting the build, and logs the resolved endpoint so the first CI run tells us whether a testcontainers host override is needed. The Spark 4.0 cell wasn't rerun locally, so CI is its first pass over these changes. The Rust 1.98 clippy fix I'd pushed got dropped in favor of #5400 from main during rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3836543179
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme (finding 8).
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
Review Comment:
**[P2] Also check Hadoop's long bucket credential aliases**
The new fallback catches the original global and short bucket aliases, but one case remains at `92ae71b8`. Hadoop's `S3AUtils.lookupPassword` also accepts these long bucket aliases, which are absent from [`s3aCredentialAliases`](https://github.com/apache/datafusion-comet/blob/92ae71b8f7bb9631dd151c0e3df522a36754b9ae/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala#L567-L574):
```text
fs.s3a.bucket.mybucket.fs.s3a.access.key
fs.s3a.bucket.mybucket.fs.s3a.secret.key
```
With only those aliases in a globally configured JCEKS and `SimpleAWSCredentialsProvider` explicitly selected, Hadoop resolves the credentials while the current `credentialAliasReason` returns `None`. `NativeConfig` forwards none of the keystore entries, and the native static provider only checks short bucket aliases and globals. A fresh native scan therefore still has no credentials. If different global plain credentials are also configured, Hadoop uses the long aliases but native uses those different globals instead.
I reproduced both cases with the unchanged current admission/extraction methods, Hadoop 3.4.1, and a real local JCEKS. The short bucket alias control correctly declines. The native consequence was checked in source, not through a full JNI scan or live S3 request.
Could we account for the long aliases when deciding whether native credentials preserve Hadoop's resolution, and add long-only and global-shadowing regressions? A conservative fallback is sufficient here. This is a remaining case of the existing P2, not a separate finding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3869521703
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1000 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.ListBuffer
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable. Only called when
+ * [[isDeltaScan]] is true. `scanHelper` is the [[CometScanExec]] built to drive
+ * [[CometDeltaNativeScan.convert]] on a claim, reused for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`: it runs at most
+ // once per claim attempt, and not at all for the common non-DV-shape case.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated.
+ val cmMode = metadata.columnMappingMode.name
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bounds native's memory for expanded DV row selectors (delta_dv.rs), pessimistically
+ // bounded by 2*cardinality + #row-groups; the conf below makes an over-pessimistic decline
+ // recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities, but the native builder resolves
+ // ObjectStoreUrl from only the FIRST selected file; force file listing and decline rather than
+ // risk reading a later file through the wrong handle.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // GCS's zero-I/O, conf-only credential-forwarding gate; ordered alongside the S3 credential
+ // gates below since all presume a single, well-formed store identity per URI.
+ val gcsAuthReason = gcsHadoopOnlyAuthReason(hadoopConf, dataFileUris ++ dvUris)
+ if (gcsAuthReason.isDefined) {
+ return gcsAuthReason
+ }
+
+ // Credentials that resolve ONLY through a Hadoop credential provider (JCEKS et al.) are
+ // invisible to the plain-conf extraction forwarded to the native S3 client; reuses hadoopConf
+ // from the encryption gate above.
+ val credentialReason = credentialAliasReason(hadoopConf, dataFileUris ++ dvUris)
+ if (credentialReason.isDefined) {
+ return credentialReason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, normalized to
+ * absolute on-disk paths. Returns `Seq.empty` for the plain shape. Shared by the DV cardinality
+ * gate and [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged, since such a URI cannot come from a Hadoop-backed source.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt).
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme + lowercased raw
+ * authority, so e.g. `S3A://Bucket` and `s3a://bucket` collapse), or `None` when they share
+ * one. `file://` paths carry no authority, so local scans across many directories are
+ * unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the parsed host/port/userinfo fields: `getHost` (and `getUserInfo`/`getPort`)
+ * return `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an
+ * underscore in a GCS bucket name, `gs://my_bucket`), which would silently collapse distinct
+ * buckets into one empty-host key. A `null` authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`, which (like [[uriAuthority]]'s getters) returns `null`
+ * for the whole authority on an RFC 3986 `reg-name` violation. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme`, then `://`, then a literal `***` masking userinfo,
+ * then `@host[:port]`, for embedding in a decline reason. NEVER interpolate `uri.getAuthority`
+ * or [[uriUserInfo]] directly into a reason string: doing so would leak credentials embedded as
+ * URI userinfo into the SQL plan's explain output, fallback-reason logging, or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted by [[credentialAliasReason]] below. `hadoop-aws` is
+ * NOT on this module's runtime classpath, so `org.apache.hadoop.fs.s3a.Constants` must never be
+ * referenced here (would raise `NoClassDefFoundError` for sessions with no S3 dependency).
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key; both
+ * must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ /**
+ * The aliases native's S3 credentials provider chain tries per bucket (short, then global),
+ * PLUS the LONG form `S3AUtils#lookupPassword` also consults. Hadoop resolves long before short
+ * before global; native reads only short+global, so a JCEKS entry set ONLY under the long alias
+ * is invisible to native -- a shadowed value under ANY alias here means decline.
+ */
+ private def s3aCredentialAliases(bucket: String): Seq[String] =
+ Seq(
+ s"fs.s3a.bucket.$bucket.fs.s3a.access.key",
+ s"fs.s3a.bucket.$bucket.fs.s3a.secret.key",
+ s"fs.s3a.bucket.$bucket.fs.s3a.session.token",
+ s"fs.s3a.bucket.$bucket.access.key",
+ s"fs.s3a.bucket.$bucket.secret.key",
+ s"fs.s3a.bucket.$bucket.session.token",
+ "fs.s3a.access.key",
+ "fs.s3a.secret.key",
+ "fs.s3a.session.token")
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority, or `None` when
+ * `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority manually rather than
+ * `URI#getHost`, avoiding the same RFC 3986 `reg-name` pitfall as [[uriAuthority]].
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ /**
+ * The three per-bucket S3A credential base keys the native S3 client's `get_config` (`s3.rs`)
+ * resolves: short bucket key first, then global (native never reads the long bucket key).
+ */
+ private val PlainCredentialBaseKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket`. Used by the credential-provider-class gates below, which read S3A options the same
+ * way native does.
+ */
+ private def effectiveOptionValue(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ private def plainLongFormCredentialDivergenceReason(bucket: String, longKey: String): String =
+ "Native Delta scan cannot forward long-form bucket credentials for " +
+ s"$bucket ($longKey is set but the native S3 client only reads the short bucket and " +
+ "global keys, so its credentials would differ from Hadoop's)"
+
+ /**
+ * Zero-I/O plain-value divergence check: native's `get_config` never reads the long form, so it
+ * can silently diverge from Hadoop's effective value (`getPassword` lets a non-empty long value
+ * WIN over short/global). Declines when the long form is set and the two sides disagree; never
+ * interpolates a resolved value, only the key name.
+ *
+ * Credentials only: every OTHER `fs.s3a` option flows through `propagateBucketOptions`, which
+ * strips only ONE `fs.s3a.bucket.B.` prefix layer, folding a long-form key into
+ * `fs.s3a.fs.s3a.<key>` (inert, unread elsewhere), so Hadoop's effective value already reduces
+ * to short.orElse(global) like native. No gate is needed for that set.
+ */
+ private def plainLongFormCredentialReason(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ PlainCredentialBaseKeys.foldLeft(Option.empty[String]) { (declined, baseKey) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val long = plainValue(hadoopConf, longKey)
+ if (long.isEmpty) {
+ None
+ } else {
+ val short = plainValue(hadoopConf, shortKey)
+ val global = plainValue(hadoopConf, baseKey)
+ val hadoopEffective = long.orElse(short).orElse(global)
+ val nativeEffective = short.orElse(global)
+ if (hadoopEffective != nativeEffective) {
+ Some(plainLongFormCredentialDivergenceReason(bucket, longKey))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * String-literal mirror of every credential-provider class name s3.rs's
+ * `build_aws_credential_provider_metadata` recognizes (Hadoop S3A plus AWS SDK v1/v2 names).
+ * `hadoop-aws` is NOT on this module's runtime classpath, so these stay string literals, never
+ * `classOf` references.
+ */
+ private val SupportedCredentialProviderClasses: Set[String] = Set(
+ "org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider",
+ "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider",
+ "com.amazonaws.auth.ContainerCredentialsProvider",
+ "com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper",
+ "software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider",
+ "com.amazonaws.auth.InstanceProfileCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider",
+ "com.amazonaws.auth.EnvironmentVariableCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider",
+ "com.amazonaws.auth.WebIdentityTokenCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider",
+ "com.amazonaws.auth.profile.ProfileCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider",
+ "com.amazonaws.auth.AnonymousAWSCredentials")
+
+ private val AnonymousCredentialProviderClasses: Set[String] = Set(
+ "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider",
+ "com.amazonaws.auth.AnonymousAWSCredentials")
+
+ private val HadoopAssumedRoleProviderClass =
+ "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider"
+
+ private val AwsCredentialsProviderKey = "fs.s3a.aws.credentials.provider"
+ private val AssumedRoleCredentialsProviderKey = "fs.s3a.assumed.role.credentials.provider"
+
+ /** Splits a comma-separated credential-provider-class list the same way s3.rs's parser does. */
+ private def parseProviderClassNames(value: String): Seq[String] =
+ value.split(",").map(_.trim).filter(_.nonEmpty).toSeq
+
+ private def unsupportedProviderReason(bucket: String, key: String, className: String): String =
+ s"Native Delta scan does not support the credential provider class $className " +
+ s"configured via $key for $bucket (the native S3 client only supports a fixed set of " +
+ "provider classes; an unsupported class would fail at scan execution time, after the " +
+ "scan was already claimed, rather than at planning time)"
+
+ private def mixedAnonymousProviderReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key for $bucket naming an anonymous credential " +
+ "provider together with any other provider (the native S3 client rejects this " +
+ "combination at scan execution time)"
+
+ private def anonymousAssumedRoleProviderReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support an anonymous credential provider in $key for " +
+ s"$bucket (the native S3 client does not allow an anonymous provider as the base " +
+ "credentials for an assumed-role chain)"
+
+ private def unsupportedProviderNameReason(
+ bucket: String,
+ key: String,
+ names: Seq[String]): Option[String] =
+ names
+ .find(name => !SupportedCredentialProviderClasses.contains(name))
+ .map(unsupportedProviderReason(bucket, key, _))
+
+ /**
+ * Decline reason when `bucket`'s effective `assumed.role.credentials.provider` names an
+ * unsupported class, or an anonymous one (native rejects ANY anonymous entry here, not just a
+ * mix). Unset defaults to native's own always-supported fallback, so `None` is safe.
+ */
+ private def assumedRoleProviderClassReason(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ effectiveOptionValue(hadoopConf, bucket, AssumedRoleCredentialsProviderKey) match {
+ case None => None
+ case Some(value) =>
+ val names = parseProviderClassNames(value)
+ unsupportedProviderNameReason(bucket, AssumedRoleCredentialsProviderKey, names).orElse {
+ if (names.exists(AnonymousCredentialProviderClasses.contains)) {
+ Some(anonymousAssumedRoleProviderReason(bucket, AssumedRoleCredentialsProviderKey))
+ } else {
+ None
+ }
+ }
+ }
+ }
+
+ /**
+ * Decline reason when `bucket`'s effective `aws.credentials.provider` names an unrecognized
+ * class, mixes an anonymous provider with any other, or a nested
+ * `AssumedRoleCredentialProvider` sub-chain has the same problem. Unset/empty falls back to
+ * native's default chain.
+ */
+ private def providerClassReason(hadoopConf: Configuration, bucket: String): Option[String] = {
+ effectiveOptionValue(hadoopConf, bucket, AwsCredentialsProviderKey).flatMap { value =>
+ val names = parseProviderClassNames(value)
+ unsupportedProviderNameReason(bucket, AwsCredentialsProviderKey, names)
+ .orElse {
+ if (names.length > 1 && names.exists(AnonymousCredentialProviderClasses.contains)) {
+ Some(mixedAnonymousProviderReason(bucket, AwsCredentialsProviderKey))
+ } else {
+ None
+ }
+ }
+ .orElse {
+ if (names.contains(HadoopAssumedRoleProviderClass)) {
+ assumedRoleProviderClassReason(hadoopConf, bucket)
+ } else {
+ None
+ }
+ }
+ }
+ }
+
+ /** First reason any bucket among `uris` names an unsupported credential-provider class. */
+ private[delta] def providerClassGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) declined else providerClassReason(hadoopConf, bucket)
+ }
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def shadowedCredentialAliasReason(bucket: String, alias: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($alias resolves through $HadoopCredentialProviderPathKey but is not present " +
+ "as a plain configuration value, so the native S3 client would have no credentials)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * Compares each [[s3aCredentialAliases]] alias's plain `Configuration#get` against
+ * `Configuration#getPassword` (providers first, plain conf fallback); a difference means
+ * decline. Runs in try/catch: `getPassword` does real keystore I/O, and a corrupt/unreadable
+ * store must decline this bucket, not abort planning for the whole session.
+ */
+ private def verifyGlobalProviderAliases(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ try {
+ s3aCredentialAliases(bucket).foldLeft(Option.empty[String]) { (declined, alias) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ val resolved =
+ Option(hadoopConf.getPassword(alias)).map(new String(_)).filter(_.nonEmpty)
+ resolved match {
+ case Some(value) if !Option(hadoopConf.get(alias)).contains(value) =>
+ Some(shadowedCredentialAliasReason(bucket, alias))
+ case _ => None
+ }
+ }
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableCredentialProviderReason(bucket, e))
+ }
+ }
+
+ /**
+ * The decline reason for `bucket` alone, or `None` when its credentials are safe to forward.
+ * [[plainLongFormCredentialReason]] is checked first (zero-I/O, decisive on its own); then
+ * `None` if no provider-path key is set. Arm A: an S3A-scoped provider-path key points at a
+ * provider `getPassword` does not consult, so any being set declines with no keystore read. Arm
+ * B: only the global path is set (which `getPassword` DOES consult); delegates to
+ * [[verifyGlobalProviderAliases]].
+ */
+ private def bucketCredentialAliasReason(
+ hadoopConf: Configuration,
+ bucket: String,
+ globalPathSet: Boolean,
+ s3aPathSet: Boolean): Option[String] = {
+ plainLongFormCredentialReason(hadoopConf, bucket).orElse {
+ val bucketPathKey = s3aBucketProviderPathKey(bucket)
+ val bucketLongPathKey = s3aBucketLongProviderPathKey(bucket)
+ val bucketPathSet = nonEmptyConf(hadoopConf, bucketPathKey)
+ val bucketLongPathSet = nonEmptyConf(hadoopConf, bucketLongPathKey)
+ if (!globalPathSet && !s3aPathSet && !bucketPathSet && !bucketLongPathSet) {
+ None
+ } else if (s3aPathSet || bucketPathSet || bucketLongPathSet) {
+ val offendingKey =
+ if (s3aPathSet) S3aCredentialProviderPathKey
+ else if (bucketPathSet) bucketPathKey
+ else bucketLongPathKey
+ Some(s3aScopedProviderPathReason(bucket, offendingKey))
+ } else {
+ verifyGlobalProviderAliases(hadoopConf, bucket)
+ }
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf key prefix for GCS authentication options (`fs.gs.auth.*`).
+ * `gcs-connector` is NOT on this module's runtime classpath by default, so referencing an
+ * actual GCS auth class would risk `NoClassDefFoundError`, same rationale as the S3A literals
+ * above.
+ */
+ private val GcsAuthKeyPrefix = "fs.gs.auth."
+
+ /**
+ * True when `uri`'s scheme is `gs` (case-insensitive) -- the ONLY scheme object_store's
+ * `ObjectStoreScheme::parse` (parquet_support.rs) routes to `GoogleCloudStorage`; `gcs` is not
+ * recognized there and is deliberately excluded.
+ */
+ private def isGcsScheme(uri: URI): Boolean =
+ Option(uri.getScheme).exists(_.equalsIgnoreCase("gs"))
+
+ /**
+ * The lowercase-scheme-checked GCS bucket name from `uri`'s authority (host, minus any userinfo
+ * or port), or `None` when `uri`'s scheme is not `gs`. Parses the raw authority manually,
+ * mirroring [[s3Bucket]]'s `URI#getHost`/RFC 3986 `reg-name` reasoning.
+ */
+ private def gcsBucket(uri: URI): Option[String] = {
+ if (!isGcsScheme(uri)) {
+ None
+ } else {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ }
+ }
+
+ /**
+ * The non-empty `fs.gs.auth.*` Hadoop conf keys set on `hadoopConf`, full key names only --
+ * NEVER their values, which are credential material and must never enter a decline reason.
+ * Iterates the conf map directly: no provider resolution, no I/O.
+ */
+ private def gcsAuthKeys(hadoopConf: Configuration): Seq[String] =
+ hadoopConf
+ .iterator()
+ .asScala
+ .collect {
+ case entry
+ if entry.getKey.startsWith(GcsAuthKeyPrefix) &&
+ entry.getValue != null && entry.getValue.nonEmpty =>
+ entry.getKey
+ }
+ .toSeq
+ .distinct
+ .sorted
+
+ /**
+ * Decline reason when any of `uris` resolves to a `gs://` authority AND `hadoopConf` sets any
+ * `fs.gs.auth.*` key, or `None` when claimable. Native forwards none of `fs.gs.*` to the object
+ * store, so a scan relying solely on Hadoop-side GCS credentials would claim here but then fail
+ * authentication natively. Application Default Credentials work identically in both engines and
+ * need no Hadoop conf key, so an ADC-only configuration still claims. Never interpolates a
+ * resolved value, only key names.
+ */
+ private[delta] def gcsHadoopOnlyAuthReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val gcsUris = uris.filter(isGcsScheme)
+ if (gcsUris.isEmpty) {
+ return None
+ }
+ val authKeys = gcsAuthKeys(hadoopConf)
+ if (authKeys.isEmpty) {
+ return None
+ }
+ val buckets = gcsUris.flatMap(gcsBucket).distinct.sorted
+ Some(
+ "Native Delta scan does not support GCS authentication configured only via Hadoop conf " +
+ s"key(s) ${authKeys.mkString(", ")} for gs://${buckets.mkString(", gs://")} " +
+ "(the native GCS client does not forward fs.gs.* options; only Application Default " +
+ "Credentials -- environment or metadata-server -- are available natively)")
+ }
+
+ /**
+ * First reason a native S3 scan cannot faithfully forward this table's Hadoop credentials, or
+ * `None` when claimable. Only `s3`/`s3a` authorities matter here (ABFS/WASB mooted by the
+ * userinfo gate, GCS handled by [[gcsHadoopOnlyAuthReason]]). `Configuration#getPassword`
+ * checks every provider first, falling back to plain conf only when unset, so a keystore-only
+ * or shadowing alias is invisible/wrong to native's plain-conf extraction; when in doubt,
+ * decline. Never interpolates a resolved value, only key names.
+ */
+ private[delta] def credentialAliasReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ if (buckets.isEmpty) {
+ return None
+ }
+ val globalPathSet = nonEmptyConf(hadoopConf, HadoopCredentialProviderPathKey)
+ val s3aPathSet = nonEmptyConf(hadoopConf, S3aCredentialProviderPathKey)
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ bucketCredentialAliasReason(hadoopConf, bucket, globalPathSet, s3aPathSet)
+ }
+ }
+ }
+
+ /**
+ * True when `dataType` is, or structurally contains (through array elements or map keys/
+ * values), a [[StructType]]. Only [[StructType]] fields carry Delta's physical, column-mapped
+ * names; array/map labels themselves are never column-mapped.
+ */
+ private def containsNestedStruct(dataType: DataType): Boolean = dataType match {
+ case _: StructType => true
+ case ArrayType(elementType, _) => containsNestedStruct(elementType)
+ case MapType(keyType, valueType, _) =>
+ containsNestedStruct(keyType) || containsNestedStruct(valueType)
+ case _ => false
+ }
+
+ /**
+ * True when `node` is a positional-output union -- `UnionExec` or `CometUnionExec`. Both
+ * compute output positionally from the FIRST child's attributes, so a value carried only by a
+ * LATER branch needs an explicit positional walk below. Compared by class name (the
+ * [[isDeltaScan]] idiom) to avoid a compile-time dependency; an unmatched name is still safe,
+ * caught by the generic multi-child safety net below.
+ */
+ private def isPositionalUnion(node: SparkPlan): Boolean = {
+ val name = node.getClass.getSimpleName
+ name == "UnionExec" || name == "CometUnionExec"
+ }
+
+ /**
+ * True when the scan's row-index column value is provably dead above the scan. The standard DV
+ * plan shape routes it only into a `named_struct(... row_index ...) AS _metadata` projection
+ * whose result the final projection discards; anything else (a query actually selecting
+ * `_metadata.row_index`) makes the value live and must decline. Conservative: any unrecognized
+ * consumption pattern returns false.
+ */
+ private def rowIndexUnusedAbove(plan: SparkPlan, scanExec: FileSourceScanExec): Boolean = {
+ val rowIndexAttrs = scanExec.output
+ .filter(_.name == CometDeltaNativeScan.RowIndexColumn)
+ .map(_.exprId)
+ .toSet
+ if (rowIndexAttrs.isEmpty) {
+ return true
+ }
+ // Transitive taint analysis: everything derived from the row-index attribute within the
+ // visible plan, via Project aliases or positionally across a union. The plan may be an AQE
+ // stage fragment, so tainted values escaping to the fragment's own output must decline too.
+ var tainted = rowIndexAttrs
+ var changed = true
+ while (changed) {
+ changed = false
+ plan.foreach {
+ case p: ProjectExec =>
+ p.projectList.foreach {
+ case a: Alias
+ if !tainted.contains(a.exprId) &&
+ a.references.exists(r => tainted.contains(r.exprId)) =>
+ tainted += a.exprId
+ changed = true
+ case _ =>
+ }
+ case u if isPositionalUnion(u) =>
+ // Output attributes carry the FIRST child's expression IDs, so a value tainted only in
+ // a LATER branch is otherwise invisible; walk it forward positionally instead.
+ // `children` can be re-parented by AQE after `output` is frozen, so an arity mismatch on
+ // ANY child (which would make a positional zip silently truncate) forces a decline.
+ if (u.children.exists(_.output.length != u.output.length)) {
+ return false
+ }
+ u.children.foreach { child =>
+ child.output.zip(u.output).foreach {
+ case (from, to) if tainted.contains(from.exprId) && !tainted.contains(to.exprId) =>
+ tainted += to.exprId
+ changed = true
+ case _ =>
+ }
+ }
+ case _ =>
+ }
+ }
+ val nonProjectConsumer = plan.exists {
+ case _: ProjectExec => false
+ case n if n ne scanExec =>
+ n.expressions.exists(_.references.exists(r => tainted.contains(r.exprId)))
+ case _ => false
+ }
+ val escapes = plan.output.exists(a => tainted.contains(a.exprId))
+ // Generic safety net for every OTHER multi-child node (joins, etc; positional unions are
+ // exempt, already handled precisely above). A tainted attribute a child contributes must
+ // either survive into the node's own output under the SAME expression ID or be consumed by
+ // one of the node's own expressions; otherwise decline (e.g. a LEFT SEMI/ANTI join dropping
+ // the side carrying the tainted attribute).
+ val multiChildLeak = plan.exists {
+ case u if isPositionalUnion(u) => false
+ case n if n.children.size >= 2 =>
Review Comment:
Fixed. The liveness walk now treats any node that drops the column from its own output as a consumer unless it's a Project, whose alias tracking already decides that case, so one-child write sinks decline. Your workload reproduced the ri=0 corruption here and is now the regression: the write declines and the readback shows (31,31) with sum 475. One note on coverage: an end-to-end DSv2 write test turned out to be blocked by delta-spark itself (its DV planning requires a pinned TahoeLogFileIndex under DSv2 writes), so DSv2 sinks are covered by the structural rule and a test pins that upstream limitation so a future delta upgrade surfaces it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5346521797
Thanks @dwsmith1983. This makes sense to me! #4952 has just been merged. Could you rebase this PR and adapt to it? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3806837471
##########
.github/workflows/delta_contrib_test.yml:
##########
@@ -0,0 +1,124 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Delta Contrib Tests
+
+# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
+# live in the umbrella workflow.
+on:
+ workflow_call:
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: rust-lld cannot resolve -ljvm against the Zulu JDK
+ # layout installed by setup-java (same rationale as pr_build_linux.yml).
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+
+jobs:
+
+ contrib-delta:
+ name: Delta contrib (Spark ${{ matrix.profile.spark }})
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ strategy:
+ matrix:
+ profile:
+ - spark: "3.5"
+ java_version: "17"
+ - spark: "4.0"
+ java_version: "17"
+ - spark: "4.1"
+ java_version: "17"
+ # spark-4.2 is intentionally absent: the contrib profile is dormant
+ # until a Delta release supports Spark 4.2.
+ fail-fast: false
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: ${{ matrix.profile.java_version }}
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-${{ matrix.profile.spark }}
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Build native library with the delta feature
+ run: |
+ cd native
+ cargo build --features delta
+
+ - name: Install Comet core jars
+ run: |
+ ./mvnw -B -q -Pspark-${{ matrix.profile.spark }} install -pl common,spark -DskipTests -Dspotless.check.skip=true
+
+ - name: Run Delta contrib test suites
+ run: |
+ SPARK_HOME=$(pwd) COMET_CONF_DIR=$(pwd)/conf ./mvnw -B -Pspark-${{ matrix.profile.spark }},delta test -pl contrib/delta
+
+ # `delta` is in the default feature set, so no regular job builds without it;
+ # this keeps the feature-off build and its "built without the delta feature"
+ # error arm (planner.rs cfg(not(feature = "delta"))) from becoming dead code.
+ feature-off-build:
+ name: Feature-off native build
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: "17"
+
+ - name: Test the delta-off error path
+ run: |
+ cd native
+ cargo test -p datafusion-comet --no-default-features --features hdfs-opendal delta_scan
+
+ # The contrib's dev tooling (benchmark and regression-harness scripts) is
+ # Python; keep it import-clean across currently supported interpreters.
+ dev-scripts-python:
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}}
[Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/128)
##########
.github/workflows/delta_contrib_test.yml:
##########
@@ -0,0 +1,124 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Delta Contrib Tests
+
+# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
+# live in the umbrella workflow.
+on:
+ workflow_call:
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: rust-lld cannot resolve -ljvm against the Zulu JDK
+ # layout installed by setup-java (same rationale as pr_build_linux.yml).
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+
+jobs:
+
+ contrib-delta:
+ name: Delta contrib (Spark ${{ matrix.profile.spark }})
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ strategy:
+ matrix:
+ profile:
+ - spark: "3.5"
+ java_version: "17"
+ - spark: "4.0"
+ java_version: "17"
+ - spark: "4.1"
+ java_version: "17"
+ # spark-4.2 is intentionally absent: the contrib profile is dormant
+ # until a Delta release supports Spark 4.2.
+ fail-fast: false
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: ${{ matrix.profile.java_version }}
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-${{ matrix.profile.spark }}
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Build native library with the delta feature
+ run: |
+ cd native
+ cargo build --features delta
+
+ - name: Install Comet core jars
+ run: |
+ ./mvnw -B -q -Pspark-${{ matrix.profile.spark }} install -pl common,spark -DskipTests -Dspotless.check.skip=true
+
+ - name: Run Delta contrib test suites
+ run: |
+ SPARK_HOME=$(pwd) COMET_CONF_DIR=$(pwd)/conf ./mvnw -B -Pspark-${{ matrix.profile.spark }},delta test -pl contrib/delta
+
+ # `delta` is in the default feature set, so no regular job builds without it;
+ # this keeps the feature-off build and its "built without the delta feature"
+ # error arm (planner.rs cfg(not(feature = "delta"))) from becoming dead code.
+ feature-off-build:
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}}
[Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/127)
##########
.github/workflows/delta_contrib_test.yml:
##########
@@ -0,0 +1,124 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Delta Contrib Tests
+
+# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
+# live in the umbrella workflow.
+on:
+ workflow_call:
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: rust-lld cannot resolve -ljvm against the Zulu JDK
+ # layout installed by setup-java (same rationale as pr_build_linux.yml).
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+
+jobs:
+
+ contrib-delta:
+ name: Delta contrib (Spark ${{ matrix.profile.spark }})
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ strategy:
+ matrix:
+ profile:
+ - spark: "3.5"
+ java_version: "17"
+ - spark: "4.0"
+ java_version: "17"
+ - spark: "4.1"
+ java_version: "17"
+ # spark-4.2 is intentionally absent: the contrib profile is dormant
+ # until a Delta release supports Spark 4.2.
+ fail-fast: false
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: ${{ matrix.profile.java_version }}
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-${{ matrix.profile.spark }}
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Build native library with the delta feature
+ run: |
+ cd native
+ cargo build --features delta
+
+ - name: Install Comet core jars
+ run: |
+ ./mvnw -B -q -Pspark-${{ matrix.profile.spark }} install -pl common,spark -DskipTests -Dspotless.check.skip=true
+
+ - name: Run Delta contrib test suites
+ run: |
+ SPARK_HOME=$(pwd) COMET_CONF_DIR=$(pwd)/conf ./mvnw -B -Pspark-${{ matrix.profile.spark }},delta test -pl contrib/delta
+
+ # `delta` is in the default feature set, so no regular job builds without it;
+ # this keeps the feature-off build and its "built without the delta feature"
+ # error arm (planner.rs cfg(not(feature = "delta"))) from becoming dead code.
+ feature-off-build:
+ name: Feature-off native build
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: "17"
+
+ - name: Test the delta-off error path
+ run: |
+ cd native
+ cargo test -p datafusion-comet --no-default-features --features hdfs-opendal delta_scan
+
+ # The contrib's dev tooling (benchmark and regression-harness scripts) is
+ # Python; keep it import-clean across currently supported interpreters.
+ dev-scripts-python:
+ name: Delta dev scripts (Python ${{ matrix.python-version }})
+ runs-on: ubuntu-24.04
+ strategy:
+ matrix:
+ python-version: ["3.11", "3.12", "3.13", "3.14"]
+ fail-fast: false
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Python
+ uses: actions/setup-python@v6
+ with:
+ python-version: ${{ matrix.python-version }}
+
+ - name: Byte-compile contrib dev scripts
+ run: |
+ python -m compileall -q contrib/delta/dev
Review Comment:
## CodeQL / Workflow does not contain permissions
Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {{contents: read}}
[Show more details](https://github.com/apache/datafusion-comet/security/code-scanning/129)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5666809558
Thanks for the second pass. Items 1 through 9 are all refactors of the shared scan path rather than defects in this PR: sharing the admission checks with core, reusing the common builder in the DV path, an explicit file-planning boundary, envelope and provider contract tests, shared metadata preparation with the scan's accounting, and delegating schema preparation to Delta's `prepareSchemaForRead`. Each one reaches into code core and the Iceberg path also use, so doing them here would widen a PR that has already been through two correctness rounds. I would like to land this as is and file each of them as a follow-up issue linked from here, with 6 and 7 first since they pin behaviour the others would refactor under. Is any single one of them a blocker for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4006974272
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,2096 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::mem::size_of;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::{ParquetAccessPlan, RowGroupAccess};
+use datafusion::execution::memory_pool::{MemoryConsumer, MemoryReservation};
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::path::Path;
+use object_store::{ObjectStore, ObjectStoreExt};
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::{PageIndexPolicy, ParquetMetaData};
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ // A corrupt footer can report a negative row count. `num_rows as u64` would otherwise
+ // wrap it into a huge positive value, silently corrupting every row-group boundary
+ // computed from `group_start`/`group_end` below (and therefore which deleted row indexes
+ // land in which row group) instead of failing loudly.
+ if num_rows < 0 {
+ return Err(GeneralError(format!(
+ "Parquet footer reports a negative row count ({num_rows}) for row group {idx}"
+ )));
+ }
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
+ }
+ cursor = row + 1;
+ }
+ if deleted_in_group == num_rows && num_rows > 0 {
+ plan.skip(idx);
+ } else if deleted_in_group > 0 {
+ if group_end > cursor {
+ selectors.push(RowSelector::select((group_end - cursor) as usize));
+ }
+ plan.scan_selection(idx, RowSelection::from(selectors));
+ }
+ group_start = group_end;
+ }
+ // A deleted index beyond the file's total row count means the DV does not
+ // belong to this file (stale or corrupted metadata); silently dropping it
+ // would under-apply deletions.
+ if let Some(&row) = deleted_iter.peek() {
+ return Err(GeneralError(format!(
+ "Deletion vector marks row {row} but the file only has {group_start} rows"
+ )));
+ }
+ Ok(plan)
+}
+
+/// Verify a decoded deletion vector's row count matches the descriptor's
+/// declared `cardinality`, mirroring Delta's JVM reader
+/// (`StoredBitmap.validateCardinality`). The CRC and framing checks catch
+/// corruption but not a stale, otherwise well-formed bitmap whose row count
+/// no longer matches the descriptor -- that would silently under- or
+/// over-delete rows.
+fn validate_cardinality(
+ file_path: &str,
+ expected: i64,
+ deleted: &RoaringTreemap,
+) -> Result<(), ExecutionError> {
+ let actual = deleted.len();
+ if actual != expected as u64 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has cardinality mismatch: descriptor says {expected}, decoded bitmap has {actual} deleted rows"
+ )));
+ }
+ Ok(())
+}
+
+/// One data file plus everything needed to apply its deletion vector. The
+/// file's size comes from `file.object_meta.size` (built by the planner from
+/// the proto's `file_size`).
+///
+/// `data_store` and `dv_store` are resolved by the caller *before* entering
+/// the async `attach_access_plans` runtime (see its doc comment): building an
+/// object store is sync I/O that, for a cold S3 authority, internally issues
+/// its own `Handle::block_on` calls, which panics if nested inside another
+/// `block_on`. Resolving up front means this module never constructs a
+/// store itself.
+pub struct DvScanFile {
+ pub file: PartitionedFile,
+ /// Full URL of the data file (proto `file_path`).
+ pub file_path: String,
+ pub dv: Option<DeltaSparkDvDescriptor>,
+ /// Object store for `file_path`, pre-resolved by the caller. Only read
+ /// when `dv` is `Some` (files without a deletion vector never open their
+ /// footer here), but every file carries one so the struct's shape
+ /// doesn't depend on whether a deletion vector is present.
+ pub data_store: Arc<dyn ObjectStore>,
+ /// Store and within-store path for an on-disk deletion vector's absolute
+ /// path, pre-resolved by the caller. `None` when the file has no
+ /// deletion vector or the deletion vector is stored inline.
+ pub dv_store: Option<(Arc<dyn ObjectStore>, Path)>,
+}
+
+/// Execution-memory-pool reservation covering one file's expanded DV row selectors across
+/// their *entire* lifetime attached to a scan -- from `build_access_plan`'s construction
+/// through DataFusion 54.1's reader normalizing the attached [`ParquetAccessPlan`]
+/// (`create_initial_plan`'s deep clone plus `into_overall_row_selection`'s combined
+/// `RowSelection`; see [`reader_peak_bytes`]) -- attached to the file's [`PartitionedFile`]
+/// extensions alongside its [`ParquetAccessPlan`]. The reservation's lifetime is tied to the
+/// `PartitionedFile` it is attached to, so it is released back to the pool exactly when the
+/// plan is dropped (query completion or an early-terminated scan), never held open longer.
+/// Newtype-wrapped so it occupies its own slot in the multi-slot, type-keyed `extensions` map
+/// (`datafusion_common::extensions::Extensions`) alongside the plan, rather than a bare
+/// `MemoryReservation` colliding with one some other extension might attach.
+pub struct DvAccessPlanReservation(pub MemoryReservation);
+
+/// Total number of [`RowSelector`]s materialized across `plan`'s per-row-group
+/// selections (`RowGroupAccess::Selection`); `Scan`/`Skip` row groups
+/// contribute none. An alternating deleted/retained bitmap produces one
+/// non-coalescing selector per row (see [`reader_peak_bytes`]'s doc comment
+/// for the worst-case accounting), so this count -- not the deletion
+/// vector's cardinality -- is the thing that must be bounded and reserved
+/// against the execution memory pool.
+fn total_selectors(plan: &ParquetAccessPlan) -> usize {
+ plan.inner()
+ .iter()
+ .map(|access| match access {
+ RowGroupAccess::Selection(selection) => selection.iter().count(),
+ _ => 0,
+ })
+ .sum()
+}
+
+/// Multiplier bounding the peak allocation live *during construction* of one
+/// file's [`RowSelection`]s, relative to the conservative selector-count
+/// bound `S = 2 * cardinality + num_row_groups` (one non-coalescing selector
+/// per deleted row in the worst-case alternating pattern, doubled, plus up to
+/// one extra boundary selector per row group). Split `S` into `r`, the
+/// selectors already retained from row groups `build_access_plan` has
+/// finished, and `c`, the selectors accumulated so far in the current row
+/// group's source `Vec`; `r` and `c` partition the selectors counted toward
+/// `S`, so `r + c <= S` always. While the current group is being built, the
+/// `Vec`'s doubling growth strategy can leave its backing allocation at up to
+/// `2 * c` (the next power-of-two capacity above `c`). Once the group
+/// finishes, `RowSelection::from(Vec)` (parquet's `FromIterator` impl,
+/// `with_capacity` + copy) builds a second, separate `Vec` of size `c` from
+/// that source while the source is still alive, so at the moment the copy
+/// begins, the retained selectors, the current group's doubled source `Vec`,
+/// and the copy are all live simultaneously: `r + 2c + c = r + 3c`. Since
+/// `r >= 0`, `r + 3c <= 3r + 3c = 3(r + c) <= 3S`. 3x covers that peak.
+const CONSTRUCTION_PEAK_FACTOR: usize = 3;
+
+/// Upper bound on how much larger a `Vec`'s backing allocation can be than its element count
+/// after being built by repeated pushes: `std`'s doubling growth strategy never leaves a `Vec`
+/// of `n` elements with a backing allocation larger than the next power of two above `n`, which
+/// is at most `2 * n` for any `n >= 1`.
+const VEC_GROWTH_CAPACITY_FACTOR: usize = 2;
+
+/// `RawVec`'s minimum non-zero capacity for element sizes `<= 1024` bytes ([`RowSelector`] is
+/// 16 bytes on 64-bit platforms: a `usize` row count plus a padded `bool`). Applied once per
+/// row group (or per contiguous run of row groups) a fresh `from_fn`/`FlatMap`-driven `Vec`
+/// gets built for (see [`reader_peak_bytes`]), so even a group or run whose true selector count
+/// is tiny still pays this floor.
+const MIN_VEC_CAPACITY_SELECTORS: usize = 4;
+
+/// Conservative upper bound, in bytes, on the peak allocation live while DataFusion 54.1's
+/// reader normalizes one file's attached [`ParquetAccessPlan`] -- the allocation this module's
+/// steady-state reservation must cover, not merely the plan's own retained selector bytes.
+/// THREE allocations can be live simultaneously by the time `into_overall_row_selection`
+/// returns, not two -- the clone is only exact when page-index pruning never touches it:
+///
+/// 1. **Attached original** (`selectors`, exact): `create_initial_plan` deep-clones the
+/// attached plan while the original remains reachable from the file's `extensions` until
+/// the scan consumes it. The ORIGINAL's own selector `Vec`s are exact -- a coalesced
+/// [`RowSelection`] built via `RowSelection::from(Vec<RowSelector>)` (what
+/// `build_access_plan` uses) has no excess capacity, because that conversion is a plain
+/// `with_capacity(len)` copy, not a `size_hint`-blind fold.
+/// 2. **The clone, possibly capacity-inflated** (`<= VEC_GROWTH_CAPACITY_FACTOR * selectors +
+/// MIN_VEC_CAPACITY_SELECTORS * num_row_groups`): if page-index pruning fires
+/// (`PagePruningAccessPlanFilter`; `access_plan.rs`'s `scan_selection` on a row group that
+/// already carries a `RowGroupAccess::Selection` calls `existing.intersection(&page_derived)`
+/// -- `RowSelection::intersection` -> `intersect_row_selections`), it replaces the CLONE's
+/// per-row-group selection with that intersection's output. `intersect_row_selections` is
+/// ANOTHER `from_fn` generator with `size_hint() == (0, None)`, so each intersected row
+/// group's backing `Vec` starts at `with_capacity(0)` and doubles as it grows, independent
+/// of whatever capacity the pre-intersection selection had. This inflated clone is still
+/// live when `into_overall_row_selection` later moves its buffer. Term 1's exactness
+/// guarantee holds for the ORIGINAL always, and for the clone only when page-index pruning
+/// never fires against it -- once it does, the clone must be charged at the SAME
+/// growth-capped bound as a fresh combined-selection `Vec` (term 3), summed once per row
+/// group rather than once per run, since each row group's `Selection` is intersected
+/// independently.
+/// 3. **Per-run combined-selection allocation** (`<= VEC_GROWTH_CAPACITY_FACTOR * (selectors +
+/// num_row_groups) + MIN_VEC_CAPACITY_SELECTORS * num_row_groups`): `into_overall_row_selection`
+/// collects each contiguous run of row groups' selectors into a *new* `RowSelection` via a
+/// `FlatMap` whose `size_hint().0 == 0`, so that run's `Vec` starts at `with_capacity(0)`
+/// and doubles as it grows -- capping its backing allocation at
+/// `max(MIN_VEC_CAPACITY_SELECTORS, next_power_of_two(len))`, which is at most
+/// `MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR * len` for a run of `len`
+/// selectors. `len` is at most that run's share of `selectors` plus one boundary selector
+/// per `RowGroupAccess::Scan` row group in the run (`Scan` always contributes exactly one
+/// `RowSelector::select(num_rows)`; see `access_plan.rs`'s `into_overall_row_selection`).
+/// Summing across at most `num_row_groups` runs (each spans >= 1 row group) bounds the total
+/// at `VEC_GROWTH_CAPACITY_FACTOR * selectors + (MIN_VEC_CAPACITY_SELECTORS +
+/// VEC_GROWTH_CAPACITY_FACTOR) * num_row_groups`.
+///
+/// Summing all three terms and converting to bytes: `((1 + 2 * VEC_GROWTH_CAPACITY_FACTOR) *
+/// selectors + (2 * MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR) * num_row_groups)
+/// * size_of::<RowSelector>()` -- with the constants above, `(5 * selectors + 10 *
+/// num_row_groups) * size_of::<RowSelector>()`. Checked against two measured worst cases:
+///
+/// - No page-index pruning (the original P2 report; term 2 stays exact): one 2,000,000-row
+/// group, 1,000,000 alternating deletions, `selectors = 2,000,000`. Measured allocator peak
+/// 97,554,457 B; the byte-for-byte accounting for the attached original plus the (here,
+/// exact) clone plus the inflated combined selection explains 97,554,432 B of that, a 25 B
+/// residue we did not attribute. This bound gives 160,000,160 B -- much looser here because
+/// it must also cover the next case, where the clone is NOT exact.
+/// - Page-index pruning fires against the clone: one 1,048,577-row group, `selectors =
+/// 1,048,577`. Measured peak 83,886,096 B; this bound gives 83,886,320 B (a 224 B, <1%
+/// margin -- deliberately tight, since this is the case that drives the bound).
+///
+/// Uses checked arithmetic throughout: a selector or row-group count large enough to overflow
+/// `usize` indicates a corrupted or malicious input, reported as a clean error rather than
+/// panicking.
+fn reader_peak_bytes(selectors: usize, num_row_groups: usize) -> Result<usize, ExecutionError> {
+ let overflow = || {
+ GeneralError(format!(
+ "Deletion vector reader-peak bound overflowed for {selectors} selectors and \
+ {num_row_groups} row groups"
+ ))
+ };
+ // Term 1: the attached original -- exact, untouched by page-index pruning (only the clone
+ // is ever intersected; see the doc comment above).
+ let attached_term = selectors;
+ // Term 2: the clone, bounded as if page-index pruning DID fire against every row group
+ // (safe even when it doesn't: term 2's bound is always >= `selectors`, so it never
+ // undershoots the exact case either).
+ let clone_growth = selectors
+ .checked_mul(VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let clone_floor = num_row_groups
+ .checked_mul(MIN_VEC_CAPACITY_SELECTORS)
+ .ok_or_else(overflow)?;
+ let clone_term = clone_growth.checked_add(clone_floor).ok_or_else(overflow)?;
+ // Term 3: into_overall_row_selection's per-run combined-selection allocation.
+ let combined_growth = selectors
+ .checked_mul(VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let combined_floor = num_row_groups
+ .checked_mul(MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let combined_term = combined_growth
+ .checked_add(combined_floor)
+ .ok_or_else(overflow)?;
+
+ let selector_bound = attached_term
+ .checked_add(clone_term)
+ .and_then(|sum| sum.checked_add(combined_term))
+ .ok_or_else(overflow)?;
+ selector_bound
+ .checked_mul(size_of::<RowSelector>())
+ .ok_or_else(overflow)
+}
+
+/// Upper bound, in [`RowSelector`]s, on how many extra selectors the parquet reader's
+/// page-index pruning can add on top of the deletion vector's own selection when normalizing
+/// one file, from that file's already-fetched [`ParquetMetaData`].
+///
+/// `intersect_row_selections` (parquet's `selection.rs`), which combines a page-pruning
+/// selection with the deletion vector's selection, is a `from_fn` generator whose
+/// `size_hint()` is `(0, None)`: for inputs of length `a` and `b`, its output can have up to
+/// `a + b` selectors -- longer than either input. Bounding the page-pruning side of that sum
+/// requires knowing how many selectors a page-index-derived selection could produce: at most
+/// two per data page (one skip, one select, in the worst case of alternating page-level
+/// pruning decisions), summed over every column of every row group.
+///
+/// Returns `0` when `metadata` carries no offset index (`metadata.offset_index()` is `None`).
+/// This is provably safe, not merely a convenient default: page-index pruning cannot produce a
+/// page-level selection without the offset index to locate pages by, so there are no
+/// page-pruning selectors to bound. The offset index is fetched with
+/// `PageIndexPolicy::Optional` from the same `FileMetadataCache` entry the scan's reader later
+/// reopens (see [`attach_access_plan`]'s footer-fetch comment), so this function observes
+/// exactly what the reader will see.
+///
+/// Uses checked arithmetic throughout for the same reason as [`admission_bound_bytes`].
+fn page_selection_bound_selectors(metadata: &ParquetMetaData) -> Result<usize, ExecutionError> {
+ let Some(offset_index) = metadata.offset_index() else {
+ return Ok(0);
+ };
+ let overflow = || {
+ GeneralError(
+ "Deletion vector page-selection bound overflowed while summing offset-index page \
+ locations"
+ .to_string(),
+ )
+ };
+ let mut total_page_locations = 0usize;
+ for row_group in offset_index {
+ for column in row_group {
+ total_page_locations = total_page_locations
+ .checked_add(column.page_locations().len())
+ .ok_or_else(overflow)?;
+ }
+ }
+ total_page_locations.checked_mul(2).ok_or_else(overflow)
+}
+
+/// Execution-memory-pool admission bound, in bytes, for one file's deletion-vector access
+/// plan -- reserved *before* calling `build_access_plan` (see [`attach_access_plan`]'s
+/// pre-reserve call site) to cover the larger of two peaks live at different points in the
+/// plan's lifetime. In practice the reader-normalization peak below dominates the construction
+/// peak unconditionally for any non-trivial input (`reader_peak_bytes(S, G) = (5S + 10G) *
+/// size_of::<RowSelector>()` always exceeds `CONSTRUCTION_PEAK_FACTOR * S *
+/// size_of::<RowSelector>() = 3S * size_of::<RowSelector>()` once `S >= 1`, since the `5S` term
+/// alone already exceeds `3S`); the construction term is retained as a documented floor rather
+/// than dropped, since it is cheap to compute and keeps this bound correct even if the reader's
+/// growth factors ever shrink below construction's.
+///
+/// - **Construction peak** (`CONSTRUCTION_PEAK_FACTOR * S`, see that constant's doc comment):
+/// live while `build_access_plan` builds the plan's `RowSelection`s. Construction's
+/// transient allocations fully unwind before `build_access_plan` returns, so this peak never
+/// overlaps the reader-normalization peak below.
+/// - **Reader-normalization peak** (`reader_peak_bytes(S + page_bound_selectors,
+/// num_row_groups)`, see that function): live later, once DataFusion's reader normalizes the
+/// attached plan. `S = 2 * cardinality + num_row_groups` is the same conservative bound on
+/// the plan's final retained selector count used for the construction peak -- it provably
+/// bounds `R = total_selectors(&plan)` (`R <= S`, from `build_access_plan`'s
+/// one-non-coalescing-selector-per-deleted-row worst case plus one boundary selector per row
+/// group), so `S + page_bound_selectors` bounds `R` after page-index inflation the same way
+/// `S` bounds `R` before it.
+///
+/// These two peaks never overlap in time, so `max` -- not `sum` -- is the correct combinator:
+/// reserving their sum would over-reserve for no safety benefit.
+///
+/// Deliberately not clamped by the file's total row count here, unlike the reader-peak target
+/// `attach_access_plan` resizes down to after construction (see that call site): `S`'s
+/// `+ num_row_groups` boundary term is a worst-case padding margin that can legitimately exceed
+/// the total row count for a small, heavily-deleted file, and admission sizing has no actual
+/// retained-selector count yet to clamp against -- only after construction, once `R` is known,
+/// is clamping to the total row count both meaningful and strictly tighter. Leaving this bound
+/// unclamped only ever makes admission more conservative, never less safe.
+///
+/// Uses checked arithmetic throughout: a cardinality, row-group count, or page bound large
+/// enough to overflow `usize` while computing this bound indicates a corrupted or malicious
+/// descriptor, reported as a clean error rather than panicking.
+fn admission_bound_bytes(
+ cardinality: i64,
+ num_row_groups: usize,
+ page_bound_selectors: usize,
+) -> Result<usize, ExecutionError> {
+ let overflow = || {
+ GeneralError(format!(
+ "Deletion vector admission bound overflowed for cardinality {cardinality}, \
+ {num_row_groups} row groups, and page bound {page_bound_selectors} selectors"
+ ))
+ };
+ let cardinality_usize = usize::try_from(cardinality).map_err(|_| overflow())?;
+ // S: the conservative bound on the plan's final *retained* selector count (what
+ // `total_selectors(&plan)` cannot exceed) -- unchanged from the pre-existing
+ // construction-only bound this function replaces.
+ let s = cardinality_usize
+ .checked_mul(2)
+ .and_then(|doubled| doubled.checked_add(num_row_groups))
+ .ok_or_else(overflow)?;
+
+ let construction_bytes = s
+ .checked_mul(size_of::<RowSelector>())
+ .and_then(|bytes| bytes.checked_mul(CONSTRUCTION_PEAK_FACTOR))
+ .ok_or_else(overflow)?;
+
+ let s_plus_page = s.checked_add(page_bound_selectors).ok_or_else(overflow)?;
+ let reader_bytes = reader_peak_bytes(s_plus_page, num_row_groups)?;
+
+ Ok(construction_bytes.max(reader_bytes))
+}
+
+/// Upper bound on concurrent DV-blob and footer fetches per partition. Both
+/// are small ranged reads, so a modest fan-out hides object-store latency
+/// without flooding the store client.
+const DV_FETCH_CONCURRENCY: usize = 8;
+
+/// Called via `block_on` at plan-creation time on the executor task: DV blobs
+/// are small ranged reads and footers are needed to learn row-group
+/// boundaries. Files are fetched concurrently (bounded by
+/// [`DV_FETCH_CONCURRENCY`]) with input order preserved. Footer fetches go
+/// through the scan's shared FileMetadataCache, so the scan's subsequent open
+/// of the same file is served from cache. That reuse relies on each input
+/// [`PartitionedFile`] being returned as-is (only `with_extension` applied),
+/// never rebuilt: the cache entry is keyed by this exact `object_meta` and the
+/// scan later looks it up through the same struct.
+///
+/// Deliberately takes no object-store options map and imports no
+/// store-construction helper: every [`DvScanFile`] arrives with its stores
+/// already resolved by the caller (see its doc comment), so this async path
+/// structurally cannot build an object store -- only `runtime_env` is still
+/// threaded through, for the shared `FileMetadataCache` and (per file) the
+/// execution `MemoryPool` each expanded access plan's row selectors are
+/// reserved against -- see [`DvAccessPlanReservation`].
+pub async fn attach_access_plans(
+ runtime_env: Arc<RuntimeEnv>,
+ files: Vec<DvScanFile>,
+) -> Result<Vec<PartitionedFile>, ExecutionError> {
+ futures::stream::iter(files)
+ .map(|scan_file| attach_access_plan(Arc::clone(&runtime_env), scan_file))
+ .buffered(DV_FETCH_CONCURRENCY)
+ .try_collect()
+ .await
+}
+
+/// Resolve one file's deletion vector into an attached [`ParquetAccessPlan`];
+/// files without a DV pass through untouched.
+async fn attach_access_plan(
+ runtime_env: Arc<RuntimeEnv>,
+ scan_file: DvScanFile,
+) -> Result<PartitionedFile, ExecutionError> {
+ let DvScanFile {
+ file,
+ file_path,
+ dv,
+ data_store,
+ dv_store,
+ } = scan_file;
+ let dv = match dv {
+ Some(dv) => dv,
+ None => return Ok(file),
+ };
+ // Delta's canonical `DeletionVectorDescriptor.EMPTY`: inline storage, empty
+ // payload, size 0, cardinality 0. Spark's reader returns all rows for it;
+ // decoding would fail (the empty payload is too short for a magic
+ // number), so pass the file through unchanged before attempting to read it.
+ if dv.cardinality == 0 && dv.size_in_bytes == 0 {
+ return Ok(file);
+ }
+ if dv.size_in_bytes < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative size {}",
+ dv.size_in_bytes
+ )));
+ }
+ if dv.cardinality < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative cardinality {}",
+ dv.cardinality
+ )));
+ }
+
+ let data: Vec<u8> = if let Some(inline) = dv.inline_data {
Review Comment:
> Could it compare the length before `deserialize_dv_bitmap`?
Added `check_inline_payload_size`, called before decoding; the error names the payload length and the descriptor size. A test covers the mismatch and the matching case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4083936869
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,2192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::mem::size_of;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::{ParquetAccessPlan, RowGroupAccess};
+use datafusion::execution::memory_pool::{MemoryConsumer, MemoryReservation};
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::path::Path;
+use object_store::{ObjectStore, ObjectStoreExt};
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::{PageIndexPolicy, ParquetMetaData};
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+/// An inline payload carries no framing, so its length is checked against the descriptor here,
+/// the way `unframe_dv_blob` checks an on-disk blob's size header.
+fn check_inline_payload_size(
+ file_path: &str,
+ payload: &[u8],
+ size_in_bytes: i32,
+) -> Result<(), ExecutionError> {
+ if payload.len() as i64 != i64::from(size_in_bytes) {
+ return Err(GeneralError(format!(
+ "Inline deletion vector for {file_path} has {} bytes but its descriptor says {size_in_bytes}",
+ payload.len()
+ )));
+ }
+ Ok(())
+}
+
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ // A corrupt footer can report a negative row count. `num_rows as u64` would otherwise
+ // wrap it into a huge positive value, silently corrupting every row-group boundary
+ // computed from `group_start`/`group_end` below (and therefore which deleted row indexes
+ // land in which row group) instead of failing loudly.
+ if num_rows < 0 {
+ return Err(GeneralError(format!(
+ "Parquet footer reports a negative row count ({num_rows}) for row group {idx}"
+ )));
+ }
+ let num_rows = num_rows as u64;
+ let group_end = group_start.checked_add(num_rows).ok_or_else(|| {
+ GeneralError(format!(
+ "Parquet footer row counts overflow at row group {idx} ({group_start} + {num_rows})"
+ ))
+ })?;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
+ }
+ cursor = row + 1;
+ }
+ if deleted_in_group == num_rows && num_rows > 0 {
+ plan.skip(idx);
+ } else if deleted_in_group > 0 {
+ if group_end > cursor {
+ selectors.push(RowSelector::select((group_end - cursor) as usize));
+ }
+ plan.scan_selection(idx, RowSelection::from(selectors));
+ }
+ group_start = group_end;
+ }
+ // A deleted index beyond the file's total row count means the DV does not
+ // belong to this file (stale or corrupted metadata); silently dropping it
+ // would under-apply deletions.
+ if let Some(&row) = deleted_iter.peek() {
+ return Err(GeneralError(format!(
+ "Deletion vector marks row {row} but the file only has {group_start} rows"
+ )));
+ }
+ Ok(plan)
+}
+
+/// Verify a decoded deletion vector's row count matches the descriptor's
+/// declared `cardinality`, mirroring Delta's JVM reader
+/// (`StoredBitmap.validateCardinality`). The CRC and framing checks catch
+/// corruption but not a stale, otherwise well-formed bitmap whose row count
+/// no longer matches the descriptor -- that would silently under- or
+/// over-delete rows.
+fn validate_cardinality(
+ file_path: &str,
+ expected: i64,
+ deleted: &RoaringTreemap,
+) -> Result<(), ExecutionError> {
+ let actual = deleted.len();
+ if actual != expected as u64 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has cardinality mismatch: descriptor says {expected}, decoded bitmap has {actual} deleted rows"
+ )));
+ }
+ Ok(())
+}
+
+/// One data file plus everything needed to apply its deletion vector. The
+/// file's size comes from `file.object_meta.size` (built by the planner from
+/// the proto's `file_size`).
+///
+/// `data_store` and `dv_store` are resolved by the caller *before* entering
+/// the async `attach_access_plans` runtime (see its doc comment): building an
+/// object store is sync I/O that, for a cold S3 authority, internally issues
+/// its own `Handle::block_on` calls, which panics if nested inside another
+/// `block_on`. Resolving up front means this module never constructs a
+/// store itself.
+pub struct DvScanFile {
+ pub file: PartitionedFile,
+ /// Full URL of the data file (proto `file_path`).
+ pub file_path: String,
+ pub dv: Option<DeltaSparkDvDescriptor>,
+ /// Object store for `file_path`, pre-resolved by the caller. Only read
+ /// when `dv` is `Some` (files without a deletion vector never open their
+ /// footer here), but every file carries one so the struct's shape
+ /// doesn't depend on whether a deletion vector is present.
+ pub data_store: Arc<dyn ObjectStore>,
+ /// Store and within-store path for an on-disk deletion vector's absolute
+ /// path, pre-resolved by the caller. `None` when the file has no
+ /// deletion vector or the deletion vector is stored inline.
+ pub dv_store: Option<(Arc<dyn ObjectStore>, Path)>,
+}
+
+/// Execution-memory-pool reservation covering one file's expanded DV row selectors across
+/// their *entire* lifetime attached to a scan -- from `build_access_plan`'s construction
+/// through DataFusion 54.1's reader normalizing the attached [`ParquetAccessPlan`]
+/// (`create_initial_plan`'s deep clone plus `into_overall_row_selection`'s combined
+/// `RowSelection`; see [`reader_peak_bytes`]) -- attached to the file's [`PartitionedFile`]
+/// extensions alongside its [`ParquetAccessPlan`]. The reservation's lifetime is tied to the
+/// `PartitionedFile` it is attached to, so it is released back to the pool exactly when the
+/// plan is dropped (query completion or an early-terminated scan), never held open longer.
+/// Newtype-wrapped so it occupies its own slot in the multi-slot, type-keyed `extensions` map
+/// (`datafusion_common::extensions::Extensions`) alongside the plan, rather than a bare
+/// `MemoryReservation` colliding with one some other extension might attach.
+pub struct DvAccessPlanReservation(pub MemoryReservation);
+
+/// Total number of [`RowSelector`]s materialized across `plan`'s per-row-group
+/// selections (`RowGroupAccess::Selection`); `Scan`/`Skip` row groups
+/// contribute none. An alternating deleted/retained bitmap produces one
+/// non-coalescing selector per row (see [`reader_peak_bytes`]'s doc comment
+/// for the worst-case accounting), so this count -- not the deletion
+/// vector's cardinality -- is the thing that must be bounded and reserved
+/// against the execution memory pool.
+fn total_selectors(plan: &ParquetAccessPlan) -> usize {
+ plan.inner()
+ .iter()
+ .map(|access| match access {
+ RowGroupAccess::Selection(selection) => selection.iter().count(),
+ _ => 0,
+ })
+ .sum()
+}
+
+/// Multiplier bounding the peak allocation live *during construction* of one
+/// file's [`RowSelection`]s, relative to the conservative selector-count
+/// bound `S = 2 * cardinality + num_row_groups` (one non-coalescing selector
+/// per deleted row in the worst-case alternating pattern, doubled, plus up to
+/// one extra boundary selector per row group). Split `S` into `r`, the
+/// selectors already retained from row groups `build_access_plan` has
+/// finished, and `c`, the selectors accumulated so far in the current row
+/// group's source `Vec`; `r` and `c` partition the selectors counted toward
+/// `S`, so `r + c <= S` always. While the current group is being built, the
+/// `Vec`'s doubling growth strategy can leave its backing allocation at up to
+/// `2 * c` (the next power-of-two capacity above `c`). Once the group
+/// finishes, `RowSelection::from(Vec)` (parquet's `FromIterator` impl,
+/// `with_capacity` + copy) builds a second, separate `Vec` of size `c` from
+/// that source while the source is still alive, so at the moment the copy
+/// begins, the retained selectors, the current group's doubled source `Vec`,
+/// and the copy are all live simultaneously: `r + 2c + c = r + 3c`. Since
+/// `r >= 0`, `r + 3c <= 3r + 3c = 3(r + c) <= 3S`. 3x covers that peak.
+const CONSTRUCTION_PEAK_FACTOR: usize = 3;
+
+/// Upper bound on how much larger a `Vec`'s backing allocation can be than its element count
+/// after being built by repeated pushes: `std`'s doubling growth strategy never leaves a `Vec`
+/// of `n` elements with a backing allocation larger than the next power of two above `n`, which
+/// is at most `2 * n` for any `n >= 1`.
+const VEC_GROWTH_CAPACITY_FACTOR: usize = 2;
+
+/// `RawVec`'s minimum non-zero capacity for element sizes `<= 1024` bytes ([`RowSelector`] is
+/// 16 bytes on 64-bit platforms: a `usize` row count plus a padded `bool`). Applied once per
+/// row group (or per contiguous run of row groups) a fresh `from_fn`/`FlatMap`-driven `Vec`
+/// gets built for (see [`reader_peak_bytes`]), so even a group or run whose true selector count
+/// is tiny still pays this floor.
+const MIN_VEC_CAPACITY_SELECTORS: usize = 4;
+
+/// Conservative upper bound, in bytes, on the peak allocation live while DataFusion 54.1's
+/// reader normalizes one file's attached [`ParquetAccessPlan`] -- the allocation this module's
+/// steady-state reservation must cover, not merely the plan's own retained selector bytes.
+/// THREE allocations can be live simultaneously by the time `into_overall_row_selection`
+/// returns, not two -- the clone is only exact when page-index pruning never touches it:
+///
+/// 1. **Attached original** (`selectors`, exact): `create_initial_plan` deep-clones the
+/// attached plan while the original remains reachable from the file's `extensions` until
+/// the scan consumes it. The ORIGINAL's own selector `Vec`s are exact -- a coalesced
+/// [`RowSelection`] built via `RowSelection::from(Vec<RowSelector>)` (what
+/// `build_access_plan` uses) has no excess capacity, because that conversion is a plain
+/// `with_capacity(len)` copy, not a `size_hint`-blind fold.
+/// 2. **The clone, possibly capacity-inflated** (`<= VEC_GROWTH_CAPACITY_FACTOR * selectors +
+/// MIN_VEC_CAPACITY_SELECTORS * num_row_groups`): if page-index pruning fires
+/// (`PagePruningAccessPlanFilter`; `access_plan.rs`'s `scan_selection` on a row group that
+/// already carries a `RowGroupAccess::Selection` calls `existing.intersection(&page_derived)`
+/// -- `RowSelection::intersection` -> `intersect_row_selections`), it replaces the CLONE's
+/// per-row-group selection with that intersection's output. `intersect_row_selections` is
+/// ANOTHER `from_fn` generator with `size_hint() == (0, None)`, so each intersected row
+/// group's backing `Vec` starts at `with_capacity(0)` and doubles as it grows, independent
+/// of whatever capacity the pre-intersection selection had. This inflated clone is still
+/// live when `into_overall_row_selection` later moves its buffer. Term 1's exactness
+/// guarantee holds for the ORIGINAL always, and for the clone only when page-index pruning
+/// never fires against it -- once it does, the clone must be charged at the SAME
+/// growth-capped bound as a fresh combined-selection `Vec` (term 3), summed once per row
+/// group rather than once per run, since each row group's `Selection` is intersected
+/// independently.
+/// 3. **Per-run combined-selection allocation** (`<= VEC_GROWTH_CAPACITY_FACTOR * (selectors +
+/// num_row_groups) + MIN_VEC_CAPACITY_SELECTORS * num_row_groups`): `into_overall_row_selection`
+/// collects each contiguous run of row groups' selectors into a *new* `RowSelection` via a
+/// `FlatMap` whose `size_hint().0 == 0`, so that run's `Vec` starts at `with_capacity(0)`
+/// and doubles as it grows -- capping its backing allocation at
+/// `max(MIN_VEC_CAPACITY_SELECTORS, next_power_of_two(len))`, which is at most
+/// `MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR * len` for a run of `len`
+/// selectors. `len` is at most that run's share of `selectors` plus one boundary selector
+/// per `RowGroupAccess::Scan` row group in the run (`Scan` always contributes exactly one
+/// `RowSelector::select(num_rows)`; see `access_plan.rs`'s `into_overall_row_selection`).
+/// Summing across at most `num_row_groups` runs (each spans >= 1 row group) bounds the total
+/// at `VEC_GROWTH_CAPACITY_FACTOR * selectors + (MIN_VEC_CAPACITY_SELECTORS +
+/// VEC_GROWTH_CAPACITY_FACTOR) * num_row_groups`.
+///
+/// Summing all three terms and converting to bytes: `((1 + 2 * VEC_GROWTH_CAPACITY_FACTOR) *
+/// selectors + (2 * MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR) * num_row_groups)
+/// * size_of::<RowSelector>()` -- with the constants above, `(5 * selectors + 10 *
+/// num_row_groups) * size_of::<RowSelector>()`. Checked against two measured worst cases:
+///
+/// - No page-index pruning (the original P2 report; term 2 stays exact): one 2,000,000-row
+/// group, 1,000,000 alternating deletions, `selectors = 2,000,000`. Measured allocator peak
+/// 97,554,457 B; the byte-for-byte accounting for the attached original plus the (here,
+/// exact) clone plus the inflated combined selection explains 97,554,432 B of that, a 25 B
+/// residue we did not attribute. This bound gives 160,000,160 B -- much looser here because
+/// it must also cover the next case, where the clone is NOT exact.
+/// - Page-index pruning fires against the clone: one 1,048,577-row group, `selectors =
+/// 1,048,577`. Measured peak 83,886,096 B; this bound gives 83,886,320 B (a 224 B, <1%
+/// margin -- deliberately tight, since this is the case that drives the bound).
+///
+/// Uses checked arithmetic throughout: a selector or row-group count large enough to overflow
+/// `usize` indicates a corrupted or malicious input, reported as a clean error rather than
+/// panicking.
+fn reader_peak_bytes(selectors: usize, num_row_groups: usize) -> Result<usize, ExecutionError> {
+ let overflow = || {
+ GeneralError(format!(
+ "Deletion vector reader-peak bound overflowed for {selectors} selectors and \
+ {num_row_groups} row groups"
+ ))
+ };
+ // Term 1: the attached original -- exact, untouched by page-index pruning (only the clone
+ // is ever intersected; see the doc comment above).
+ let attached_term = selectors;
+ // Term 2: the clone, bounded as if page-index pruning DID fire against every row group
+ // (safe even when it doesn't: term 2's bound is always >= `selectors`, so it never
+ // undershoots the exact case either).
+ let clone_growth = selectors
+ .checked_mul(VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let clone_floor = num_row_groups
+ .checked_mul(MIN_VEC_CAPACITY_SELECTORS)
+ .ok_or_else(overflow)?;
+ let clone_term = clone_growth.checked_add(clone_floor).ok_or_else(overflow)?;
+ // Term 3: into_overall_row_selection's per-run combined-selection allocation.
+ let combined_growth = selectors
+ .checked_mul(VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let combined_floor = num_row_groups
+ .checked_mul(MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let combined_term = combined_growth
+ .checked_add(combined_floor)
+ .ok_or_else(overflow)?;
+
+ let selector_bound = attached_term
+ .checked_add(clone_term)
+ .and_then(|sum| sum.checked_add(combined_term))
+ .ok_or_else(overflow)?;
+ selector_bound
+ .checked_mul(size_of::<RowSelector>())
+ .ok_or_else(overflow)
+}
+
+/// Upper bound, in [`RowSelector`]s, on how many extra selectors the parquet reader's
+/// page-index pruning can add on top of the deletion vector's own selection when normalizing
+/// one file, from that file's already-fetched [`ParquetMetaData`].
+///
+/// `intersect_row_selections` (parquet's `selection.rs`), which combines a page-pruning
+/// selection with the deletion vector's selection, is a `from_fn` generator whose
+/// `size_hint()` is `(0, None)`: for inputs of length `a` and `b`, its output can have up to
+/// `a + b` selectors -- longer than either input. Bounding the page-pruning side of that sum
+/// requires knowing how many selectors a page-index-derived selection could produce: at most
+/// two per data page (one skip, one select, in the worst case of alternating page-level
+/// pruning decisions), summed over every column of every row group.
+///
+/// Returns `0` when `metadata` carries no offset index (`metadata.offset_index()` is `None`).
+/// This is provably safe, not merely a convenient default: page-index pruning cannot produce a
+/// page-level selection without the offset index to locate pages by, so there are no
+/// page-pruning selectors to bound. The offset index is fetched with
+/// `PageIndexPolicy::Optional` from the same `FileMetadataCache` entry the scan's reader later
+/// reopens (see [`attach_access_plan`]'s footer-fetch comment), so this function observes
+/// exactly what the reader will see.
+///
+/// Uses checked arithmetic throughout for the same reason as [`admission_bound_bytes`].
+fn page_selection_bound_selectors(metadata: &ParquetMetaData) -> Result<usize, ExecutionError> {
+ let Some(offset_index) = metadata.offset_index() else {
+ return Ok(0);
+ };
+ let overflow = || {
+ GeneralError(
+ "Deletion vector page-selection bound overflowed while summing offset-index page \
+ locations"
+ .to_string(),
+ )
+ };
+ let mut total_page_locations = 0usize;
+ for row_group in offset_index {
+ for column in row_group {
+ total_page_locations = total_page_locations
+ .checked_add(column.page_locations().len())
+ .ok_or_else(overflow)?;
+ }
+ }
+ total_page_locations.checked_mul(2).ok_or_else(overflow)
+}
+
+/// Execution-memory-pool admission bound, in bytes, for one file's deletion-vector access
+/// plan -- reserved *before* calling `build_access_plan` (see [`attach_access_plan`]'s
+/// pre-reserve call site) to cover the larger of two peaks live at different points in the
+/// plan's lifetime. In practice the reader-normalization peak below dominates the construction
+/// peak unconditionally for any non-trivial input (`reader_peak_bytes(S, G) = (5S + 10G) *
+/// size_of::<RowSelector>()` always exceeds `CONSTRUCTION_PEAK_FACTOR * S *
+/// size_of::<RowSelector>() = 3S * size_of::<RowSelector>()` once `S >= 1`, since the `5S` term
+/// alone already exceeds `3S`); the construction term is retained as a documented floor rather
+/// than dropped, since it is cheap to compute and keeps this bound correct even if the reader's
+/// growth factors ever shrink below construction's.
+///
+/// - **Construction peak** (`CONSTRUCTION_PEAK_FACTOR * S`, see that constant's doc comment):
+/// live while `build_access_plan` builds the plan's `RowSelection`s. Construction's
+/// transient allocations fully unwind before `build_access_plan` returns, so this peak never
+/// overlaps the reader-normalization peak below.
+/// - **Reader-normalization peak** (`reader_peak_bytes(S + page_bound_selectors,
+/// num_row_groups)`, see that function): live later, once DataFusion's reader normalizes the
+/// attached plan. `S = 2 * cardinality + num_row_groups` is the same conservative bound on
+/// the plan's final retained selector count used for the construction peak -- it provably
+/// bounds `R = total_selectors(&plan)` (`R <= S`, from `build_access_plan`'s
+/// one-non-coalescing-selector-per-deleted-row worst case plus one boundary selector per row
+/// group), so `S + page_bound_selectors` bounds `R` after page-index inflation the same way
+/// `S` bounds `R` before it.
+///
+/// These two peaks never overlap in time, so `max` -- not `sum` -- is the correct combinator:
+/// reserving their sum would over-reserve for no safety benefit.
+///
+/// Deliberately not clamped by the file's total row count here, unlike the reader-peak target
+/// `attach_access_plan` resizes down to after construction (see that call site): `S`'s
+/// `+ num_row_groups` boundary term is a worst-case padding margin that can legitimately exceed
+/// the total row count for a small, heavily-deleted file, and admission sizing has no actual
+/// retained-selector count yet to clamp against -- only after construction, once `R` is known,
+/// is clamping to the total row count both meaningful and strictly tighter. Leaving this bound
+/// unclamped only ever makes admission more conservative, never less safe.
+///
+/// Uses checked arithmetic throughout: a cardinality, row-group count, or page bound large
+/// enough to overflow `usize` while computing this bound indicates a corrupted or malicious
+/// descriptor, reported as a clean error rather than panicking.
+fn admission_bound_bytes(
+ cardinality: i64,
+ num_row_groups: usize,
+ page_bound_selectors: usize,
+) -> Result<usize, ExecutionError> {
+ let overflow = || {
+ GeneralError(format!(
+ "Deletion vector admission bound overflowed for cardinality {cardinality}, \
+ {num_row_groups} row groups, and page bound {page_bound_selectors} selectors"
+ ))
+ };
+ let cardinality_usize = usize::try_from(cardinality).map_err(|_| overflow())?;
+ // S: the conservative bound on the plan's final *retained* selector count (what
+ // `total_selectors(&plan)` cannot exceed) -- unchanged from the pre-existing
+ // construction-only bound this function replaces.
+ let s = cardinality_usize
+ .checked_mul(2)
+ .and_then(|doubled| doubled.checked_add(num_row_groups))
+ .ok_or_else(overflow)?;
+
+ let construction_bytes = s
+ .checked_mul(size_of::<RowSelector>())
+ .and_then(|bytes| bytes.checked_mul(CONSTRUCTION_PEAK_FACTOR))
+ .ok_or_else(overflow)?;
+
+ let s_plus_page = s.checked_add(page_bound_selectors).ok_or_else(overflow)?;
+ let reader_bytes = reader_peak_bytes(s_plus_page, num_row_groups)?;
+
+ Ok(construction_bytes.max(reader_bytes))
+}
+
+/// Upper bound on concurrent DV-blob and footer fetches per partition. Both
+/// are small ranged reads, so a modest fan-out hides object-store latency
+/// without flooding the store client.
+const DV_FETCH_CONCURRENCY: usize = 8;
+
+/// Called via `block_on` at plan-creation time on the executor task: DV blobs
+/// are small ranged reads and footers are needed to learn row-group
+/// boundaries. Files are fetched concurrently (bounded by
+/// [`DV_FETCH_CONCURRENCY`]) with input order preserved. Footer fetches go
+/// through the scan's shared FileMetadataCache, so the scan's subsequent open
+/// of the same file is served from cache. That reuse relies on each input
+/// [`PartitionedFile`] being returned as-is (only `with_extension` applied),
+/// never rebuilt: the cache entry is keyed by this exact `object_meta` and the
+/// scan later looks it up through the same struct.
+///
+/// Deliberately takes no object-store options map and imports no
+/// store-construction helper: every [`DvScanFile`] arrives with its stores
+/// already resolved by the caller (see its doc comment), so this async path
+/// structurally cannot build an object store -- only `runtime_env` is still
+/// threaded through, for the shared `FileMetadataCache` and (per file) the
+/// execution `MemoryPool` each expanded access plan's row selectors are
+/// reserved against -- see [`DvAccessPlanReservation`].
+pub async fn attach_access_plans(
+ runtime_env: Arc<RuntimeEnv>,
+ files: Vec<DvScanFile>,
+) -> Result<Vec<PartitionedFile>, ExecutionError> {
+ futures::stream::iter(files)
+ .map(|scan_file| attach_access_plan(Arc::clone(&runtime_env), scan_file))
+ .buffered(DV_FETCH_CONCURRENCY)
+ .try_collect()
+ .await
+}
+
+/// Resolve one file's deletion vector into an attached [`ParquetAccessPlan`];
+/// files without a DV pass through untouched.
+async fn attach_access_plan(
+ runtime_env: Arc<RuntimeEnv>,
+ scan_file: DvScanFile,
+) -> Result<PartitionedFile, ExecutionError> {
+ let DvScanFile {
+ file,
+ file_path,
+ dv,
+ data_store,
+ dv_store,
+ } = scan_file;
+ let dv = match dv {
+ Some(dv) => dv,
+ None => return Ok(file),
+ };
+ // Delta's canonical `DeletionVectorDescriptor.EMPTY`: inline storage, empty
+ // payload, size 0, cardinality 0. Spark's reader returns all rows for it;
+ // decoding would fail (the empty payload is too short for a magic
+ // number), so pass the file through unchanged before attempting to read it.
+ if dv.cardinality == 0 && dv.size_in_bytes == 0 {
+ return Ok(file);
+ }
+ if dv.size_in_bytes < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative size {}",
+ dv.size_in_bytes
+ )));
+ }
+ if dv.cardinality < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative cardinality {}",
+ dv.cardinality
+ )));
+ }
+
+ let data: Vec<u8> = if let Some(inline) = dv.inline_data {
+ check_inline_payload_size(&file_path, &inline, dv.size_in_bytes)?;
+ inline
+ } else if let Some(dv_path) = &dv.absolute_path {
+ let offset = dv
+ .offset
+ .ok_or_else(|| GeneralError("On-disk deletion vector missing offset".into()))?;
+ if offset < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative offset {offset}"
+ )));
+ }
+ let offset = offset as u64;
+ // [i32 BE size][data: size_in_bytes][i32 BE crc]
+ let framed_len = 4 + dv.size_in_bytes as u64 + 4;
+ let (store, dv_store_path) = dv_store.ok_or_else(|| {
+ GeneralError(format!(
+ "Deletion vector for {file_path} has an absolute path but no pre-resolved object store"
+ ))
+ })?;
+ let blob = store
+ .get_range(&dv_store_path, offset..offset + framed_len)
+ .await
+ .map_err(|e| GeneralError(format!("Failed to read deletion vector {dv_path}: {e}")))?;
+ unframe_dv_blob(&blob, dv.size_in_bytes as usize)?.to_vec()
+ } else {
+ return Err(GeneralError(
+ "Deletion vector descriptor has neither inline data nor a path".into(),
+ ));
+ };
+ let deleted = deserialize_dv_bitmap(&data)
+ .map_err(|e| GeneralError(format!("Invalid deletion vector for {file_path}: {e}")))?;
+ validate_cardinality(&file_path, dv.cardinality, &deleted)?;
+
+ // Row-group boundaries come from the data file's footer, fetched through the scan's
+ // shared FileMetadataCache with the page index loaded eagerly and the scan's metadata
+ // size hint (mirroring EagerPageIndexReaderFactory): the one fetch here also serves the
+ // subsequent data-file open, so DV files pay no extra footer round-trip. Keyed by
+ // `file.object_meta`, the exact ObjectMeta the scan's reader factory will look up.
+ let metadata_cache = runtime_env.cache_manager.get_file_metadata_cache();
+ let metadata = DFParquetMetadata::new(data_store.as_ref(), &file.object_meta)
+ .with_file_metadata_cache(Some(metadata_cache))
+ .with_page_index_policy(Some(PageIndexPolicy::Optional))
+ .with_metadata_size_hint(Some(crate::parquet::parquet_exec::METADATA_SIZE_HINT))
+ .fetch_metadata()
+ .await
+ .map_err(|e| GeneralError(format!("Failed to read parquet footer of {file_path}: {e}")))?;
+ let row_counts: Vec<i64> = metadata
+ .row_groups()
+ .iter()
+ .map(|rg| rg.num_rows())
+ .collect();
+
+ // Pre-reserve the admission bound *before* calling build_access_plan: this bound covers
+ // both construction's own transient peak AND the larger peak DataFusion's reader hits
+ // later while normalizing the attached plan (`create_initial_plan`'s deep clone plus
+ // `into_overall_row_selection`'s combined RowSelection) -- see admission_bound_bytes and
+ // reader_peak_bytes. Reserving first means a rejection happens before any large `Vec` is
+ // allocated, not after -- see reader_peak_bytes's doc comment for the measured worst
+ // cases. The error message names this as a construction-phase rejection (contains
+ // "construct"), textually distinct from the steady-state message below, so callers/logs
+ // can tell which phase failed.
+ let page_bound_selectors = page_selection_bound_selectors(&metadata)?;
+ let admission_bytes =
+ admission_bound_bytes(dv.cardinality, row_counts.len(), page_bound_selectors)?;
+ let reservation =
+ MemoryConsumer::new("DeltaDeletionVectorAccessPlan").register(&runtime_env.memory_pool);
Review Comment:
Every DV'd file registers its own consumer here. Because the reservation lives in the `PartitionedFile` extensions, the registration stays until the plan is dropped at the end of the task. The default `fair_unified` pool rejects a request once the task's total would exceed `pool_size / num_consumers`, and it counts every consumer in the task. So a partition with many DV'd files lowers the limit for every other native operator in that task, including a hash join build that can't spill.
I reproduced this on Spark 3.5 with `spark.memory.offHeap.size=256m`. I used 16 files packed into one partition, each with a DV deleting a single row, joined to a broadcast of `range(1500000)`. With the native Delta scan it fails with `Failed to acquire 24000000 bytes where 11520 bytes already reserved and the fair limit is 14913080 bytes, 18 registered`. It passes with the contrib off, and it passes natively on the same rows written without DVs. The DV reservations add up to 11.5 KB, so the consumer count alone is failing the join.
Could `attach_access_plans` register one consumer per call and hand each file `new_empty()` from it? That shares one registration across the partition and keeps the per-file grow, resize and release. The existing reservation tests all use `GreedyMemoryPool`, which has no consumer-count term. A test pool that mirrors `CometFairMemoryPool`'s check, or one that just counts `register` calls, would pin this. Related, the `maxDeletedRowsPerFile` doc says the selectors are retained for the file's scan. They're actually held until the task finishes, for every file in the partition. Could the doc say that the cap bounds one file rather than what a task holds?
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so its rewrite (via
+ * [[CometScanWithPlanData]]) survives plan copies -- a transient field would be dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning (the CometIcebergNativeScanExec lesson).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node
+ // automatically gets a helper consistent with ITS runtimeFilters, avoiding the #3510 class of
+ // bug where a stored helper field desyncs from rewritten filters. Costs one extra file listing
+ // per executed instance; correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below, and AQE's UI
+ // plan-walk calls `.metrics` on every node MID-PLANNING, including while a DPP subquery is
+ // still an adaptive placeholder or a partition filter holds an unresolved ScalarSubquery (see
+ // `hasUnevaluableSubqueryFilter` below). That's safe ONLY because constructing `scanHelper` is a
+ // cheap case-class build with no file listing, and core's `CometScanExec.metrics` touches only
+ // `wrapped.driverMetrics` (populated by Spark's own planning) plus a static metric-node
+ // constructor -- neither file listing nor subquery resolution. If core's `metrics` ever touches
+ // either, forcing `scanHelper` here would resurrect the AQE mid-planning crashes this invariant
+ // prevents.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element, so checking runtimeFilters
+ // here avoids constructing/forcing the derived scanHelper just to read partitioning. The
+ // InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep in sync. The
+ // ScalarSubquery case is probed rather than treated as permanently unevaluable: Spark exposes no
+ // public finished/updated flag on ExecSubqueryExpression, but `eval()` doubles as one -- it only
+ // reads the cached `result` behind a `require(updated, ...)` guard, while the subquery is
+ // actually run by `updateResult()` (invoked separately during prepare/AQE), never by `eval()`.
+ // Once resolved, outputPartitioning below reports the real perPartitionData.length instead of
+ // staying at zero -- a fused native parent's buildNativeContext requires that count to match.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case s: ScalarSubquery => !isScalarSubqueryResolved(s)
+ case _ => false
+ })
+
+ // `eval()` never triggers the subquery's execution: on a resolved subquery it is a pure cached
+ // read of `result` (verified against bytecode: `Predef.require(updated(), ...)` then a plain
+ // field read), so this probe is safe to call repeatedly, including from AQE's mid-planning plan
+ // walks. Pre-resolution, the ONLY throw is `require`'s `IllegalArgumentException`; catch exactly
+ // that, since anything else escaping is a genuine bug we must not mask as unpartitioned.
+ private def isScalarSubqueryResolved(s: ScalarSubquery): Boolean =
+ try {
+ s.eval()
+ true
+ } catch {
+ case _: IllegalArgumentException => false
+ }
+
+ private def isAdaptivePlaceholder(p: SparkPlan): Boolean = p match {
+ case ReusedSubqueryExec(inner) => isAdaptivePlaceholder(inner)
+ case _: CometSubqueryAdaptiveBroadcastExec => true
+ case _: SubqueryAdaptiveBroadcastExec => true
+ case _ => false
+ }
+
+ override lazy val outputOrdering: Seq[SortOrder] = originalPlan.outputOrdering
+
+ override def dynamicPruningFilters: Seq[Expression] = runtimeFilters
+
+ override def withDynamicPruningFilters(filters: Seq[Expression]): SparkPlan = {
+ // A real copy: runtimeFilters is a constructor field included in equality, so the copy
+ // survives enclosing-block rebuilds, and the derived scanHelper picks up the rewritten
+ // filters automatically.
+ copy(runtimeFilters = filters)
+ }
+
+ /**
+ * Lazy split-mode serialization, mirroring CometNativeScanExec: common data was serialized at
+ * planning; per-partition file lists serialize here, at execution time.
+ */
+ @transient private lazy val serializedPartitionData
+ : (Array[Byte], Array[Array[Byte]], Array[Seq[String]]) = {
+ // Resolve the helper's DPP subqueries: it holds its own InSubqueryExec instances that
+ // Spark's expressions walk does not see (the helper is derived, not a child).
+ scanHelper.partitionFilters.foreach {
+ case DynamicPruningExpression(e: InSubqueryExec) if e.values().isEmpty =>
+ e.updateResult()
+ case _ =>
+ }
+
+ val commonBytes = {
+ val deltaScan = DeltaSparkScanEnvelope.unpack(nativeOp)
+ // Scalar subqueries in dataFilters were unresolved at planning; resolve them now and
+ // append them as pushed filters, as CometNativeScanExec.serializedPartitionData does.
+ // has_data_filters follows their presence, not the serialized count: a filter that fails
+ // to serialize still keeps native on the safe timestamp conversion for a filtered scan.
+ val resolved = org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .resolvedSubqueryFilters(dataFilters, output, requiredSchema, conf)
+ val common = if (!resolved.hasResolvedFilters) {
+ deltaScan.getCommon
+ } else {
+ val builder = deltaScan.getCommon.toBuilder
+ builder.setHasDataFilters(true)
+ resolved.protos.foreach(builder.addDataFilters)
+ builder.build()
+ }
+ OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaScan.getDeltaCommon)
+ .build()
+ .toByteArray
+ }
+
+ val filePartitions = scanHelper.getFilePartitions()
+
+ val tableRoot = DeltaSparkScanEnvelope.unpack(nativeOp).getDeltaCommon.getTableRoot
+ val perPartitionBytes = filePartitions.map { filePartition =>
+ org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .serializePartition(filePartition, originalPlan, tableRoot)
+ }.toArray
+
+ val perPartitionPaths = filePartitions.map(_.files.map(_.filePath.toString).toSeq).toArray
+
+ (commonBytes, perPartitionBytes, perPartitionPaths)
+ }
+
+ override def commonData: Array[Byte] = serializedPartitionData._1
+
+ override def perPartitionData: Array[Array[Byte]] = serializedPartitionData._2
+
+ def perPartitionFilePaths: Array[Seq[String]] = serializedPartitionData._3
+
+ override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+ val nativeMetrics = CometMetricNode.fromCometPlan(this)
+ val serializedPlan = CometExec.serializeNativePlan(nativeOp)
+
+ new CometExecRDD(
+ sparkContext,
+ Seq.empty,
+ Map(sourceKey -> commonData),
+ Map(sourceKey -> perPartitionData),
+ serializedPlan,
+ PlanDataInjector.planFingerprint(serializedPlan),
+ perPartitionData.length,
+ output.length,
+ nativeMetrics,
+ Seq.empty,
+ None,
+ Seq.empty,
+ perPartitionFilePaths = perPartitionFilePaths,
+ reportScanInputMetrics = true)
+ }
+
+ override def doCanonicalize(): CometDeltaNativeScanExec = {
+ val canonOriginal = if (originalPlan != null) {
+ val stripped = originalPlan.copy(partitionFilters =
+ CometScanUtils.filterUnusedDynamicPruningExpressions(originalPlan.partitionFilters))
+ stripped.doCanonicalize()
+ } else {
+ null
+ }
+ CometDeltaNativeScanExec(
+ nativeOp,
+ output.map(QueryPlan.normalizeExpressions(_, output)),
+ requiredSchema,
+ QueryPlan.normalizePredicates(
+ CometScanUtils.filterUnusedDynamicPruningExpressions(runtimeFilters),
+ output),
+ QueryPlan.normalizePredicates(dataFilters, output),
+ relation,
+ canonOriginal,
+ SerializedPlan(None),
+ "")
+ }
+
+ override def stringArgs: Iterator[Any] = Iterator(output, runtimeFilters)
+
+ override def equals(obj: Any): Boolean = obj match {
+ case other: CometDeltaNativeScanExec =>
+ this.originalPlan == other.originalPlan &&
+ this.serializedPlanOpt == other.serializedPlanOpt &&
+ this.runtimeFilters == other.runtimeFilters &&
+ this.dataFilters == other.dataFilters
+ case _ => false
+ }
+
+ override def hashCode(): Int =
+ java.util.Objects.hash(originalPlan, serializedPlanOpt, runtimeFilters, dataFilters)
+
+ private val driverMetricKeys =
+ Set(
+ "numFiles",
+ "filesSize",
+ "numPartitions",
+ "metadataTime",
+ "staticFilesNum",
+ "staticFilesSize",
+ "pruningTime")
+
+ // Forces `scanHelper` (see its doc above for why that -- and reading `.metrics` off it -- is
+ // safe even when AQE calls `.metrics` mid-planning against an unresolved DPP/scalar subquery).
+ override lazy val metrics: Map[String, SQLMetric] = {
+ CometMetricNode.nativeScanMetrics(session.sparkContext) ++
+ scanHelper.metrics.filter { case (k, _) => driverMetricKeys.contains(k) }
+ }
+}
+
+object CometDeltaNativeScanExec {
+
+ /** File-planning helper: reuses CometScanExec's listing/splitting/DPP machinery. */
+ def planningHelper(
Review Comment:
Claimed DV scans do get split today. The claim requires `optimizationsEnabled`, `DeltaParquetFileFormat.isSplitable` returns exactly that, and `isNeededForSchema` is `false` in both the 3.5 and 4.x shims. So this helper splits any DV'd file bigger than `maxSplitBytes`, which by default is every file over 128 MB. The results are right. A single 177 KB DV'd file with small row groups, read with `spark.sql.files.maxPartitionBytes=4096`, came out as 44 native partitions and matched Spark. That works because DataFusion's `prune_by_range` skips row groups whose first page is outside the split, while the access plan stays in file coordinates.
Could we add a test along those lines? It's the default shape for large files, and #5655 plans to change exactly this path. Could you also update #5655, which says splitting is disabled? Today every split fetches and decodes the whole DV, reads the footer, builds the whole-file plan and reserves memory for the whole file.
##########
contrib/delta-spark/src/test/scala/org/apache/comet/contrib/delta/CometDeltaNativeScanSuite.scala:
##########
@@ -0,0 +1,3584 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.File
+
+import scala.collection.mutable
+import scala.collection.mutable.ListBuffer
+import scala.concurrent.duration.DurationInt
+
+import org.apache.spark.scheduler.{SparkListener, SparkListenerTaskEnd}
+import org.apache.spark.sql.{DataFrame, Row}
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, DynamicPruningExpression, NamedExpression, StructsToJson}
+import org.apache.spark.sql.comet.CometDeltaNativeScanExec
+import org.apache.spark.sql.execution.{FileSourceScanExec, QueryExecution, ScalarSubquery, SparkPlan, SubqueryExec}
+import org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec
+import org.apache.spark.sql.functions.{col, lit, to_json}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ByteType, LongType, StringType, StructField, StructType}
+import org.apache.spark.sql.util.QueryExecutionListener
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus
+import org.apache.comet.ExtendedExplainInfo
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.operator.CometNativeScan
+
+/**
+ * Differential suite: append-only Delta tables read through the native Delta scan must produce
+ * results identical to Spark's Delta reader, engage the native operator, and prune at row-group
+ * and page level.
+ */
+class CometDeltaNativeScanSuite extends CometDeltaTestBase {
+
+ test("plain delta table reads natively with identical results") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark
+ .range(0, 1000)
+ .selectExpr("id", "id * 2 as v", "cast(id as string) as s")
+ .write
+ .format("delta")
+ .save(path)
+
+ val df = spark.read.format("delta").load(path).filter(col("id") > 500)
+ checkDeltaNativeScanAnswer(df)
+ }
+ }
+
+ test("projection and filter on delta table") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark
+ .range(0, 1000)
+ .selectExpr("id", "id % 10 as bucket", "cast(id as double) as d")
+ .write
+ .format("delta")
+ .save(path)
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .select("bucket", "d")
+ .filter(col("d") < 100.0)
+ checkDeltaNativeScanAnswer(df)
+ }
+ }
+
+ test("partitioned delta table with partition filter") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark
+ .range(0, 1000)
+ .selectExpr("id", "id % 7 as p")
+ .write
+ .format("delta")
+ .partitionBy("p")
+ .save(path)
+
+ val df = spark.read.format("delta").load(path).filter(col("p") === 3)
+ checkDeltaNativeScanAnswer(df)
+ assert(df.count() > 0)
+ }
+ }
+
+ test("multi-file delta table after several appends") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ for (i <- 0 until 4) {
+ spark
+ .range(i * 100, (i + 1) * 100)
+ .selectExpr("id", "id * 3 as v")
+ .write
+ .format("delta")
+ .mode("append")
+ .save(path)
+ }
+ val df = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(df)
+ assert(df.count() == 400)
+ }
+ }
+
+ test("time travel VERSION AS OF reads natively") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark.range(0, 100).write.format("delta").save(path)
+ spark.range(100, 200).write.format("delta").mode("append").save(path)
+
+ val v0 = spark.read.format("delta").option("versionAsOf", 0).load(path)
+ checkDeltaNativeScanAnswer(v0)
+ assert(v0.count() == 100)
+ }
+ }
+
+ test("selective predicate prunes row groups and pages") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ // Small row groups + page-level stats: sorted data so min/max stats are tight. The Delta
+ // writer ignores parquet.* DataFrameWriter options, so set them on the Hadoop conf.
+ val hadoopConf = spark.sparkContext.hadoopConfiguration
+ val oldBlockSize = hadoopConf.get("parquet.block.size")
+ val oldPageSize = hadoopConf.get("parquet.page.size")
+ hadoopConf.setInt("parquet.block.size", 256 * 1024)
+ hadoopConf.setInt("parquet.page.size", 16 * 1024)
+ try {
+ spark
+ .range(0, 500000)
+ .selectExpr("id", "id * 2 as v")
+ .sort("id")
+ .coalesce(1)
+ .write
+ .format("delta")
+ .save(path)
+ } finally {
+ if (oldBlockSize == null) hadoopConf.unset("parquet.block.size")
+ else hadoopConf.set("parquet.block.size", oldBlockSize)
+ if (oldPageSize == null) hadoopConf.unset("parquet.page.size")
+ else hadoopConf.set("parquet.page.size", oldPageSize)
+ }
+
+ def query = spark.read
+ .format("delta")
+ .load(path)
+ .filter(col("id") >= 100 && col("id") < 200)
+ checkDeltaNativeScanAnswer(query)
+
+ // checkSparkAnswer re-plans the query, so read metrics from a DataFrame we execute
+ // ourselves (collect() runs THIS Dataset's queryExecution; count() would plan a new one):
+ // its executed plan holds the metric objects native execution updated.
+ val df = query
+ assert(df.collect().length == 100)
+ val scans = deltaNativeScans(df)
+ assert(scans.size == 1)
+ val metrics = scans.head.metrics
+ val rowGroupsPruned = metrics.get("row_groups_pruned_statistics").map(_.value).getOrElse(0L)
+ val pagesPruned = metrics.get("page_index_rows_pruned").map(_.value).getOrElse(0L)
+ assert(
+ rowGroupsPruned > 0,
+ s"expected row-group pruning; metrics: ${metrics.map { case (k, v) => s"$k=${v.value}" }}")
+ assert(
+ pagesPruned > 0,
+ s"expected page-index pruning; metrics: ${metrics.map { case (k, v) =>
+ s"$k=${v.value}"
+ }}")
+ }
+ }
+
+ test("scalar subquery data filter is pushed down and prunes row groups and pages") {
+ withTempPath { dir =>
+ val path = s"${dir.getAbsolutePath}/data"
+ val thresholds = s"${dir.getAbsolutePath}/thresholds"
+ // Same layout as the selective-predicate test: small row groups + tight page stats.
+ val hadoopConf = spark.sparkContext.hadoopConfiguration
+ val oldBlockSize = hadoopConf.get("parquet.block.size")
+ val oldPageSize = hadoopConf.get("parquet.page.size")
+ hadoopConf.setInt("parquet.block.size", 256 * 1024)
+ hadoopConf.setInt("parquet.page.size", 16 * 1024)
+ try {
+ spark
+ .range(0, 500000)
+ .selectExpr("id", "id * 2 as v")
+ .sort("id")
+ .coalesce(1)
+ .write
+ .format("delta")
+ .save(path)
+ } finally {
+ if (oldBlockSize == null) hadoopConf.unset("parquet.block.size")
+ else hadoopConf.set("parquet.block.size", oldBlockSize)
+ if (oldPageSize == null) hadoopConf.unset("parquet.page.size")
+ else hadoopConf.set("parquet.page.size", oldPageSize)
+ }
+ spark
+ .sql("SELECT CAST(100 AS BIGINT) AS lo, CAST(200 AS BIGINT) AS hi")
+ .write
+ .format("delta")
+ .save(thresholds)
+
+ // Scalar subqueries are PlanExpressions: unresolved at planning, so the bounds can
+ // only reach the native reader via the execution-time resolve-and-append path.
+ def query = spark.sql(
+ s"SELECT * FROM delta.`$path` WHERE id >= (SELECT lo FROM delta.`$thresholds`) " +
+ s"AND id < (SELECT hi FROM delta.`$thresholds`)")
+ checkDeltaNativeScanAnswer(query)
+
+ val df = query
+ assert(df.collect().length == 100)
+ // The thresholds table inside the subquery is also claimed natively; pick the
+ // main data-table scan by its output.
+ assertSubqueryFilterPushed(df, dataColumn = "v")
+ val scans = deltaNativeScans(df).filter(_.output.exists(_.name == "v"))
+ assert(scans.size == 1)
+ val metrics = scans.head.metrics
+ val rowGroupsPruned = metrics.get("row_groups_pruned_statistics").map(_.value).getOrElse(0L)
+ val pagesPruned = metrics.get("page_index_rows_pruned").map(_.value).getOrElse(0L)
+ assert(
+ rowGroupsPruned > 0,
+ s"expected row-group pruning from the resolved subquery bounds; metrics: ${metrics.map {
+ case (k, v) => s"$k=${v.value}"
+ }}")
+ assert(
+ pagesPruned > 0,
+ s"expected page-index pruning from the resolved subquery bounds; metrics: ${metrics.map {
+ case (k, v) => s"$k=${v.value}"
+ }}")
+ }
+ }
+
+ test("deletion vectors: scalar subquery filter composes with DV application") {
+ withTempPath { dir =>
+ val path = s"${dir.getAbsolutePath}/data"
+ val thresholds = s"${dir.getAbsolutePath}/thresholds"
+ createDvTable(path, rows = 10000)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+ spark
+ .sql("SELECT CAST(5000 AS BIGINT) AS lo")
+ .write
+ .format("delta")
+ .save(thresholds)
+
+ def query =
+ spark.sql(s"SELECT * FROM delta.`$path` WHERE id >= (SELECT lo FROM delta.`$thresholds`)")
+ checkDeltaNativeScanAnswer(query)
+ // Deleted rows must stay deleted with the pushed bound applied in-scan.
+ val df = query
+ val rows = df.collect()
+ assert(rows.length == 2500)
+ assert(rows.forall(r => r.getLong(0) % 2 == 1 && r.getLong(0) >= 5000))
+ assertSubqueryFilterPushed(df, dataColumn = "v")
+ }
+ }
+
+ test("column mapping: scalar subquery filter on a renamed column") {
+ withTempPath { dir =>
+ val path = s"${dir.getAbsolutePath}/data"
+ val thresholds = s"${dir.getAbsolutePath}/thresholds"
+ spark.range(0, 1000).selectExpr("id", "id * 2 as v").write.format("delta").save(path)
+ enableColumnMapping(path)
+ spark.sql(s"ALTER TABLE delta.`$path` RENAME COLUMN v TO w")
+ spark
+ .sql("SELECT CAST(900 AS BIGINT) AS lo")
+ .write
+ .format("delta")
+ .save(thresholds)
+
+ // The pushed filter references the renamed column: it must bind against the
+ // physical read schema, not the logical name.
+ def query =
+ spark.sql(s"SELECT * FROM delta.`$path` WHERE w >= (SELECT lo FROM delta.`$thresholds`)")
+ checkDeltaNativeScanAnswer(query)
+ val df = query
+ assert(df.collect().length == 550)
+ assertSubqueryFilterPushed(df, dataColumn = "w")
+ }
+ }
+
+ /**
+ * Assert the resolved scalar-subquery bound was actually appended to the native scan's
+ * execution-time common data (answers alone cannot show this: Spark's covering FilterExec would
+ * mask a silently-skipped pushdown). `df` must already have been executed.
+ */
+ private def assertSubqueryFilterPushed(df: DataFrame, dataColumn: String): Unit = {
+ val scans = deltaNativeScans(df).collect {
+ case s: CometDeltaNativeScanExec if s.output.exists(_.name == dataColumn) => s
+ }
+ assert(scans.size == 1)
+ val scan = scans.head
+ val planTimeFilters =
+ DeltaSparkScanEnvelope.unpack(scan.nativeOp).getCommon.getDataFiltersCount
+ val executedFilters = OperatorOuterClass.DeltaSparkScan
+ .parseFrom(scan.commonData)
+ .getCommon
+ .getDataFiltersCount
+ assert(
+ executedFilters > planTimeFilters,
+ "expected resolved subquery filters appended at execution: " +
+ s"plan-time=$planTimeFilters executed=$executedFilters " +
+ s"dataFilters=${scan.dataFilters.mkString("; ")}")
+ }
+
+ test("scalar subquery filter is NOT pushed below a limit") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark.range(0, 3).selectExpr("id").write.format("delta").save(path)
+ spark.read.format("delta").load(path).createOrReplaceTempView("t_limit_pushdown")
+
+ val df = spark.sql(
+ "SELECT id FROM (SELECT id FROM t_limit_pushdown ORDER BY id LIMIT 1) q " +
+ "WHERE id > (SELECT max(id) FROM range(1))")
+ checkSparkAnswer(df)
+ assert(df.collect().isEmpty)
+ assertNoSubqueryFilterPushed(df)
+ }
+ }
+
+ test("scalar subquery filter is NOT pushed across a nondeterministic projection") {
+ withSQLConf(CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark.range(0, 5).coalesce(1).write.format("delta").save(path)
+ spark.read.format("delta").load(path).createOrReplaceTempView("t_monotonic_id")
+
+ // A deterministic conjunct does not commute with a nondeterministic projection: the
+ // subquery bound must not be pushed into the scan below `seq`, or the surviving rows'
+ // monotonically_increasing_id() values change and the answer is wrong.
+ val df = spark.sql(
+ "SELECT id FROM (SELECT id, monotonically_increasing_id() AS seq " +
+ "FROM t_monotonic_id) q WHERE id > (SELECT max(id) FROM range(1)) AND seq = 1")
+ checkSparkAnswer(df)
+ assert(df.collect().toSeq == Seq(Row(1)))
+ assertNoSubqueryFilterPushed(df)
+ }
+ }
+ }
+
+ /**
+ * Assert no scalar-subquery filter was harvested and pushed into the native scan's
+ * execution-time common data: the scan must sit below a non-commuting operator (e.g. LIMIT /
+ * TopN), so the covering FilterExec's predicate must stay above it rather than move into the
+ * scan. Also confirms the query still engaged the native Delta scan, i.e. this exercises the
+ * commutativity guard rather than a plan that fell back to Spark entirely. `df` must already
+ * have been executed.
+ */
+ private def assertNoSubqueryFilterPushed(df: DataFrame): Unit = {
+ val scans = deltaNativeScans(df).collect { case s: CometDeltaNativeScanExec => s }
+ assert(scans.size == 1, s"expected exactly one native Delta scan; found ${scans.size}")
+ val scan = scans.head
+ val planTimeFilters =
+ DeltaSparkScanEnvelope.unpack(scan.nativeOp).getCommon.getDataFiltersCount
+ val executedFilters = OperatorOuterClass.DeltaSparkScan
+ .parseFrom(scan.commonData)
+ .getCommon
+ .getDataFiltersCount
+ assert(
+ executedFilters == planTimeFilters,
+ "expected no subquery filter pushed across the non-commuting operator between the " +
+ s"covering filter and the scan: plan-time=$planTimeFilters executed=$executedFilters " +
+ s"dataFilters=${scan.dataFilters.mkString("; ")}")
+ }
+
+ test("scalar subquery filter rejected by serde still marks the scan as filtered") {
+ withTempPath { dir =>
+ val path = s"${dir.getAbsolutePath}/data"
+ val bounds = s"${dir.getAbsolutePath}/bounds"
+ spark.range(0, 100).selectExpr("id", "id * 2 as v").write.format("delta").save(path)
+ spark.sql("SELECT CAST(42 AS BIGINT) AS lo").write.format("delta").save(bounds)
+
+ // With EqualNullSafe disabled the resolved bound cannot serialize, yet the scan must still
+ // carry has_data_filters so native treats it as a filtered read, exactly like core does.
+ withSQLConf("spark.comet.expression.EqualNullSafe.enabled" -> "false") {
+ def query =
+ spark.sql(
+ s"SELECT * FROM delta.`$path` WHERE id <=> (SELECT max(lo) FROM delta.`$bounds`)")
+ checkDeltaNativeScanAnswer(query)
+ val df = query
+ assert(df.collect().toSeq == Seq(Row(42L, 84L)))
+ assertUnserializedSubqueryFilterMarksScanFiltered(df, dataColumn = "v")
+ }
+ }
+ }
+
+ test("unserializable scalar subquery filter keeps the safe TIMESTAMP_MILLIS conversion") {
+ // Same fixture as core's "filtered TIMESTAMP_MILLIS scans do not convert values Spark can
+ // skip": a raw file whose only overflowing millisecond value Spark prunes from the footer
+ // statistics once the resolved bound is pushed, so native must not convert it either.
+ withTempPath { dir =>
+ val path = s"${dir.getAbsolutePath}/data"
+ val bounds = s"${dir.getAbsolutePath}/bounds"
+ writeRawParquetFile(
+ path,
+ """message root {
+ | optional int32 id;
+ | optional int64 ts(TIMESTAMP_MILLIS);
+ |}""".stripMargin) { factory =>
+ (1 to 16).map(id => factory.newGroup().append("id", id).append("ts", 1717243200000L)) :+
+ factory.newGroup().append("id", 17).append("ts", 9223372036854776L)
+ }
+ spark.sql(s"CONVERT TO DELTA parquet.`$path` NO STATISTICS")
+ spark.sql("SELECT timestamp_seconds(0) AS bound").write.format("delta").save(bounds)
+
+ withSQLConf(
+ "spark.comet.expression.EqualNullSafe.enabled" -> "false",
+ "spark.sql.parquet.datetimeRebaseModeInRead" -> "CORRECTED",
+ "spark.sql.parquet.int96RebaseModeInRead" -> "CORRECTED") {
+ def query = spark.sql(
+ s"SELECT id, ts FROM delta.`$path` " +
+ s"WHERE ts <=> (SELECT max(bound) FROM delta.`$bounds`)")
+ // Spark 3.x never pushes subquery filters into its parquet reader and converts the
+ // overflowing value itself, so the answer comparison is meaningful on Spark 4.0+ only.
+ if (isSpark40Plus) {
+ checkDeltaNativeScanAnswer(query)
+ }
+ val df = query
+ assert(df.collect().isEmpty)
+ assert(
+ deltaNativeScans(df).nonEmpty,
+ s"expected a native Delta scan:\n${df.queryExecution}")
+ assertUnserializedSubqueryFilterMarksScanFiltered(df, dataColumn = "ts")
+ }
+ }
+ }
+
+ /**
+ * Assert the execution-time common data of the scan producing `dataColumn` reports
+ * `has_data_filters` with no serialized data filter: the plan-time proto carries neither, and
+ * the resolved subquery filter is the only data filter, so only the execution-time path can set
+ * the bit. `df` must already have been executed.
+ */
+ private def assertUnserializedSubqueryFilterMarksScanFiltered(
+ df: DataFrame,
+ dataColumn: String): Unit = {
+ val scans = deltaNativeScans(df).collect {
+ case s: CometDeltaNativeScanExec if s.output.exists(_.name == dataColumn) => s
+ }
+ assert(scans.size == 1, s"expected exactly one native Delta scan; found ${scans.size}")
+ val scan = scans.head
+ assert(
+ scan.dataFilters.exists(_.exists(_.isInstanceOf[ScalarSubquery])),
+ s"expected a scalar subquery data filter: ${scan.dataFilters.mkString("; ")}")
+ val planTime = DeltaSparkScanEnvelope.unpack(scan.nativeOp).getCommon
+ assert(!planTime.getHasDataFilters && planTime.getDataFiltersCount == 0)
+ val executed = OperatorOuterClass.DeltaSparkScan.parseFrom(scan.commonData).getCommon
+ assert(
+ executed.getHasDataFilters,
+ "expected has_data_filters at execution even though the resolved subquery filter did " +
+ s"not serialize: dataFilters=${scan.dataFilters.mkString("; ")}")
+ assert(executed.getDataFiltersCount == 0)
+ }
+
+ test("aggregation over delta table") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark
+ .range(0, 10000)
+ .selectExpr("id", "id % 13 as g", "id * 2 as v")
+ .write
+ .format("delta")
+ .save(path)
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .groupBy("g")
+ .sum("v")
+ checkDeltaNativeScanAnswer(df)
+ }
+ }
+
+ test("conf disables the native delta scan") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark.range(0, 100).write.format("delta").save(path)
+
+ withSQLConf(DeltaScanConf.COMET_DELTA_NATIVE_ENABLED.key -> "false") {
+ val df = spark.read.format("delta").load(path)
+ checkSparkAnswer(df)
+ assert(deltaNativeScans(df).isEmpty)
+ }
+ }
+ }
+
+ test("native delta scan is opt-in: disabled when the conf is not set") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark.range(0, 100).write.format("delta").save(path)
+
+ // The suite base enables the scan globally; drop the key entirely to
+ // observe the out-of-the-box default.
+ spark.conf.unset(DeltaScanConf.COMET_DELTA_NATIVE_ENABLED.key)
+ try {
+ assert(!DeltaScanConf.scanEnabled)
+ val df = spark.read.format("delta").load(path)
+ checkSparkAnswer(df)
+ assert(deltaNativeScans(df).isEmpty)
+ } finally {
+ spark.conf.set(DeltaScanConf.COMET_DELTA_NATIVE_ENABLED.key, "true")
+ }
+ }
+ }
+
+ test("table root under a directory whose name contains a newline falls back to Spark") {
+ // object_store recognizes the `file` scheme but rejects the control character in the
+ // directory name (`%0A` in the URI), so native execution could not open the table where
+ // Spark's Hadoop-backed reader can. The claim gate must decline before native planning.
+ withTempPath { dir =>
+ val path = new File(new File(dir, "dir\n"), "data").getAbsolutePath
+ spark.range(0, 100).write.format("delta").save(path)
+
+ val df = spark.read.format("delta").load(path)
+ assert(
+ deltaNativeScans(df).isEmpty,
+ "Expected no native Delta scan under a newline directory:\n" +
+ s"${df.queryExecution.executedPlan}")
+ checkSparkAnswerAndFallbackReason(
+ df,
+ "Native Delta scan cannot open path 'file:" + dir.getAbsolutePath +
+ "/dir%0A/data': object_store rejects it")
+ }
+ }
+
+ test("shallow clone whose source data files sit under a newline directory falls back") {
+ // The clone's own root is an ordinary path, so only the selected data files (resolved to
+ // the source table's directory) carry the rejected segment: this exercises the
+ // selected-paths probe, not the root gate. The reason names the first such complete path,
+ // a data file under the source directory.
+ withTempPath { dir =>
+ val sourcePath = new File(new File(dir, "dir\n"), "source").getAbsolutePath
+ val clonePath = new File(dir, "clone").getAbsolutePath
+ spark.range(0, 100).write.format("delta").save(sourcePath)
+ spark.sql(s"CREATE TABLE delta.`$clonePath` SHALLOW CLONE delta.`$sourcePath`")
+
+ val df = spark.read.format("delta").load(clonePath)
+ assert(
+ deltaNativeScans(df).isEmpty,
+ "Expected no native Delta scan for a clone of a newline-directory source:\n" +
+ s"${df.queryExecution.executedPlan}")
+ checkSparkAnswerAndFallbackReason(
+ df,
+ "Native Delta scan cannot open path 'file:" + dir.getAbsolutePath +
+ "/dir%0A/source/")
+ }
+ }
+
+ test("converted Parquet table with a newline in a data file basename falls back to Spark") {
+ // CONVERT TO DELTA keeps the existing Parquet file names, so the rejected character sits in
+ // the basename rather than a directory segment: the table root and every parent directory
+ // pass the path probe, and only a check of the complete selected path can decline.
+ withTempPath { dir =>
+ val path = new File(dir, "data").getAbsolutePath
+ spark.range(0, 100).repartition(2).write.parquet(path)
+ val original = new File(path).listFiles().filter(_.getName.endsWith(".parquet")).head
+ val renamed = new File(path, "part-00000\n.snappy.parquet")
+ java.nio.file.Files.move(original.toPath, renamed.toPath)
+ spark.sql(s"CONVERT TO DELTA parquet.`$path`")
+
+ val df = spark.read.format("delta").load(path)
+ assert(
+ deltaNativeScans(df).isEmpty,
+ "Expected no native Delta scan for a converted table with a newline basename:\n" +
+ s"${df.queryExecution.executedPlan}")
+ checkSparkAnswerAndFallbackReason(
+ df,
+ s"Native Delta scan cannot open path 'file:$path/part-00000%0A.snappy.parquet': " +
+ "object_store rejects it")
+ }
+ }
+
+ private def createDvTable(path: String, rows: Long = 1000): Unit = {
+ spark.range(0, rows).selectExpr("id", "id * 2 as v").write.format("delta").save(path)
+ spark.sql(
+ s"ALTER TABLE delta.`$path` SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true')")
+ }
+
+ /**
+ * Same shape as `createDvTable`, plus one extra TINYINT column (value 7) under `columnName`.
+ */
+ private def createDvTableWithExtraColumn(
+ path: String,
+ columnName: String,
+ rows: Long = 1000): Unit = {
+ spark
+ .range(0, rows)
+ .selectExpr("id", s"cast(7 as tinyint) as `$columnName`")
+ .write
+ .format("delta")
+ .save(path)
+ spark.sql(
+ s"ALTER TABLE delta.`$path` SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true')")
+ }
+
+ test("deletion vectors: DELETE-produced DVs read natively with correct results") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ val df = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(df)
+ assert(df.count() == 500)
+ }
+ }
+
+ test(
+ "deletion vectors: user column named like the synthetic internal-column slot keeps its " +
+ "own values") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ val collidingName = "_comet_delta___delta_internal_is_row_deleted"
+ createDvTableWithExtraColumn(path, collidingName)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id = 0")
+
+ val df = spark.read.format("delta").load(path).select("id", collidingName)
+ checkDeltaNativeScanAnswer(df)
+ val survivingValues = df.collect().map(_.getAs[Byte](collidingName)).distinct
+ assert(
+ survivingValues.sameElements(Array(7.toByte)),
+ "expected the user column's own value (7) to survive DV filtering, " +
+ s"got ${survivingValues.toSeq}")
+ }
+ }
+
+ test("deletion vectors: normally named extra column alongside DVs reads natively") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTableWithExtraColumn(path, "tag")
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id = 0")
+
+ val df = spark.read.format("delta").load(path).select("id", "tag")
+ checkDeltaNativeScanAnswer(df)
+ val survivingValues = df.collect().map(_.getAs[Byte]("tag")).distinct
+ assert(
+ survivingValues.sameElements(Array(7.toByte)),
+ "expected the extra column's value (7) to survive DV filtering, got " +
+ survivingValues.toSeq)
+ }
+ }
+
+ test("deletion vectors: UPDATE-produced DVs read natively with correct results") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"UPDATE delta.`$path` SET v = -1 WHERE id < 100")
+
+ val df = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(df)
+ assert(df.filter(col("v") === -1).count() == 100)
+ assert(df.count() == 1000)
+ }
+ }
+
+ test("deletion vectors: multiple DELETEs accumulate correctly") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 3 = 0")
+
+ val df = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(df)
+ // odd ids not divisible by 3
+ assert(df.count() == (0L until 1000L).count(i => i % 2 != 0 && i % 3 != 0))
+ }
+ }
+
+ test(
+ "deletion vectors: maxDeletedRowsPerFile budget declines an oversized DV and " +
+ "claims once raised") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ // repartition(4) guarantees >= 2 physical files so the per-file cardinality gate has
+ // more than one file to inspect, mirroring design F3's multi-file test shape.
+ spark
+ .range(0, 1000)
+ .selectExpr("id", "id * 2 as v")
+ .repartition(4)
+ .write
+ .format("delta")
+ .save(path)
+ spark.sql(
+ s"ALTER TABLE delta.`$path` SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true')")
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ withSQLConf(DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key -> "1") {
+ val df = spark.read.format("delta").load(path)
+ checkSparkAnswer(df)
+ assert(
+ deltaNativeScans(df).isEmpty,
+ "a budget of 1 deleted row per file must decline every DV-bearing file")
+ }
+
+ withSQLConf(DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key -> "1000000") {
+ val df = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(df)
+ }
+ }
+ }
+
+ test("deletion vectors: maxDeletedRowsPerFile decline reason names the conf key") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ withSQLConf(DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key -> "1") {
+ checkSparkAnswerAndFallbackReason(
+ spark.read.format("delta").load(path),
+ DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key)
+ }
+ }
+ }
+
+ test("deletion vectors: fully-deleted region and selective predicate still prune pages") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ val hadoopConf = spark.sparkContext.hadoopConfiguration
+ val oldBlockSize = hadoopConf.get("parquet.block.size")
+ val oldPageSize = hadoopConf.get("parquet.page.size")
+ hadoopConf.setInt("parquet.block.size", 256 * 1024)
+ hadoopConf.setInt("parquet.page.size", 16 * 1024)
+ try {
+ spark
+ .range(0, 500000)
+ .selectExpr("id", "id * 2 as v")
+ .sort("id")
+ .coalesce(1)
+ .write
+ .format("delta")
+ .save(path)
+ } finally {
+ if (oldBlockSize == null) hadoopConf.unset("parquet.block.size")
+ else hadoopConf.set("parquet.block.size", oldBlockSize)
+ if (oldPageSize == null) hadoopConf.unset("parquet.page.size")
+ else hadoopConf.set("parquet.page.size", oldPageSize)
+ }
+ spark.sql(
+ s"ALTER TABLE delta.`$path` SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true')")
+ // Delete a slice inside the predicate range and a large slice outside it.
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id >= 150 AND id < 160")
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id >= 300000")
+
+ def query = spark.read
+ .format("delta")
+ .load(path)
+ .filter(col("id") >= 100 && col("id") < 200)
+ checkDeltaNativeScanAnswer(query)
+
+ val df = query
+ assert(df.collect().length == 90)
+ val scans = deltaNativeScans(df)
+ assert(scans.size == 1)
+ val metrics = scans.head.metrics
+ val pagesPruned = metrics.get("page_index_rows_pruned").map(_.value).getOrElse(0L)
+ assert(
+ pagesPruned > 0,
+ s"expected page-index pruning to compose with DVs; metrics: ${metrics.map { case (k, v) =>
+ s"$k=${v.value}"
+ }}")
+ }
+ }
+
+ test("deletion vectors: aggregation over DV table") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path, rows = 10000)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 7 = 0")
+
+ val df = spark.read.format("delta").load(path).groupBy(col("id") % 13).count()
+ checkDeltaNativeScanAnswer(df)
+ }
+ }
+
+ test("deletion vectors: partitioned table reads natively with correct results") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ spark
+ .range(0, 1000)
+ .selectExpr("id", "id % 5 as p", "id * 2 as v")
+ .write
+ .format("delta")
+ .partitionBy("p")
+ .save(path)
+ spark.sql(
+ s"ALTER TABLE delta.`$path` SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true')")
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 3 = 0")
+
+ val df = spark.read.format("delta").load(path).filter(col("p") === 2)
+ checkDeltaNativeScanAnswer(df)
+ assert(df.count() == (0L until 1000L).count(i => i % 5 == 2 && i % 3 != 0))
+
+ val all = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(all)
+ assert(all.count() == (0L until 1000L).count(_ % 3 != 0))
+ }
+ }
+
+ test("deletion vectors: combined with constant metadata columns") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id < 250")
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .selectExpr("id", "v", "_metadata.file_name as fn")
+ checkSparkAnswer(df.selectExpr("id", "v", "length(fn) > 0"))
+ // Whether this claims or declines, results must match; if it claimed, verify the
+ // native node is present so the combination is actually exercised when supported.
+ val rows = df.collect()
+ assert(rows.length == 750)
+ assert(rows.forall(_.getString(2).nonEmpty))
+ }
+ }
+
+ test(
+ "deletion vectors: constant-metadata field names are deduplicated against the physical " +
+ "data and partition schemas") {
+ // End-to-end coverage is not possible here: selecting any `_metadata.*` field in the DV
+ // shape always declines today for an unrelated, pre-existing reason -- Spark reuses the
+ // scan's own row-index bookkeeping attribute as `_metadata.row_index`'s source, and
+ // `DeltaScanSupport.rowIndexUnusedAbove` conservatively treats extracting ANY `_metadata`
+ // field as making that attribute live (see "combined with constant metadata columns"
+ // above, which hedges its assertions for the same reason). That decline fires before
+ // `buildDvScanCommon` ever runs, regardless of collision, so it cannot exercise the fix.
+ // Test the builder's dedup logic directly instead, the same way `storeUris` and
+ // `mergedObjectStoreOptions` are unit-tested without a live scan.
+ val physicalDataSchema =
+ StructType(Seq(StructField("_comet_metadata_file_path", ByteType)))
+ val physicalPartitionSchema =
+ StructType(Seq(StructField("_comet_metadata_file_size", LongType)))
+ val fileConstantMetadataColumns = Seq(
+ AttributeReference("file_path", StringType, nullable = false)(),
+ AttributeReference("file_size", LongType, nullable = false)())
+
+ val constantMetadataFields = CometNativeScan.uniqueConstantMetadataFields(
+ fileConstantMetadataColumns,
+ physicalDataSchema.fields.map(_.name).toSet ++ physicalPartitionSchema.fields
+ .map(_.name)
+ .toSet)
+ assert(
+ constantMetadataFields.map(_.name) == Seq(
+ "_comet_metadata_file_path_",
+ "_comet_metadata_file_size_"),
+ "expected both constant-metadata names to be uniquified on collision, got " +
+ s"${constantMetadataFields.map(_.name)}")
+
+ // The DV builder must feed these already-unique names into allocateUniqueInternalFields's
+ // reserved set so the internal-column suffix chain stays consistent with them.
+ val requiredSchema = StructType(
+ Seq(
+ StructField("id", LongType),
+ StructField(CometDeltaNativeScan.IsRowDeletedColumn, ByteType),
+ StructField(CometDeltaNativeScan.RowIndexColumn, LongType)))
+ val internalFields = CometDeltaNativeScan.allocateUniqueInternalFields(
+ requiredSchema,
+ physicalDataSchema,
+ physicalPartitionSchema,
+ constantMetadataFields)
+
+ val allNames = physicalDataSchema.fields.map(_.name) ++
+ physicalPartitionSchema.fields.map(_.name) ++
+ constantMetadataFields.map(_.name) ++
+ internalFields.map(_.name)
+ assert(allNames.distinct.length == allNames.length, s"expected all names distinct: $allNames")
+ }
+
+ test(
+ "non-DV shape: user column named like the synthetic constant-metadata slot keeps its " +
+ "own values") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ val collidingName = "_comet_metadata_file_path"
+ spark
+ .range(0, 100)
+ .selectExpr("id", s"cast(7 as tinyint) as `$collidingName`")
+ .write
+ .format("delta")
+ .save(path)
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .selectExpr("id", s"`$collidingName`", "_metadata.file_path as fp")
+ checkDeltaNativeScanAnswer(df)
+ val rows = df.collect()
+ val survivingValues = rows.map(_.getAs[Byte](collidingName)).distinct
+ assert(
+ survivingValues.sameElements(Array(7.toByte)),
+ "expected the user column's own value (7) to survive the constant-metadata " +
+ s"collision, got ${survivingValues.toSeq}")
+ assert(
+ rows.forall(_.getString(2).nonEmpty),
+ "expected _metadata.file_path to still report a real path")
+ }
+ }
+
+ test("deletion vectors: special characters in table path") {
+ withTempDir { base =>
+ val dir = new java.io.File(base, "s p a r k %dv% test")
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ val df = spark.read.format("delta").load(path)
+ checkDeltaNativeScanAnswer(df)
+ assert(df.count() == 500)
+ }
+ }
+
+ test("deletion vectors: decline when row_index is consumed via multi-hop aliases") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .selectExpr("id", "_metadata.row_index as ri")
+ .selectExpr("id", "ri + 1 as ri2")
+ .filter(col("ri2") > 10)
+ checkSparkAnswer(df)
+ assert(deltaNativeScans(df).isEmpty, "derived row_index consumption must decline")
+ }
+ }
+
+ test("deletion vectors: decline when row_index feeds a non-Project operator") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .groupBy(col("_metadata.row_index") % 7)
+ .count()
+ checkSparkAnswer(df)
+ assert(deltaNativeScans(df).isEmpty, "aggregate over row_index must decline")
+ }
+ }
+
+ test("deletion vectors: decline when _metadata.row_index is referenced above the scan") {
+ withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ createDvTable(path)
+ spark.sql(s"DELETE FROM delta.`$path` WHERE id % 2 = 0")
+
+ val df = spark.read
+ .format("delta")
+ .load(path)
+ .selectExpr("id", "_metadata.row_index as ri")
+ checkSparkAnswer(df)
+ assert(
+ deltaNativeScans(df).isEmpty,
+ "plans consuming a real row_index must fall back to Spark")
+ }
+ }
+
+ /**
+ * Every [[SparkPlan]] executed during `body`, captured via a [[QueryExecutionListener]] rather
+ * than a returned `DataFrame`'s own plan: a `DataFrameWriter` action such as `.write.parquet`
+ * has no result `Dataset` to call `.queryExecution` on, so the write's physical plan -- the one
+ * `DeltaScanSupport.declineReason` actually saw -- is only observable this way.
+ */
+ private def capturePlansDuring(body: => Unit): Seq[SparkPlan] = {
Review Comment:
This test failed in one of my two full local runs on Spark 3.5, with an empty reason list. It passes on its own. `QueryExecutionListener` callbacks arrive on the listener bus asynchronously, and this helper unregisters as soon as `body` returns. So the write's plan can arrive after we've stopped listening. When that happens `nativeScans.isEmpty` passes vacuously and only the reason check notices. A 300 ms sleep in `onSuccess` makes it fail every time. Adding `CometListenerBusUtils.waitUntilEmpty(spark.sparkContext)` after `body` makes it pass again with the sleep still there, which is what `CometIcebergTestBase.capturePlans` does.
Could both copies of this helper drain the bus, this one and the one in `CometDeltaDmlReproSuite`? The comment on `collectTaskInputMetrics` says the suite can't reach `waitUntilEmpty`, but `org.apache.spark.CometListenerBusUtils` is in the spark test-jar this module already depends on. `delta_3_5` runs in the merge queue for almost any change under `native/` or `spark/src/main`, so a flake here would evict other people's PRs.
##########
pom.xml:
##########
@@ -45,11 +45,10 @@ under the License.
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>17</java.version>
<!--
- Default Delta dep version (only consumed when `-Pcontrib-delta` is also
- active). Spark profiles override this: spark-3.5 -> 3.3.2, spark-4.1 ->
- 4.1.0. The top-level default lets Maven invocations that don't activate a
- Spark profile (e.g. `mvn -Pcontrib-delta spotless:apply`) resolve the
- property without an error.
+ Default Delta dep version, read by both Delta contribs (`-Pcontrib-delta`
Review Comment:
Following up on my earlier `delta.version` comment. This comment says the property is read by both contribs and that every Spark profile overrides it. But `spark/pom.xml` overrides it again in its own Spark profiles for `-Pcontrib-delta`, so the two modules resolve different versions on the same profile. `help:evaluate` gives 4.0.0 for `spark` and 4.0.1 for `contrib/delta-spark` on `spark-4.0`, and 4.1.0 against 4.3.1 on `spark-4.1`. Could this module use its own property, something like `delta.spark.version`, so bumping one pairing can't quietly leave the other behind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4084980019
##########
pom.xml:
##########
@@ -45,11 +45,10 @@ under the License.
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>17</java.version>
<!--
- Default Delta dep version (only consumed when `-Pcontrib-delta` is also
- active). Spark profiles override this: spark-3.5 -> 3.3.2, spark-4.1 ->
- 4.1.0. The top-level default lets Maven invocations that don't activate a
- Spark profile (e.g. `mvn -Pcontrib-delta spotless:apply`) resolve the
- property without an error.
+ Default Delta dep version, read by both Delta contribs (`-Pcontrib-delta`
Review Comment:
You are right, and the comment was wrong. `spark/pom.xml` sets `delta.version` in its own Spark profiles for the in-tree contrib, and a child's profile property wins over the parent's, so the spark module kept those values while `contrib/delta-spark` read the overrides this branch added to the root profiles. The root comment claimed one property served both.
`contrib/delta-spark` now reads its own `delta.spark.version`. The root pom defines it once per Spark profile with the values the module already resolved (2.4.0 on 3.4, 3.3.2 on 3.5, 4.0.1 on 4.0, 4.3.1 on 4.1 and 4.2) and a top-level default of 4.3.1 for invocations with no Spark profile. `delta.version` is left to the in-tree contrib and now resolves to exactly what main gives the spark module on every profile, which also undoes a drift this branch had introduced on spark-4.2. Both comments say which module reads which property. Checked with `help:evaluate` on both modules across all five profiles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4081118624
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,2096 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::mem::size_of;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::{ParquetAccessPlan, RowGroupAccess};
+use datafusion::execution::memory_pool::{MemoryConsumer, MemoryReservation};
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::path::Path;
+use object_store::{ObjectStore, ObjectStoreExt};
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::{PageIndexPolicy, ParquetMetaData};
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ // A corrupt footer can report a negative row count. `num_rows as u64` would otherwise
+ // wrap it into a huge positive value, silently corrupting every row-group boundary
+ // computed from `group_start`/`group_end` below (and therefore which deleted row indexes
+ // land in which row group) instead of failing loudly.
+ if num_rows < 0 {
+ return Err(GeneralError(format!(
+ "Parquet footer reports a negative row count ({num_rows}) for row group {idx}"
+ )));
+ }
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
+ }
+ cursor = row + 1;
+ }
+ if deleted_in_group == num_rows && num_rows > 0 {
+ plan.skip(idx);
+ } else if deleted_in_group > 0 {
+ if group_end > cursor {
+ selectors.push(RowSelector::select((group_end - cursor) as usize));
+ }
+ plan.scan_selection(idx, RowSelection::from(selectors));
+ }
+ group_start = group_end;
+ }
+ // A deleted index beyond the file's total row count means the DV does not
+ // belong to this file (stale or corrupted metadata); silently dropping it
+ // would under-apply deletions.
+ if let Some(&row) = deleted_iter.peek() {
+ return Err(GeneralError(format!(
+ "Deletion vector marks row {row} but the file only has {group_start} rows"
+ )));
+ }
+ Ok(plan)
+}
+
+/// Verify a decoded deletion vector's row count matches the descriptor's
+/// declared `cardinality`, mirroring Delta's JVM reader
+/// (`StoredBitmap.validateCardinality`). The CRC and framing checks catch
+/// corruption but not a stale, otherwise well-formed bitmap whose row count
+/// no longer matches the descriptor -- that would silently under- or
+/// over-delete rows.
+fn validate_cardinality(
+ file_path: &str,
+ expected: i64,
+ deleted: &RoaringTreemap,
+) -> Result<(), ExecutionError> {
+ let actual = deleted.len();
+ if actual != expected as u64 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has cardinality mismatch: descriptor says {expected}, decoded bitmap has {actual} deleted rows"
+ )));
+ }
+ Ok(())
+}
+
+/// One data file plus everything needed to apply its deletion vector. The
+/// file's size comes from `file.object_meta.size` (built by the planner from
+/// the proto's `file_size`).
+///
+/// `data_store` and `dv_store` are resolved by the caller *before* entering
+/// the async `attach_access_plans` runtime (see its doc comment): building an
+/// object store is sync I/O that, for a cold S3 authority, internally issues
+/// its own `Handle::block_on` calls, which panics if nested inside another
+/// `block_on`. Resolving up front means this module never constructs a
+/// store itself.
+pub struct DvScanFile {
+ pub file: PartitionedFile,
+ /// Full URL of the data file (proto `file_path`).
+ pub file_path: String,
+ pub dv: Option<DeltaSparkDvDescriptor>,
+ /// Object store for `file_path`, pre-resolved by the caller. Only read
+ /// when `dv` is `Some` (files without a deletion vector never open their
+ /// footer here), but every file carries one so the struct's shape
+ /// doesn't depend on whether a deletion vector is present.
+ pub data_store: Arc<dyn ObjectStore>,
+ /// Store and within-store path for an on-disk deletion vector's absolute
+ /// path, pre-resolved by the caller. `None` when the file has no
+ /// deletion vector or the deletion vector is stored inline.
+ pub dv_store: Option<(Arc<dyn ObjectStore>, Path)>,
+}
+
+/// Execution-memory-pool reservation covering one file's expanded DV row selectors across
+/// their *entire* lifetime attached to a scan -- from `build_access_plan`'s construction
+/// through DataFusion 54.1's reader normalizing the attached [`ParquetAccessPlan`]
+/// (`create_initial_plan`'s deep clone plus `into_overall_row_selection`'s combined
+/// `RowSelection`; see [`reader_peak_bytes`]) -- attached to the file's [`PartitionedFile`]
+/// extensions alongside its [`ParquetAccessPlan`]. The reservation's lifetime is tied to the
+/// `PartitionedFile` it is attached to, so it is released back to the pool exactly when the
+/// plan is dropped (query completion or an early-terminated scan), never held open longer.
+/// Newtype-wrapped so it occupies its own slot in the multi-slot, type-keyed `extensions` map
+/// (`datafusion_common::extensions::Extensions`) alongside the plan, rather than a bare
+/// `MemoryReservation` colliding with one some other extension might attach.
+pub struct DvAccessPlanReservation(pub MemoryReservation);
+
+/// Total number of [`RowSelector`]s materialized across `plan`'s per-row-group
+/// selections (`RowGroupAccess::Selection`); `Scan`/`Skip` row groups
+/// contribute none. An alternating deleted/retained bitmap produces one
+/// non-coalescing selector per row (see [`reader_peak_bytes`]'s doc comment
+/// for the worst-case accounting), so this count -- not the deletion
+/// vector's cardinality -- is the thing that must be bounded and reserved
+/// against the execution memory pool.
+fn total_selectors(plan: &ParquetAccessPlan) -> usize {
+ plan.inner()
+ .iter()
+ .map(|access| match access {
+ RowGroupAccess::Selection(selection) => selection.iter().count(),
+ _ => 0,
+ })
+ .sum()
+}
+
+/// Multiplier bounding the peak allocation live *during construction* of one
+/// file's [`RowSelection`]s, relative to the conservative selector-count
+/// bound `S = 2 * cardinality + num_row_groups` (one non-coalescing selector
+/// per deleted row in the worst-case alternating pattern, doubled, plus up to
+/// one extra boundary selector per row group). Split `S` into `r`, the
+/// selectors already retained from row groups `build_access_plan` has
+/// finished, and `c`, the selectors accumulated so far in the current row
+/// group's source `Vec`; `r` and `c` partition the selectors counted toward
+/// `S`, so `r + c <= S` always. While the current group is being built, the
+/// `Vec`'s doubling growth strategy can leave its backing allocation at up to
+/// `2 * c` (the next power-of-two capacity above `c`). Once the group
+/// finishes, `RowSelection::from(Vec)` (parquet's `FromIterator` impl,
+/// `with_capacity` + copy) builds a second, separate `Vec` of size `c` from
+/// that source while the source is still alive, so at the moment the copy
+/// begins, the retained selectors, the current group's doubled source `Vec`,
+/// and the copy are all live simultaneously: `r + 2c + c = r + 3c`. Since
+/// `r >= 0`, `r + 3c <= 3r + 3c = 3(r + c) <= 3S`. 3x covers that peak.
+const CONSTRUCTION_PEAK_FACTOR: usize = 3;
+
+/// Upper bound on how much larger a `Vec`'s backing allocation can be than its element count
+/// after being built by repeated pushes: `std`'s doubling growth strategy never leaves a `Vec`
+/// of `n` elements with a backing allocation larger than the next power of two above `n`, which
+/// is at most `2 * n` for any `n >= 1`.
+const VEC_GROWTH_CAPACITY_FACTOR: usize = 2;
+
+/// `RawVec`'s minimum non-zero capacity for element sizes `<= 1024` bytes ([`RowSelector`] is
+/// 16 bytes on 64-bit platforms: a `usize` row count plus a padded `bool`). Applied once per
+/// row group (or per contiguous run of row groups) a fresh `from_fn`/`FlatMap`-driven `Vec`
+/// gets built for (see [`reader_peak_bytes`]), so even a group or run whose true selector count
+/// is tiny still pays this floor.
+const MIN_VEC_CAPACITY_SELECTORS: usize = 4;
+
+/// Conservative upper bound, in bytes, on the peak allocation live while DataFusion 54.1's
+/// reader normalizes one file's attached [`ParquetAccessPlan`] -- the allocation this module's
+/// steady-state reservation must cover, not merely the plan's own retained selector bytes.
+/// THREE allocations can be live simultaneously by the time `into_overall_row_selection`
+/// returns, not two -- the clone is only exact when page-index pruning never touches it:
+///
+/// 1. **Attached original** (`selectors`, exact): `create_initial_plan` deep-clones the
+/// attached plan while the original remains reachable from the file's `extensions` until
+/// the scan consumes it. The ORIGINAL's own selector `Vec`s are exact -- a coalesced
+/// [`RowSelection`] built via `RowSelection::from(Vec<RowSelector>)` (what
+/// `build_access_plan` uses) has no excess capacity, because that conversion is a plain
+/// `with_capacity(len)` copy, not a `size_hint`-blind fold.
+/// 2. **The clone, possibly capacity-inflated** (`<= VEC_GROWTH_CAPACITY_FACTOR * selectors +
+/// MIN_VEC_CAPACITY_SELECTORS * num_row_groups`): if page-index pruning fires
+/// (`PagePruningAccessPlanFilter`; `access_plan.rs`'s `scan_selection` on a row group that
+/// already carries a `RowGroupAccess::Selection` calls `existing.intersection(&page_derived)`
+/// -- `RowSelection::intersection` -> `intersect_row_selections`), it replaces the CLONE's
+/// per-row-group selection with that intersection's output. `intersect_row_selections` is
+/// ANOTHER `from_fn` generator with `size_hint() == (0, None)`, so each intersected row
+/// group's backing `Vec` starts at `with_capacity(0)` and doubles as it grows, independent
+/// of whatever capacity the pre-intersection selection had. This inflated clone is still
+/// live when `into_overall_row_selection` later moves its buffer. Term 1's exactness
+/// guarantee holds for the ORIGINAL always, and for the clone only when page-index pruning
+/// never fires against it -- once it does, the clone must be charged at the SAME
+/// growth-capped bound as a fresh combined-selection `Vec` (term 3), summed once per row
+/// group rather than once per run, since each row group's `Selection` is intersected
+/// independently.
+/// 3. **Per-run combined-selection allocation** (`<= VEC_GROWTH_CAPACITY_FACTOR * (selectors +
+/// num_row_groups) + MIN_VEC_CAPACITY_SELECTORS * num_row_groups`): `into_overall_row_selection`
+/// collects each contiguous run of row groups' selectors into a *new* `RowSelection` via a
+/// `FlatMap` whose `size_hint().0 == 0`, so that run's `Vec` starts at `with_capacity(0)`
+/// and doubles as it grows -- capping its backing allocation at
+/// `max(MIN_VEC_CAPACITY_SELECTORS, next_power_of_two(len))`, which is at most
+/// `MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR * len` for a run of `len`
+/// selectors. `len` is at most that run's share of `selectors` plus one boundary selector
+/// per `RowGroupAccess::Scan` row group in the run (`Scan` always contributes exactly one
+/// `RowSelector::select(num_rows)`; see `access_plan.rs`'s `into_overall_row_selection`).
+/// Summing across at most `num_row_groups` runs (each spans >= 1 row group) bounds the total
+/// at `VEC_GROWTH_CAPACITY_FACTOR * selectors + (MIN_VEC_CAPACITY_SELECTORS +
+/// VEC_GROWTH_CAPACITY_FACTOR) * num_row_groups`.
+///
+/// Summing all three terms and converting to bytes: `((1 + 2 * VEC_GROWTH_CAPACITY_FACTOR) *
+/// selectors + (2 * MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR) * num_row_groups)
+/// * size_of::<RowSelector>()` -- with the constants above, `(5 * selectors + 10 *
+/// num_row_groups) * size_of::<RowSelector>()`. Checked against two measured worst cases:
+///
+/// - No page-index pruning (the original P2 report; term 2 stays exact): one 2,000,000-row
+/// group, 1,000,000 alternating deletions, `selectors = 2,000,000`. Measured allocator peak
+/// 97,554,457 B; the byte-for-byte accounting for the attached original plus the (here,
+/// exact) clone plus the inflated combined selection explains 97,554,432 B of that, a 25 B
+/// residue we did not attribute. This bound gives 160,000,160 B -- much looser here because
+/// it must also cover the next case, where the clone is NOT exact.
+/// - Page-index pruning fires against the clone: one 1,048,577-row group, `selectors =
+/// 1,048,577`. Measured peak 83,886,096 B; this bound gives 83,886,320 B (a 224 B, <1%
+/// margin -- deliberately tight, since this is the case that drives the bound).
+///
+/// Uses checked arithmetic throughout: a selector or row-group count large enough to overflow
+/// `usize` indicates a corrupted or malicious input, reported as a clean error rather than
+/// panicking.
+fn reader_peak_bytes(selectors: usize, num_row_groups: usize) -> Result<usize, ExecutionError> {
+ let overflow = || {
+ GeneralError(format!(
+ "Deletion vector reader-peak bound overflowed for {selectors} selectors and \
+ {num_row_groups} row groups"
+ ))
+ };
+ // Term 1: the attached original -- exact, untouched by page-index pruning (only the clone
+ // is ever intersected; see the doc comment above).
+ let attached_term = selectors;
+ // Term 2: the clone, bounded as if page-index pruning DID fire against every row group
+ // (safe even when it doesn't: term 2's bound is always >= `selectors`, so it never
+ // undershoots the exact case either).
+ let clone_growth = selectors
+ .checked_mul(VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let clone_floor = num_row_groups
+ .checked_mul(MIN_VEC_CAPACITY_SELECTORS)
+ .ok_or_else(overflow)?;
+ let clone_term = clone_growth.checked_add(clone_floor).ok_or_else(overflow)?;
+ // Term 3: into_overall_row_selection's per-run combined-selection allocation.
+ let combined_growth = selectors
+ .checked_mul(VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let combined_floor = num_row_groups
+ .checked_mul(MIN_VEC_CAPACITY_SELECTORS + VEC_GROWTH_CAPACITY_FACTOR)
+ .ok_or_else(overflow)?;
+ let combined_term = combined_growth
+ .checked_add(combined_floor)
+ .ok_or_else(overflow)?;
+
+ let selector_bound = attached_term
+ .checked_add(clone_term)
+ .and_then(|sum| sum.checked_add(combined_term))
+ .ok_or_else(overflow)?;
+ selector_bound
+ .checked_mul(size_of::<RowSelector>())
+ .ok_or_else(overflow)
+}
+
+/// Upper bound, in [`RowSelector`]s, on how many extra selectors the parquet reader's
+/// page-index pruning can add on top of the deletion vector's own selection when normalizing
+/// one file, from that file's already-fetched [`ParquetMetaData`].
+///
+/// `intersect_row_selections` (parquet's `selection.rs`), which combines a page-pruning
+/// selection with the deletion vector's selection, is a `from_fn` generator whose
+/// `size_hint()` is `(0, None)`: for inputs of length `a` and `b`, its output can have up to
+/// `a + b` selectors -- longer than either input. Bounding the page-pruning side of that sum
+/// requires knowing how many selectors a page-index-derived selection could produce: at most
+/// two per data page (one skip, one select, in the worst case of alternating page-level
+/// pruning decisions), summed over every column of every row group.
+///
+/// Returns `0` when `metadata` carries no offset index (`metadata.offset_index()` is `None`).
+/// This is provably safe, not merely a convenient default: page-index pruning cannot produce a
+/// page-level selection without the offset index to locate pages by, so there are no
+/// page-pruning selectors to bound. The offset index is fetched with
+/// `PageIndexPolicy::Optional` from the same `FileMetadataCache` entry the scan's reader later
+/// reopens (see [`attach_access_plan`]'s footer-fetch comment), so this function observes
+/// exactly what the reader will see.
+///
+/// Uses checked arithmetic throughout for the same reason as [`admission_bound_bytes`].
+fn page_selection_bound_selectors(metadata: &ParquetMetaData) -> Result<usize, ExecutionError> {
+ let Some(offset_index) = metadata.offset_index() else {
+ return Ok(0);
+ };
+ let overflow = || {
+ GeneralError(
+ "Deletion vector page-selection bound overflowed while summing offset-index page \
+ locations"
+ .to_string(),
+ )
+ };
+ let mut total_page_locations = 0usize;
+ for row_group in offset_index {
+ for column in row_group {
+ total_page_locations = total_page_locations
+ .checked_add(column.page_locations().len())
+ .ok_or_else(overflow)?;
+ }
+ }
+ total_page_locations.checked_mul(2).ok_or_else(overflow)
+}
+
+/// Execution-memory-pool admission bound, in bytes, for one file's deletion-vector access
+/// plan -- reserved *before* calling `build_access_plan` (see [`attach_access_plan`]'s
+/// pre-reserve call site) to cover the larger of two peaks live at different points in the
+/// plan's lifetime. In practice the reader-normalization peak below dominates the construction
+/// peak unconditionally for any non-trivial input (`reader_peak_bytes(S, G) = (5S + 10G) *
+/// size_of::<RowSelector>()` always exceeds `CONSTRUCTION_PEAK_FACTOR * S *
+/// size_of::<RowSelector>() = 3S * size_of::<RowSelector>()` once `S >= 1`, since the `5S` term
+/// alone already exceeds `3S`); the construction term is retained as a documented floor rather
+/// than dropped, since it is cheap to compute and keeps this bound correct even if the reader's
+/// growth factors ever shrink below construction's.
+///
+/// - **Construction peak** (`CONSTRUCTION_PEAK_FACTOR * S`, see that constant's doc comment):
+/// live while `build_access_plan` builds the plan's `RowSelection`s. Construction's
+/// transient allocations fully unwind before `build_access_plan` returns, so this peak never
+/// overlaps the reader-normalization peak below.
+/// - **Reader-normalization peak** (`reader_peak_bytes(S + page_bound_selectors,
+/// num_row_groups)`, see that function): live later, once DataFusion's reader normalizes the
+/// attached plan. `S = 2 * cardinality + num_row_groups` is the same conservative bound on
+/// the plan's final retained selector count used for the construction peak -- it provably
+/// bounds `R = total_selectors(&plan)` (`R <= S`, from `build_access_plan`'s
+/// one-non-coalescing-selector-per-deleted-row worst case plus one boundary selector per row
+/// group), so `S + page_bound_selectors` bounds `R` after page-index inflation the same way
+/// `S` bounds `R` before it.
+///
+/// These two peaks never overlap in time, so `max` -- not `sum` -- is the correct combinator:
+/// reserving their sum would over-reserve for no safety benefit.
+///
+/// Deliberately not clamped by the file's total row count here, unlike the reader-peak target
+/// `attach_access_plan` resizes down to after construction (see that call site): `S`'s
+/// `+ num_row_groups` boundary term is a worst-case padding margin that can legitimately exceed
+/// the total row count for a small, heavily-deleted file, and admission sizing has no actual
+/// retained-selector count yet to clamp against -- only after construction, once `R` is known,
+/// is clamping to the total row count both meaningful and strictly tighter. Leaving this bound
+/// unclamped only ever makes admission more conservative, never less safe.
+///
+/// Uses checked arithmetic throughout: a cardinality, row-group count, or page bound large
+/// enough to overflow `usize` while computing this bound indicates a corrupted or malicious
+/// descriptor, reported as a clean error rather than panicking.
+fn admission_bound_bytes(
+ cardinality: i64,
+ num_row_groups: usize,
+ page_bound_selectors: usize,
+) -> Result<usize, ExecutionError> {
+ let overflow = || {
+ GeneralError(format!(
+ "Deletion vector admission bound overflowed for cardinality {cardinality}, \
+ {num_row_groups} row groups, and page bound {page_bound_selectors} selectors"
+ ))
+ };
+ let cardinality_usize = usize::try_from(cardinality).map_err(|_| overflow())?;
+ // S: the conservative bound on the plan's final *retained* selector count (what
+ // `total_selectors(&plan)` cannot exceed) -- unchanged from the pre-existing
+ // construction-only bound this function replaces.
+ let s = cardinality_usize
+ .checked_mul(2)
+ .and_then(|doubled| doubled.checked_add(num_row_groups))
+ .ok_or_else(overflow)?;
+
+ let construction_bytes = s
+ .checked_mul(size_of::<RowSelector>())
+ .and_then(|bytes| bytes.checked_mul(CONSTRUCTION_PEAK_FACTOR))
+ .ok_or_else(overflow)?;
+
+ let s_plus_page = s.checked_add(page_bound_selectors).ok_or_else(overflow)?;
+ let reader_bytes = reader_peak_bytes(s_plus_page, num_row_groups)?;
+
+ Ok(construction_bytes.max(reader_bytes))
+}
+
+/// Upper bound on concurrent DV-blob and footer fetches per partition. Both
+/// are small ranged reads, so a modest fan-out hides object-store latency
+/// without flooding the store client.
+const DV_FETCH_CONCURRENCY: usize = 8;
+
+/// Called via `block_on` at plan-creation time on the executor task: DV blobs
+/// are small ranged reads and footers are needed to learn row-group
+/// boundaries. Files are fetched concurrently (bounded by
+/// [`DV_FETCH_CONCURRENCY`]) with input order preserved. Footer fetches go
+/// through the scan's shared FileMetadataCache, so the scan's subsequent open
+/// of the same file is served from cache. That reuse relies on each input
+/// [`PartitionedFile`] being returned as-is (only `with_extension` applied),
+/// never rebuilt: the cache entry is keyed by this exact `object_meta` and the
+/// scan later looks it up through the same struct.
+///
+/// Deliberately takes no object-store options map and imports no
+/// store-construction helper: every [`DvScanFile`] arrives with its stores
+/// already resolved by the caller (see its doc comment), so this async path
+/// structurally cannot build an object store -- only `runtime_env` is still
+/// threaded through, for the shared `FileMetadataCache` and (per file) the
+/// execution `MemoryPool` each expanded access plan's row selectors are
+/// reserved against -- see [`DvAccessPlanReservation`].
+pub async fn attach_access_plans(
+ runtime_env: Arc<RuntimeEnv>,
+ files: Vec<DvScanFile>,
+) -> Result<Vec<PartitionedFile>, ExecutionError> {
+ futures::stream::iter(files)
+ .map(|scan_file| attach_access_plan(Arc::clone(&runtime_env), scan_file))
+ .buffered(DV_FETCH_CONCURRENCY)
+ .try_collect()
+ .await
+}
+
+/// Resolve one file's deletion vector into an attached [`ParquetAccessPlan`];
+/// files without a DV pass through untouched.
+async fn attach_access_plan(
+ runtime_env: Arc<RuntimeEnv>,
+ scan_file: DvScanFile,
+) -> Result<PartitionedFile, ExecutionError> {
+ let DvScanFile {
+ file,
+ file_path,
+ dv,
+ data_store,
+ dv_store,
+ } = scan_file;
+ let dv = match dv {
+ Some(dv) => dv,
+ None => return Ok(file),
+ };
+ // Delta's canonical `DeletionVectorDescriptor.EMPTY`: inline storage, empty
+ // payload, size 0, cardinality 0. Spark's reader returns all rows for it;
+ // decoding would fail (the empty payload is too short for a magic
+ // number), so pass the file through unchanged before attempting to read it.
+ if dv.cardinality == 0 && dv.size_in_bytes == 0 {
+ return Ok(file);
+ }
+ if dv.size_in_bytes < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative size {}",
+ dv.size_in_bytes
+ )));
+ }
+ if dv.cardinality < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative cardinality {}",
+ dv.cardinality
+ )));
+ }
+
+ let data: Vec<u8> = if let Some(inline) = dv.inline_data {
+ inline
+ } else if let Some(dv_path) = &dv.absolute_path {
+ let offset = dv
+ .offset
+ .ok_or_else(|| GeneralError("On-disk deletion vector missing offset".into()))?;
+ if offset < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative offset {offset}"
+ )));
+ }
+ let offset = offset as u64;
+ // [i32 BE size][data: size_in_bytes][i32 BE crc]
+ let framed_len = 4 + dv.size_in_bytes as u64 + 4;
+ let (store, dv_store_path) = dv_store.ok_or_else(|| {
+ GeneralError(format!(
+ "Deletion vector for {file_path} has an absolute path but no pre-resolved object store"
+ ))
+ })?;
+ let blob = store
+ .get_range(&dv_store_path, offset..offset + framed_len)
+ .await
+ .map_err(|e| GeneralError(format!("Failed to read deletion vector {dv_path}: {e}")))?;
+ unframe_dv_blob(&blob, dv.size_in_bytes as usize)?.to_vec()
+ } else {
+ return Err(GeneralError(
+ "Deletion vector descriptor has neither inline data nor a path".into(),
+ ));
+ };
+ let deleted = deserialize_dv_bitmap(&data)
+ .map_err(|e| GeneralError(format!("Invalid deletion vector for {file_path}: {e}")))?;
+ validate_cardinality(&file_path, dv.cardinality, &deleted)?;
+
+ // Row-group boundaries come from the data file's footer, fetched through the scan's
+ // shared FileMetadataCache with the page index loaded eagerly and the scan's metadata
+ // size hint (mirroring EagerPageIndexReaderFactory): the one fetch here also serves the
+ // subsequent data-file open, so DV files pay no extra footer round-trip. Keyed by
+ // `file.object_meta`, the exact ObjectMeta the scan's reader factory will look up.
+ let metadata_cache = runtime_env.cache_manager.get_file_metadata_cache();
+ let metadata = DFParquetMetadata::new(data_store.as_ref(), &file.object_meta)
Review Comment:
> Could DV preparation use a shared metadata-reading boundary connected to the final scan's reader configuration and metrics?
Tracked in #5948. Preparation keeps its direct footer fetch here; the issue asks for the cold-cache and warm-cache accounting tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4205091948
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so its rewrite (via
+ * [[CometScanWithPlanData]]) survives plan copies -- a transient field would be dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning (the CometIcebergNativeScanExec lesson).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node
+ // automatically gets a helper consistent with ITS runtimeFilters, avoiding the #3510 class of
+ // bug where a stored helper field desyncs from rewritten filters. Costs one extra file listing
+ // per executed instance; correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below, and AQE's UI
+ // plan-walk calls `.metrics` on every node MID-PLANNING, including while a DPP subquery is
+ // still an adaptive placeholder or a partition filter holds an unresolved ScalarSubquery (see
+ // `hasUnevaluableSubqueryFilter` below). That's safe ONLY because constructing `scanHelper` is a
+ // cheap case-class build with no file listing, and core's `CometScanExec.metrics` touches only
+ // `wrapped.driverMetrics` (populated by Spark's own planning) plus a static metric-node
+ // constructor -- neither file listing nor subquery resolution. If core's `metrics` ever touches
+ // either, forcing `scanHelper` here would resurrect the AQE mid-planning crashes this invariant
+ // prevents.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element, so checking runtimeFilters
+ // here avoids constructing/forcing the derived scanHelper just to read partitioning. The
+ // InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep in sync. The
+ // ScalarSubquery case is probed rather than treated as permanently unevaluable: Spark exposes no
+ // public finished/updated flag on ExecSubqueryExpression, but `eval()` doubles as one -- it only
+ // reads the cached `result` behind a `require(updated, ...)` guard, while the subquery is
+ // actually run by `updateResult()` (invoked separately during prepare/AQE), never by `eval()`.
+ // Once resolved, outputPartitioning below reports the real perPartitionData.length instead of
+ // staying at zero -- a fused native parent's buildNativeContext requires that count to match.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case s: ScalarSubquery => !isScalarSubqueryResolved(s)
+ case _ => false
+ })
+
+ // `eval()` never triggers the subquery's execution: on a resolved subquery it is a pure cached
+ // read of `result` (verified against bytecode: `Predef.require(updated(), ...)` then a plain
+ // field read), so this probe is safe to call repeatedly, including from AQE's mid-planning plan
+ // walks. Pre-resolution, the ONLY throw is `require`'s `IllegalArgumentException`; catch exactly
+ // that, since anything else escaping is a genuine bug we must not mask as unpartitioned.
+ private def isScalarSubqueryResolved(s: ScalarSubquery): Boolean =
+ try {
+ s.eval()
+ true
+ } catch {
+ case _: IllegalArgumentException => false
+ }
+
+ private def isAdaptivePlaceholder(p: SparkPlan): Boolean = p match {
+ case ReusedSubqueryExec(inner) => isAdaptivePlaceholder(inner)
+ case _: CometSubqueryAdaptiveBroadcastExec => true
+ case _: SubqueryAdaptiveBroadcastExec => true
+ case _ => false
+ }
+
+ override lazy val outputOrdering: Seq[SortOrder] = originalPlan.outputOrdering
+
+ override def dynamicPruningFilters: Seq[Expression] = runtimeFilters
+
+ override def withDynamicPruningFilters(filters: Seq[Expression]): SparkPlan = {
+ // A real copy: runtimeFilters is a constructor field included in equality, so the copy
+ // survives enclosing-block rebuilds, and the derived scanHelper picks up the rewritten
+ // filters automatically.
+ copy(runtimeFilters = filters)
Review Comment:
[P2] [P2] Preserve live pruning filters when restoring the Spark scan
With native Delta scanning, AQE and dynamic partition pruning enabled, a broadcast join fails if transition reversion restores this scan. For example, enable `spark.comet.exec.transitionRevert.enabled=true`, set `spark.comet.exec.transitionRevert.maxTransitions=0`, and disable native projection with `spark.comet.exec.project.enabled=false`. This rewrite updates `runtimeFilters`, but leaves `originalPlan` holding its original `SubqueryAdaptiveBroadcastExec`. The class inherits `CometExec.sparkFallback`, which restores that frozen plan after AQE’s pruning rewrite has already run. Spark then throws `SubqueryAdaptiveBroadcastExec does not support the execute() code path` instead of returning the matching rows.
A source-extracted probe on a real partitioned Delta scan reproduced the exception. Stock Spark and a control restoring the live filters both returned 200 rows. Could this scan override `sparkFallback`, following `CometNativeScanExec`, to preserve the current partition filters and logical link? Please add a Delta regression covering AQE pruning together with transition reversion.
Evidence: Spark 3.5.9 / Delta 3.3.2 reproduction: write 400 fact rows with fact_id=i, fact_key=i%10 and fact_str='f'+i, partitioned by fact_key, plus a Parquet dimension containing dim_id=dim_key=0..9 and dim_str='d'+i. Execute `SELECT f.fact_id, f.fact_str, d.dim_str FROM f JOIN d ON f.fact_key=d.dim_key WHERE d.dim_id < 5`. Capture the initial and final AQE fact scans. Apply the exact current `withDynamicPruningFilters` and inherited `sparkFallback` implementations in reduced scan classes. Executing the restored scan through `ColumnarToRowExec` throws the unsupported-execution exception. Copying live partition filters into the original scan returns 200 rows, matching the stock query. Source routing confirms `RevertNativeForTransitionHeavyStages.revertToSpark` calls this inherited fallback after the AQE pruning rules. Official Spark 3.5.9, 4.0.4 and 4.1.3 sources all define the restored placeholder as unexecutable. Probe and output: `/tmp/5365-review-05cc-oct7/FallbackProbe.scala
` and `/tmp/5365-review-05cc-oct7/fallback-run.log`. This is a method-level reproduction using real Spark/Delta plans, not a full Comet/JNI run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4032432601
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so its rewrite (via
+ * [[CometScanWithPlanData]]) survives plan copies -- a transient field would be dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning (the CometIcebergNativeScanExec lesson).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node
+ // automatically gets a helper consistent with ITS runtimeFilters, avoiding the #3510 class of
+ // bug where a stored helper field desyncs from rewritten filters. Costs one extra file listing
+ // per executed instance; correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below, and AQE's UI
+ // plan-walk calls `.metrics` on every node MID-PLANNING, including while a DPP subquery is
+ // still an adaptive placeholder or a partition filter holds an unresolved ScalarSubquery (see
+ // `hasUnevaluableSubqueryFilter` below). That's safe ONLY because constructing `scanHelper` is a
+ // cheap case-class build with no file listing, and core's `CometScanExec.metrics` touches only
+ // `wrapped.driverMetrics` (populated by Spark's own planning) plus a static metric-node
+ // constructor -- neither file listing nor subquery resolution. If core's `metrics` ever touches
+ // either, forcing `scanHelper` here would resurrect the AQE mid-planning crashes this invariant
+ // prevents.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element, so checking runtimeFilters
+ // here avoids constructing/forcing the derived scanHelper just to read partitioning. The
+ // InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep in sync. The
+ // ScalarSubquery case is probed rather than treated as permanently unevaluable: Spark exposes no
+ // public finished/updated flag on ExecSubqueryExpression, but `eval()` doubles as one -- it only
+ // reads the cached `result` behind a `require(updated, ...)` guard, while the subquery is
+ // actually run by `updateResult()` (invoked separately during prepare/AQE), never by `eval()`.
+ // Once resolved, outputPartitioning below reports the real perPartitionData.length instead of
+ // staying at zero -- a fused native parent's buildNativeContext requires that count to match.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case s: ScalarSubquery => !isScalarSubqueryResolved(s)
+ case _ => false
+ })
+
+ // `eval()` never triggers the subquery's execution: on a resolved subquery it is a pure cached
+ // read of `result` (verified against bytecode: `Predef.require(updated(), ...)` then a plain
+ // field read), so this probe is safe to call repeatedly, including from AQE's mid-planning plan
+ // walks. Pre-resolution, the ONLY throw is `require`'s `IllegalArgumentException`; catch exactly
+ // that, since anything else escaping is a genuine bug we must not mask as unpartitioned.
+ private def isScalarSubqueryResolved(s: ScalarSubquery): Boolean =
+ try {
+ s.eval()
+ true
+ } catch {
+ case _: IllegalArgumentException => false
+ }
+
+ private def isAdaptivePlaceholder(p: SparkPlan): Boolean = p match {
+ case ReusedSubqueryExec(inner) => isAdaptivePlaceholder(inner)
+ case _: CometSubqueryAdaptiveBroadcastExec => true
+ case _: SubqueryAdaptiveBroadcastExec => true
+ case _ => false
+ }
+
+ override lazy val outputOrdering: Seq[SortOrder] = originalPlan.outputOrdering
+
+ override def dynamicPruningFilters: Seq[Expression] = runtimeFilters
+
+ override def withDynamicPruningFilters(filters: Seq[Expression]): SparkPlan = {
+ // A real copy: runtimeFilters is a constructor field included in equality, so the copy
+ // survives enclosing-block rebuilds, and the derived scanHelper picks up the rewritten
+ // filters automatically.
+ copy(runtimeFilters = filters)
+ }
+
+ /**
+ * Lazy split-mode serialization, mirroring CometNativeScanExec: common data was serialized at
+ * planning; per-partition file lists serialize here, at execution time.
+ */
+ @transient private lazy val serializedPartitionData
+ : (Array[Byte], Array[Array[Byte]], Array[Seq[String]]) = {
+ // Resolve the helper's DPP subqueries: it holds its own InSubqueryExec instances that
+ // Spark's expressions walk does not see (the helper is derived, not a child).
+ scanHelper.partitionFilters.foreach {
+ case DynamicPruningExpression(e: InSubqueryExec) if e.values().isEmpty =>
+ e.updateResult()
+ case _ =>
+ }
+
+ val commonBytes = {
+ val deltaScan = DeltaSparkScanEnvelope.unpack(nativeOp)
+ // Scalar subqueries in dataFilters were unresolved at planning; resolve them now and
+ // append them as pushed filters, as CometNativeScanExec.serializedPartitionData does.
+ // has_data_filters follows their presence, not the serialized count: a filter that fails
+ // to serialize still keeps native on the safe timestamp conversion for a filtered scan.
+ val resolved = org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .resolvedSubqueryFilters(dataFilters, output, requiredSchema, conf)
+ val common = if (!resolved.hasResolvedFilters) {
+ deltaScan.getCommon
+ } else {
+ val builder = deltaScan.getCommon.toBuilder
+ builder.setHasDataFilters(true)
+ resolved.protos.foreach(builder.addDataFilters)
+ builder.build()
+ }
+ OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaScan.getDeltaCommon)
+ .build()
+ .toByteArray
+ }
+
+ val filePartitions = scanHelper.getFilePartitions()
+
+ val tableRoot = DeltaSparkScanEnvelope.unpack(nativeOp).getDeltaCommon.getTableRoot
+ val perPartitionBytes = filePartitions.map { filePartition =>
+ org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .serializePartition(filePartition, originalPlan, tableRoot)
+ }.toArray
+
+ val perPartitionPaths = filePartitions.map(_.files.map(_.filePath.toString).toSeq).toArray
+
+ (commonBytes, perPartitionBytes, perPartitionPaths)
+ }
+
+ override def commonData: Array[Byte] = serializedPartitionData._1
+
+ override def perPartitionData: Array[Array[Byte]] = serializedPartitionData._2
+
+ def perPartitionFilePaths: Array[Seq[String]] = serializedPartitionData._3
+
+ override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+ val nativeMetrics = CometMetricNode.fromCometPlan(this)
+ val serializedPlan = CometExec.serializeNativePlan(nativeOp)
+
+ new CometExecRDD(
+ sparkContext,
+ Seq.empty,
+ Map(sourceKey -> commonData),
+ Map(sourceKey -> perPartitionData),
+ serializedPlan,
+ perPartitionData.length,
Review Comment:
### Correctness
[P2] Adapt both Delta integration sites to the merged plan-data API
The main merge brings in #5615, but this constructor call still uses its previous argument list. `CometExecRDD` now requires `planFingerprint: Long` after `serializedPlan`, so `nativeMetrics` here is matched to `numOutputCols: Int` and compilation fails. There is a second stale site in [DeltaPlanDataInjector](https://github.com/apache/datafusion-comet/blob/d21f5d4d195e8db5ac6df1c22fce4fb057f33176/contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/DeltaPlanDataInjector.scala#L33-L62): it still overrides `inject(..., Array[Byte], ...)` and does not implement the new `prepareCommon`/`Prepared` contract.
Source-extracted probes reproduce both failures with Scala 2.12.18 and 2.13.16. The prior signatures and adapted controls compile. These are isolated probes with type stubs, not a full module build. Could you pass `PlanDataInjector.planFingerprint(serializedPlan)` here, adapt the provider to prepare the common protobuf once, and compile the Delta contrib for both Scala lines? The runtime opt-in does not avoid compiling these sources under `-Pdelta`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5608491136
The pom item is resolved, and the way you resolved it is better than the alternative I suggested.
The root `<build>` enforcer no longer carries `org.apache.comet.*`; it is back to `UnusedStubClass` and `TypeQualifier` only, and the exception now lives in `contrib/delta-spark/pom.xml` under the same execution id with `combine.children="append"`, so it extends the inherited rule for that module alone. Thanks for actually trying the dependency-exclusion route and reporting why it fails: a reactor test run resolving `comet-spark` from unshaded `target/classes` and then hitting `NoClassDefFoundError: org/apache/comet/CometRuntimeException` is a good reason, and it is the sort of thing that would otherwise be re-proposed every six months.
The build gate is still stale, though, and I owe you a correction on part of what I asked for.
### What I got wrong
I suggested pinning that `--no-default-features` pulls in neither `roaring` nor `crc32fast`. That invariant does not exist and never did. Both crates are already in the tree transitively without the `delta` feature:
```
crc32fast v1.5.1
├── apache-avro v0.21.0 -> iceberg v0.10.1 -> datafusion-comet
└── datafusion-comet-shuffle -> datafusion-comet
roaring v0.11.5
└── iceberg v0.10.1 -> datafusion-comet
```
So `delta = ["dep:roaring", "dep:crc32fast"]` adds no new crate to the default build; it only promotes two existing transitive deps to direct ones. That strengthens your case for keeping `delta` in the default set, and it should go in the `Cargo.toml` comment next to the #5411 pointer, because "it pulls in two extra crates" is the objection a reader will otherwise assume.
### What is still wrong
The gate conflates the two features. `dev/verify-contrib-delta-gate.sh`'s header says it verifies that the build "keeps Delta surface out of default builds" and that layer 1 checks "default `cargo build` doesn't compile `comet-contrib-delta`". Neither statement matches the tree:
- `default = ["hdfs-opendal", "delta"]`, and `delta` gates real code, `delta_dv.rs` plus eight `#[cfg(feature = "delta")]` sites in `planner.rs`. So the default dylib does carry Delta surface. The header claims otherwise.
- Layer 1 runs `cargo tree -p datafusion-comet --no-default-features` and calls that the default build. It is not: the default tree has 30 `opendal` lines against 25 without default features, so the command under test is a configuration nobody ships.
The check's substance is fine and I verified it holds where it matters. `comet-contrib-delta` and `delta_kernel` are absent from the *actual* default tree, not just from the `--no-default-features` one:
```
default tree contains contrib-delta/delta_kernel: 0
--no-default-features tree contains them: 0
```
So the fix is small: point layer 1 at `cargo tree -p datafusion-comet` with no flag, and reword the header and the `.github/workflows/delta_build_gate.yml` comment to the invariant that actually holds, which is that the heavy kernel-backed `contrib-delta` crate stays out of every shipped build while the small default-on `delta` feature is deliberately in. Keeping `--no-default-features` as an additional case is fine, it just is not the one the prose describes.
The `delta_syms` grep is the other half. It matches `comet_contrib_delta|delta_kernel|deltadvfilter|deltasynthetic`, none of which the default-on `delta` code exports, so the symbol layer reports OK for the same reason the tree layer does, not because the default build is Delta-free. Pinning the default-on surface near the 82 KB you measured would make that layer say something the grep cannot drift away from.
On the selected-path finding, probing every distinct data-file and deletion-vector URI rather than their parents is the right shape, and 0.75 microseconds per file as a pure URL parse with no I/O is comfortably under the scan's own per-file cost. The `CONVERT TO DELTA` test with a newline in a retained basename is a good regression, and better than a synthetic one because it is how the shape actually arises.
Everything else from my last pass still holds. `ParquetAccessPlan::scan_selection` intersecting rather than overwriting, `SparkDatetimeRebaseExpr` being opaque to `PruningPredicate`, and throwing on `RowIndexFilterType.IF_NOT_CONTAINED` are all still correct at this head, and my maintainer call on keeping `delta` default-on stands, now with a better justification than the one I gave.
Happy to approve once the gate says what it checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "erikbogado-nstech (via GitHub)" <gi...@apache.org>.
erikbogado-nstech commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5529185736
erikbogado@gmail.com
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5522992607
Thanks for the depth, and for the independent verification of the DV decode path.
Timing note first: your review and my last push crossed. 053ec14b4 (pushed about an hour after your review) plumbs the effective session read modes into the scan, so the resolve_spec comment you quoted is gone: metadata-free files now follow the resolved datetimeRebaseModeInRead and int96RebaseModeInRead exactly as ParquetFileFormat resolves them (CORRECTED reads verbatim, LEGACY rebases, EXCEPTION refuses ancient), and the nested branch now checks the applicable leaf policy, so STRUCT with a DATE leaf under corrected dates plus legacy INT96 stays native. Both of your reproductions are regression tests on raw parquet-mr files with no Spark metadata.
On the sharper point that refusals bypass the decline mechanism: after the modes fix the residual execution-time refusal class is much narrower, but it is not empty and I want to be precise about what remains. It fires only when ancient values actually appear in a file whose policy resolved LEGACY with a non-UTC or unrecorded writer zone, or with contradictory legacy flags. Under EXCEPTION mode an execution-time failure matches Spark's own behavior for those values, so the true fallback-bypass is the LEGACY non-UTC sliver, where Spark succeeds using zone tables native does not have. Declining at planning time requires knowing the file's writer zone, which the JVM side cannot see without reading footers; the conservative alternative is declining every scan whenever an effective mode is LEGACY, which over-declines tables that are entirely modern or UTC-written and would give up the native path for data that reads correctly. That is a genuine tradeoff rather than an oversight, and I
am happy to implement the planning-time decline for LEGACY modes if you and @sunchao prefer safety-of-mechanism over coverage here; it is a small change to the existing gate.
Core-path gaps: agreed on both, and they deserve to exist independently of this PR. The shared comparator extraction was already filed as #5658; I have now filed #5662 for the concrete correctness gap you named (the regular native scan forwarding S3 options with no divergence check). The calendar gap for the plain path is #5010, and the rebase module here was built so it can be lifted wholesale when core wants it; the delta-arm flag is the isolation seam, not a design commitment.
On delta in default cargo features: the motivation is exactly what you inferred, letting the contrib jar work against stock Comet binaries so opting in is a Spark-side decision rather than a custom native build. I agree the philosophy conflict with the adjacent contrib-delta comment and #5411 should be settled explicitly rather than by default; I am fine either way and will move it out of default if that is the call. Same for sequencing against #4366, which is yours and sunchao's decision to make; happy to record whatever is agreed in the Cargo.toml comment and the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5523616898
One correction on the default cargo feature, since it has now come up twice: this was proposed and accepted earlier in this thread (Aug 19), and sunchao clarified on Aug 27 that the adjacent contrib-delta comment means zero kernel surface, not zero Delta surface, so the two features are not in conflict. The cost is about 82 KB in the dylib and the code is unreachable without the contrib jar plus the config. I have added a cross-reference to that decision in the Cargo.toml comment and reworded the contrib-delta one to say delta-kernel explicitly (70c5be8b4) so it does not need re-litigating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3920753757
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala:
##########
@@ -125,166 +150,233 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with CometTypeS
scan: CometScanExec,
builder: Operator.Builder,
childOp: OperatorOuterClass.Operator*): Option[OperatorOuterClass.Operator] = {
- val nativeScanBuilder = OperatorOuterClass.NativeScan.newBuilder()
+ // Extract object store options from first file (S3 configs apply to all files in scan).
+ // Use selectedPartitions (static) instead of getFilePartitions() because at planning time
+ // DPP subqueries haven't been resolved yet. Object store options don't depend on DPP.
+ val firstFileUri = scan.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ // Collect S3/cloud storage configurations
+ val hadoopConf = scan.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scan.relation.options)
+
+ buildNativeScanCommon(
+ source = scan.simpleStringWithNodeId(),
+ output = scan.output,
+ requiredSchema = scan.requiredSchema,
+ dataSchema = scan.relation.dataSchema,
+ partitionSchema = scan.relation.partitionSchema,
+ fileConstantMetadataColumns = scan.wrapped.fileConstantMetadataColumns,
+ dataFilters = scan.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scan.conf) match {
+ case Some(commonBuilder) =>
+ // Sink operators don't have children
+ builder.clearChildren()
+ val nativeScanBuilder = OperatorOuterClass.NativeScan.newBuilder()
+ // Set common data in NativeScan (file_partition will be populated at execution time)
+ nativeScanBuilder.setCommon(commonBuilder.build())
+ Some(builder.setNativeScan(nativeScanBuilder).build())
+ case None =>
+ // There are unsupported scan type
+ withFallbackReason(
+ scan,
+ s"unsupported Comet operator: ${scan.nodeName}, due to unsupported data types above")
+ None
+ }
+ }
+
+ /**
+ * Build the `NativeScanCommon` proto shared by the core parquet scan and contrib scans that
+ * delegate to the same native parquet machinery (e.g. a Delta scan contrib, which passes
+ * physical-name schemas under column mapping). Returns `None` when an output data type cannot
+ * be serialized; the caller is responsible for tagging a fallback reason.
+ *
+ * Visibility note: `private[comet]` means a contrib caller must live under an
+ * `org.apache.comet.*` package (the same constraint `PlanDataInjector` implementers have).
+ */
+ private[comet] def buildNativeScanCommon(
+ source: String,
+ output: Seq[Attribute],
+ requiredSchema: StructType,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ fileConstantMetadataColumns: Seq[AttributeReference],
+ dataFilters: Seq[Expression],
+ firstFileUri: Option[URI],
+ hadoopConf: Configuration,
+ conf: SQLConf): Option[OperatorOuterClass.NativeScanCommon.Builder] = {
val commonBuilder = OperatorOuterClass.NativeScanCommon.newBuilder()
// Set source in common (used as part of injection key)
- commonBuilder.setSource(scan.simpleStringWithNodeId())
+ commonBuilder.setSource(source)
- val scanTypes = scan.output.flatten { attr =>
+ val scanTypes = output.flatten { attr =>
serializeDataType(attr.dataType)
}
- if (scanTypes.length == scan.output.length) {
- commonBuilder.addAllFields(scanTypes.asJava)
-
- // Sink operators don't have children
- builder.clearChildren()
-
- if (scan.conf.getConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
- val dataFilters = new ListBuffer[Expr]()
- for (filter <- scan.supportedDataFilters) {
- exprToProto(filter, scan.output) match {
- case Some(proto) => dataFilters += proto
- case _ =>
- logWarning(s"Unsupported data filter $filter")
- }
+ if (scanTypes.length != output.length) {
+ // There are unsupported scan types
+ return None
+ }
+ commonBuilder.addAllFields(scanTypes.asJava)
+
+ if (conf.getConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ val filterProtos = new ListBuffer[Expr]()
+ for (filter <- dataFilters) {
+ exprToProto(filter, output) match {
+ case Some(proto) => filterProtos += proto
+ case _ =>
+ logWarning(s"Unsupported data filter $filter")
}
- commonBuilder.addAllDataFilters(dataFilters.asJava)
- }
-
- val possibleDefaultValues = getExistenceDefaultValues(scan.requiredSchema)
- if (possibleDefaultValues.exists(_ != null)) {
- // Our schema has default values. Serialize two lists, one with the default values
- // and another with the indexes in the schema so the native side can map missing
- // columns to these default values.
- val (defaultValues, indexes) = possibleDefaultValues.iterator.zipWithIndex
- .filter { case (expr, _) => expr != null }
- .map { case (expr, index) =>
- // ResolveDefaultColumnsUtil.getExistenceDefaultValues has evaluated these
- // expressions and they should now just be literals.
- (Literal(expr), index.toLong.asInstanceOf[java.lang.Long])
- }
- .toList
- .unzip
- commonBuilder.addAllDefaultValues(
- defaultValues.flatMap(exprToProto(_, scan.output)).asJava)
- commonBuilder.addAllDefaultValuesIndexes(indexes.asJava)
}
+ commonBuilder.addAllDataFilters(filterProtos.asJava)
+ }
- // Extract object store options from first file (S3 configs apply to all files in scan).
- // Use selectedPartitions (static) instead of getFilePartitions() because at planning time
- // DPP subqueries haven't been resolved yet. Object store options don't depend on DPP.
- val firstFileUri = scan.selectedPartitions
- .flatMap(_.files.headOption)
- .headOption
- .map(_.getPath.toUri)
-
- // Constant metadata columns (file_path, file_name, file_size, file_block_start,
- // file_block_length, file_modification_time) are known before opening the file and
- // constant for every row read from it, exactly like partition columns. Spark places
- // them immediately after partition columns in `scan.output`
- // (FileSourceStrategy.scala: readDataColumns ++ generatedMetadataColumns ++
- // partitionColumns ++ constantMetadataColumns), so appending them after the real
- // partition schema here keeps the two in lockstep.
- val constantMetadataFields = scan.wrapped.fileConstantMetadataColumns.map(attr =>
- StructField(s"$constantMetadataFieldPrefix${attr.name}", attr.dataType, attr.nullable))
- val partitionSchemaFields = scan.relation.partitionSchema.fields.toSeq ++
- constantMetadataFields
- val partitionSchema = schema2Proto(partitionSchemaFields)
- val requiredSchema = schema2Proto(scan.requiredSchema)
-
- // Spark's required schema can prune a Variant column, including a Variant nested under an
- // unrequested struct. The complete relation schema still contains that unsupported type,
- // and serializing it would throw even though the native reader never needs those bytes.
- // Keep ordinary fields unchanged and replace a requested Variant-bearing root with its
- // already-validated, pruned required field. A requested actual Variant never reaches this
- // point because CometScanRule keeps those scans on Spark.
- val nativeDataSchema = StructType(scan.relation.dataSchema.fields.flatMap { field =>
- if (containsVariantType(field.dataType)) {
- scan.requiredSchema.fields.find(requiredField =>
- scan.conf.resolver(requiredField.name, field.name))
- } else {
- Some(field)
- }
- })
- val dataSchema = schema2Proto(nativeDataSchema)
-
- val dataSchemaIndexes = scan.requiredSchema.map(field => {
- nativeDataSchema.fieldIndex(field.name)
- })
- val partitionSchemaIndexes = nativeDataSchema.fields.length until
- (nativeDataSchema.length + partitionSchemaFields.length)
-
- val projectionVector = (dataSchemaIndexes ++ partitionSchemaIndexes).map(idx =>
- idx.toLong.asInstanceOf[java.lang.Long])
-
- commonBuilder.addAllProjectionVector(projectionVector.asJava)
-
- // In `CometScanRule`, we ensure partitionSchema (including constant metadata columns)
- // is supported.
- assert(partitionSchema.length == partitionSchemaFields.length)
-
- commonBuilder.addAllDataSchema(dataSchema.asJava)
- commonBuilder.addAllRequiredSchema(requiredSchema.asJava)
- commonBuilder.addAllPartitionSchema(partitionSchema.asJava)
- commonBuilder.setSessionTimezone(scan.conf.getConfString("spark.sql.session.timeZone"))
- commonBuilder.setCaseSensitive(scan.conf.getConf[Boolean](SQLConf.CASE_SENSITIVE))
-
- // SPARK-53535 (Spark 4.1+): when reading a struct whose requested fields are all
- // missing in the Parquet file, the new default preserves the parent struct's
- // nullness from the file (so non-null parents materialize as a struct of all-null
- // fields). Pre-4.1 Spark hardcodes the legacy behavior (whole struct null), which
- // matches the Comet default we use as fallback.
- val returnNullStructConfKey =
- "spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing"
- val returnNullStructDefault = if (isSpark41Plus) "false" else "true"
- commonBuilder.setReturnNullStructIfAllFieldsMissing(
- scan.conf.getConfString(returnNullStructConfKey, returnNullStructDefault).toBoolean)
-
- // Field-ID matching: only ask the native side to do extra work when the conf is on AND
- // the requested schema actually carries IDs. Spark's ParquetReadSupport applies the same
- // gate before invoking matchIdField.
- val useFieldId =
- scan.conf.getConf(SQLConf.PARQUET_FIELD_ID_READ_ENABLED) &&
- ParquetUtils.hasFieldIds(scan.requiredSchema)
- commonBuilder.setUseFieldId(useFieldId)
- commonBuilder.setIgnoreMissingFieldId(
- scan.conf.getConf(SQLConf.IGNORE_MISSING_PARQUET_FIELD_ID))
-
- commonBuilder.setAllowTypePromotion(CometConf.COMET_SCHEMA_EVOLUTION_ENABLED)
- commonBuilder.setAllowTimestampLtzToNtz(CometConf.COMET_ALLOW_TIMESTAMP_LTZ_AS_NTZ)
-
- // Collect S3/cloud storage configurations
- val hadoopConf = scan.relation.sparkSession.sessionState
- .newHadoopConfWithOptions(scan.relation.options)
-
- commonBuilder.setEncryptionEnabled(CometParquetUtils.encryptionEnabled(hadoopConf))
-
- firstFileUri.foreach { uri =>
- val objectStoreOptions =
- NativeConfig.extractObjectStoreOptions(hadoopConf, uri)
- objectStoreOptions.foreach { case (key, value) =>
- commonBuilder.putObjectStoreOptions(key, value)
+ val possibleDefaultValues = getExistenceDefaultValues(requiredSchema)
+ if (possibleDefaultValues.exists(_ != null)) {
+ // Our schema has default values. Serialize two lists, one with the default values
+ // and another with the indexes in the schema so the native side can map missing
+ // columns to these default values.
+ val (defaultValues, indexes) = possibleDefaultValues.iterator.zipWithIndex
+ .filter { case (expr, _) => expr != null }
+ .map { case (expr, index) =>
+ // ResolveDefaultColumnsUtil.getExistenceDefaultValues has evaluated these
+ // expressions and they should now just be literals.
+ (Literal(expr), index.toLong.asInstanceOf[java.lang.Long])
}
+ .toList
+ .unzip
+ commonBuilder.addAllDefaultValues(defaultValues.flatMap(exprToProto(_, output)).asJava)
+ commonBuilder.addAllDefaultValuesIndexes(indexes.asJava)
+ }
+
+ // Constant metadata columns (file_path, file_name, file_size, file_block_start,
+ // file_block_length, file_modification_time) are known before opening the file and
+ // constant for every row read from it, exactly like partition columns. Spark places
+ // them immediately after partition columns in the scan output
+ // (FileSourceStrategy.scala: readDataColumns ++ generatedMetadataColumns ++
+ // partitionColumns ++ constantMetadataColumns), so appending them after the real
+ // partition schema here keeps the two in lockstep.
+ val constantMetadataFields = uniqueConstantMetadataFields(
+ fileConstantMetadataColumns,
+ dataSchema.fields.map(_.name).toSet ++ partitionSchema.fields.map(_.name).toSet)
+ val partitionSchemaFields = partitionSchema.fields.toSeq ++ constantMetadataFields
+ val partitionSchemaProto = schema2Proto(partitionSchemaFields)
+ val requiredSchemaProto = schema2Proto(requiredSchema)
+
+ // Spark's required schema can prune a Variant column, including a Variant nested under an
+ // unrequested struct. The complete relation schema still contains that unsupported type,
+ // and serializing it would throw even though the native reader never needs those bytes.
+ // Keep ordinary fields unchanged and replace a requested Variant-bearing root with its
+ // already-validated, pruned required field. A requested actual Variant never reaches this
+ // point because CometScanRule keeps those scans on Spark.
+ val nativeDataSchema = StructType(dataSchema.fields.flatMap { field =>
+ if (containsVariantType(field.dataType)) {
+ requiredSchema.fields.find(requiredField => conf.resolver(requiredField.name, field.name))
+ } else {
+ Some(field)
}
+ })
+ val dataSchemaProto = schema2Proto(nativeDataSchema)
- // Set common data in NativeScan (file_partition will be populated at execution time)
- nativeScanBuilder.setCommon(commonBuilder.build())
+ val dataSchemaIndexes = requiredSchema.map(field => {
+ nativeDataSchema.fieldIndex(field.name)
+ })
+ val partitionSchemaIndexes = nativeDataSchema.fields.length until
+ (nativeDataSchema.length + partitionSchemaFields.length)
- Some(builder.setNativeScan(nativeScanBuilder).build())
+ val projectionVector = (dataSchemaIndexes ++ partitionSchemaIndexes).map(idx =>
+ idx.toLong.asInstanceOf[java.lang.Long])
- } else {
- // There are unsupported scan type
- withFallbackReason(
- scan,
- s"unsupported Comet operator: ${scan.nodeName}, due to unsupported data types above")
- None
+ commonBuilder.addAllProjectionVector(projectionVector.asJava)
+
+ // In `CometScanRule`, we ensure partitionSchema (including constant metadata columns)
+ // is supported.
+ assert(partitionSchemaProto.length == partitionSchemaFields.length)
+
+ commonBuilder.addAllDataSchema(dataSchemaProto.asJava)
+ commonBuilder.addAllRequiredSchema(requiredSchemaProto.asJava)
+ commonBuilder.addAllPartitionSchema(partitionSchemaProto.asJava)
+
+ populateScanConfFlags(commonBuilder, requiredSchema, firstFileUri, hadoopConf, conf)
+
+ Some(commonBuilder)
+ }
+
+ /**
+ * Populate the configuration-derived flags of a `NativeScanCommon`: session timezone, case
+ * sensitivity, struct-nullness legacy flag, field-ID matching, type promotion, encryption, and
+ * object-store options. Shared with contrib scans that assemble their own schemas/projection
+ * (e.g. the Delta contrib's deletion-vector shape) so new flags added here reach them without
+ * drift.
+ */
+ private[comet] def populateScanConfFlags(
+ commonBuilder: OperatorOuterClass.NativeScanCommon.Builder,
+ requiredSchema: StructType,
+ firstFileUri: Option[URI],
+ hadoopConf: Configuration,
+ conf: SQLConf): Unit = {
+ commonBuilder.setSessionTimezone(conf.getConfString("spark.sql.session.timeZone"))
+ val caseSensitive = conf.getConf[Boolean](SQLConf.CASE_SENSITIVE)
+ commonBuilder.setCaseSensitive(caseSensitive)
+ if (!caseSensitive) {
+ // Ship THIS JVM's case data so native's case-insensitive footer matching reproduces
+ // this JVM's `toLowerCase(Locale.ROOT)` exactly, whatever Unicode version it bundles.
+ JvmCaseTables.populate(commonBuilder)
Review Comment:
Gone with the case tables: nothing rides in NativeScanCommon now, so plans are back to their interned size and QueryContextInternerSuite passes on the default configuration again with no pin. Good catch on the pin masking that, the suite is untouched from main on this branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5519861917
Thanks for the depth here, and especially for independently verifying the sigma port over 212k strings before asking for the split. Agreed on the restructure, and #5602 landing made most of it the obviously right call. Here is where things stand:
1. The case folding stack is gone entirely. JvmCaseTables, the schema_adapter port, the three proto fields, the parity suite, and the QueryContextInternerSuite pin are all dropped in favor of main's name_fold implementation, which the Delta scan inherits automatically through the shared parquet path. That also dissolves the #5495 concern, since #5602 owns that fix now.
2. The three small fixes moved to #5653 (object store variable expansion, constant metadata uniquification, dead JNI removal). The first two are prerequisites of the contrib, so this branch is based on that PR and shows its commits until it merges.
3. The duplicate field and field id semantics are going out as their own PR shortly, re-derived on top of #5602's folding. Main's version already covers the id-routed duplicate exemption and the case insensitive struct duplicate error, so that PR carries only the shield generalization, the duplicate requested field id error, and a last-field-wins fix for case sensitive duplicate names that main currently resolves first-wins.
What remains here is the Delta contrib plus the refactors it genuinely needs, per your list. The branch is rebased on current main, both suite profiles pass, and QueryContextInternerSuite passes on the default configuration again. The description is updated to match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5525514652
All three addressed in 52efcfe28, with the semantics checked against the 3.5.9 sources first.
Timestamp cutoff: createTimestampRebaseFuncInRead under EXCEPTION throws only below RebaseDateTime.lastSwitchJulianTs, which computes to 1900-01-01T00:00:00Z from the rebase json (the latest zone switch across all 602 zones; UTC's own is the 1582 cutover). MILLIS columns convert to micros before the check. The native check now uses that cutoff in whichever unit the column carries, dates keep lastSwitchJulianDay, and there are tests on both sides of the boundary per unit plus the Scala regressions you asked for (a metadata-free INT64 at 1969-12-31T23:59:59.999999Z reads verbatim under EXCEPTION, one before 1900 fails loudly).
INT64 vs INT96 attribution: you were right that merging the two specs was wrong, and the arrow schema alone cannot fix it since the opener runs Int96Coercer before the adapter sees anything, so INT96 and INT64 MICROS both arrive as Timestamp(us, UTC). The honest source is the footer's SchemaDescriptor physical types, so our metadata reader factory now records the INT96 leaf ordinals into the file's in-memory key-value metadata when it loads the footer, and the adapter attributes each timestamp leaf by depth-first ordinal, validated against the leaf count and replacing any pre-existing key rather than trusting it. INT64 columns follow the datetime spec and INT96 columns follow the INT96 spec independently; with attribution available, files whose two legacy flags disagree now read correctly instead of refusing. Tests cover both directions at the Rust and Spark levels, including an INT96 column written through the low-level writer alongside an INT64 one, and the old mixed-flags refus
al test became a four-combination round-trip against Spark's own read.
Nested types: rather than declining, the wrapper now rebuilds struct, list, large list, fixed size list, map, and dictionary arrays with each leaf transformed under its own policy, preserving nulls and offsets, so STRUCT with a DATE leaf under EXCEPTION reads modern and null data natively and only an actual ancient leaf value fails. Tests cover modern and null leaves under every policy, an ancient leaf rebasing exactly under LEGACY UTC and erroring under EXCEPTION, and list of struct of date round trips, at both levels.
On the narrowing: the description now states exactly what still fails at execution time instead of reading, which after these changes is a LEGACY file with a non UTC or unrecorded writer zone when a timestamp before 1900 actually appears, and an EXCEPTION file when an ancient value actually appears. It also notes that pruning is lost on every column that receives a policy wrapper, modern only LEGACY and check only EXCEPTION files included, and that the 65x figure is decoded rows rather than cold bytes or latency.
Battery at this head: 235/235 on both spark-3.5 and spark-4.0, 339 native tests, clippy and fmt clean. An independent pass over the change also caught an i64 overflow in the nanosecond identity check for the LEGACY UTC path, fixed by comparing in days, with a per-unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5618601437
Rebased onto #5827. The size-assertion removal and the `nm` requirement now come from main, so this head carries only the rest on top of it: layer 1 checking the default tree first and `--no-default-features` second, the header and workflow wording, and the `delta` feature presence check with the 512 KB footprint cap. The gate passes end to end at the head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3871835645
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,981 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.ListBuffer
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands. Anything else on the protocol declines the
+ * table. Note `deletionVectors` and `columnMapping` are declined separately (below) so their
+ * fallback reasons are specific.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: this is
+ * the first gate on every V1 scan and must stay inert when delta-spark is absent from the
+ * classpath, where `classOf[DeltaParquetFileFormat]` would raise NoClassDefFoundError inside
+ * CometScanRule and take down every parquet scan in the session. A name match proves
+ * delta-spark is present, so Delta types used past this gate are safe.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Returns the first reason this Delta scan cannot go native, or None when claimable. Only
+ * called when [[isDeltaScan]] is true. `scanHelper` is the same [[CometScanExec]] the caller
+ * builds to drive [[CometDeltaNativeScan.convert]] on a claim, reused here to resolve the
+ * scan's selected files for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Descriptor deserialization is expensive; hoisted once so it runs at most once per claim
+ // attempt. `lazy` since most scans are not DV-shaped and gates that return earlier should not
+ // pay for it.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported (the same type checker core's own
+ // FileSourceScanExec path runs) so scan-time type gates -- notably the default-on
+ // COMET_PARQUET_UNSIGNED_SMALL_INT_CHECK safety fallback for ShortType, plus the
+ // collation and shredded-variant-struct gates -- apply identically whether the scan is
+ // claimed through core or through this contrib. A pure in-memory schema check with no
+ // file or store I/O, so it runs first, ahead of every other gate below.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated.
+ val cmMode = metadata.columnMappingMode.name
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder uses the
+ // required schema verbatim as output: name-sensitive expressions (e.g. to_json) would leak
+ // physical names into results. Needs a rename adapter for the logical schema; decline until
+ // then.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
+ CometDeltaNativeScan.internalColumnNames
+ val unknownOutput = scanExec.output.map(_.name).filterNot(knownColNames.contains)
+ if (unknownOutput.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support generated column(s) ${unknownOutput.mkString(", ")}")
+ }
+
+ // Deletion-vector shape invariants (see CometDeltaNativeScan.buildDvScanCommon).
+ if (CometDeltaNativeScan.isDvShape(scanExec)) {
+ // A row-index column WITHOUT is_row_deleted is Delta DML bookkeeping (real row indexes),
+ // not a DV read; claiming it with a constant would corrupt the DVs being written.
+ val hasIsRowDeleted =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.IsRowDeletedColumn)
+ val hasRowIndex =
+ scanExec.requiredSchema.exists(_.name == CometDeltaNativeScan.RowIndexColumn)
+ if (hasRowIndex && !hasIsRowDeleted) {
+ return Some(
+ "Native Delta scan does not support row-index reads outside a deletion-vector scan")
+ }
+ // Internal columns must form a suffix of the read schema so data-column positions agree
+ // between Spark's output and the stripped native schema.
+ val names = scanExec.requiredSchema.fields.map(_.name)
+ val firstInternal = names.indexWhere(CometDeltaNativeScan.internalColumnNames.contains)
+ if (!names.drop(firstInternal).forall(CometDeltaNativeScan.internalColumnNames.contains)) {
+ return Some("Native Delta scan requires DV bookkeeping columns to trail the read schema")
+ }
+ // Native applies the DV itself and emits a dead constant for row-index, so the real value
+ // must be provably unused above the scan.
+ if (!rowIndexUnusedAbove(plan, scanExec)) {
+ return Some(
+ "Native Delta scan cannot supply _metadata.row_index values consumed by the query")
+ }
+ // The DV common builder does not serialize existence defaults yet.
+ if (getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with deletion vectors")
+ }
+ // Bound native's memory for expanded DV row selectors (delta_dv.rs), bounded above by
+ // 2*cardinality + #row-groups; the descriptor's cardinality is a sound, pessimistic upper
+ // bound. The conf below makes an over-pessimistic decline recoverable.
+ val maxDeletedRowsPerFile = DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.get()
+ val oversizedCardinalities = dvDescriptors
+ .map(_.cardinality)
+ .filter(_ > maxDeletedRowsPerFile)
+ if (oversizedCardinalities.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support a deletion vector deleting " +
+ s"${oversizedCardinalities.max} rows in a single file, exceeding " +
+ s"${DeltaScanConf.COMET_DELTA_MAX_DELETED_ROWS_PER_FILE.key}=$maxDeletedRowsPerFile")
+ }
+ }
+
+ // input_file_name & friends read from a thread-local Spark's FileScanRDD sets; the native scan
+ // does not populate it, and Delta's DML find-touched-files scans use it (mirrors core's check
+ // in CometScanRule.nativeScan).
+ if (plan.exists(node =>
+ node.expressions.exists(_.exists {
+ case _: InputFileName | _: InputFileBlockStart | _: InputFileBlockLength => true
+ case _ => false
+ }))) {
+ return Some(
+ "Native Delta scan is not compatible with input_file_name, " +
+ "input_file_block_start, or input_file_block_length")
+ }
+
+ // Row-index metadata columns are generated per-row by Spark's reader (mirrors core); the DV
+ // shape's trailing row-index column is exempt since the gates above already proved it dead.
+ if (!CometDeltaNativeScan.isDvShape(scanExec) &&
+ ShimFileFormat.findRowIndexColumnIndexInSchema(scanExec.requiredSchema) >= 0) {
+ return Some("Native Delta scan does not support row index generation")
+ }
+
+ // Mirror core's vectorized-reader compatibility gate.
+ if (!SQLConf.get.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED) &&
+ !CometConf.COMET_SCAN_ALLOW_DISABLED_PARQUET_VECTORIZED_READER.get()) {
+ return Some(
+ "Native Delta scan is incompatible with " +
+ s"${SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key}=false")
+ }
+
+ // Decline ALL encrypted-parquet configurations (stricter than core): the exec node does not
+ // yet wire the decryption-key broadcast to executors.
+ val hadoopConf = scanExec.relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(scanExec.relation.options)
+ if (CometParquetUtils.encryptionEnabled(hadoopConf)) {
+ return Some("Native Delta scan does not support encrypted parquet")
+ }
+
+ // Nested-type column defaults cannot be serialized; a dropped default would misalign the
+ // value/index lists consumed positionally on the native side. Mirrors core's
+ // transformV1Scan gate.
+ val possibleDefaultValues = getExistenceDefaultValues(scanExec.requiredSchema)
+ if (possibleDefaultValues.exists(d =>
+ d != null && (d.isInstanceOf[ArrayBasedMapData] || d
+ .isInstanceOf[GenericInternalRow] || d.isInstanceOf[GenericArrayData]))) {
+ return Some("Native Delta scan does not support default values for nested types")
+ }
+
+ // Only claim scans whose root paths object_store (or the configured libhdfs schemes) can
+ // actually read (mirrors core's unsupportedFsSchemes gate).
+ val libhdfs = libhdfsSchemes
+ val unsupportedRootSchemes =
+ unsupportedSchemes(scanExec.relation.location.rootPaths.map(_.toUri), libhdfs)
+ if (unsupportedRootSchemes.nonEmpty) {
+ return Some(
+ "Native Delta scan does not support filesystem scheme(s) " +
+ s"${unsupportedRootSchemes.mkString(", ")}")
+ }
+
+ // A shallow clone can span multiple object-store authorities; the native builder resolves the
+ // scan's ObjectStoreUrl from the FIRST selected file only, so a later file under a different
+ // store would silently read through the wrong handle. Force file listing (scanHelper is
+ // already built for the claim path) and decline rather than risk it.
+ val dataFileUris =
+ scanHelper.selectedPartitions.iterator.flatMap(_.files).map(_.getPath.toUri).toSeq
+
+ // Both gates below need the DV absolute-path URIs; dvDescriptors is already memoized.
+ val dvUris = dvDescriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(new Path(tableRoot)).toUri)
+
+ // The root-path gate above only inspects the table root(s); selected files can resolve
+ // through a different scheme (e.g. `viewfs:`). Checked before the authority gates below,
+ // which presume every URI is natively resolvable.
+ val unsupportedSelected = unsupportedSelectedSchemeReason(dataFileUris ++ dvUris, libhdfs)
+ if (unsupportedSelected.isDefined) {
+ return unsupportedSelected
+ }
+
+ // Checked before multiStoreReason, which presumes every URI resolves to a single store
+ // identity -- a userinfo-bearing authority provably does not (store keying drops userinfo).
+ val userInfoReason = userInfoBearingAuthorityReason(dataFileUris ++ dvUris)
+ if (userInfoReason.isDefined) {
+ return userInfoReason
+ }
+
+ val multiStore = multiStoreReason(dataFileUris)
+ if (multiStore.isDefined) {
+ return multiStore
+ }
+
+ // Credentials that resolve ONLY through a Hadoop credential provider (JCEKS et al.) are
+ // invisible to the plain-conf extraction forwarded to the native S3 client; reuses hadoopConf
+ // from the encryption gate above.
+ val credentialReason = credentialAliasReason(hadoopConf, dataFileUris ++ dvUris)
+ if (credentialReason.isDefined) {
+ return credentialReason
+ }
+
+ // A credential-provider class native's build_aws_credential_provider_metadata (s3.rs) does
+ // not recognize errors at scan EXECUTION time, after the scan was already claimed; decline
+ // eagerly instead.
+ val providerReason = providerClassGateReason(hadoopConf, dataFileUris ++ dvUris)
+ if (providerReason.isDefined) {
+ return providerReason
+ }
+
+ // Reuse core's generic native-scan gates (ignoreCorruptFiles/ignoreMissingFiles, AQE DPP on
+ // Spark 3.4, exec enabled); tags its own fallback reasons.
+ if (!CometNativeScan.isSupported(scanExec)) {
+ return Some("Core native scan gates rejected the scan (see reasons above)")
+ }
+
+ None
+ }
+
+ /**
+ * Deletion-vector descriptors for every file this DV-shape scan selected, deserialized once and
+ * normalized to absolute on-disk paths via `copyWithAbsolutePath`. Returns `Seq.empty` for the
+ * plain shape ([[CometDeltaNativeScan.isDvShape]] false). Shared by the DV cardinality gate and
+ * [[CometDeltaNativeScan.convert]]'s object-store option merge.
+ */
+ private[delta] def selectedDvDescriptors(
+ scanHelper: CometScanExec,
+ tableRoot: String): Seq[DeletionVectorDescriptor] = {
+ if (!CometDeltaNativeScan.isDvShape(scanHelper.wrapped)) {
+ return Seq.empty
+ }
+ val tableRootPath = new Path(tableRoot)
+ scanHelper.selectedPartitions.iterator
+ .flatMap(_.files)
+ .flatMap { file =>
+ file.metadata
+ .get(DeltaParquetFileFormat.FILE_ROW_INDEX_FILTER_ID_ENCODED)
+ .map(enc => DeletionVectorDescriptor.deserializeFromBase64(enc.asInstanceOf[String]))
+ }
+ .map(_.copyWithAbsolutePath(tableRootPath))
+ .toSeq
+ }
+
+ /**
+ * The libhdfs scheme exemption set from [[org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES]],
+ * lowercased and defaulting to `Set("hdfs")` when unset. Hoisted so [[declineReason]]'s
+ * root-path and selected-file scheme gates share one parsed set.
+ */
+ private[delta] def libhdfsSchemes: Set[String] = COMET_LIBHDFS_SCHEMES.get() match {
+ case Some(s) =>
+ s.split(",").map(_.trim.toLowerCase(Locale.ROOT)).filter(_.nonEmpty).toSet
+ case None => Set("hdfs")
+ }
+
+ /**
+ * The lowercased, deduplicated schemes among `uris` that neither `libhdfs` nor Comet's native
+ * object_store layer ([[CometScanRule.isNativelyReadableScheme]]) can read. A `null` scheme is
+ * tolerated, not flagged: such a URI cannot come from a Hadoop-backed source. Factored out of
+ * [[declineReason]]'s two scheme gates so both share one predicate, unit-testable without a
+ * Spark session.
+ */
+ private[delta] def unsupportedSchemes(uris: Seq[URI], libhdfs: Set[String]): Set[String] = {
+ uris
+ .filter { uri =>
+ val sch = uri.getScheme
+ sch != null && {
+ val sl = sch.toLowerCase(Locale.ROOT)
+ !libhdfs.contains(sl) && !CometScanRule.isNativelyReadableScheme(uri)
+ }
+ }
+ .map(_.getScheme.toLowerCase(Locale.ROOT))
+ .toSet
+ }
+
+ /**
+ * Decline reason when any of `uris` -- the scan's selected data-file and deletion-vector URIs
+ * -- use a scheme [[unsupportedSchemes]] flags, or `None` when every URI is natively readable
+ * (or libhdfs-exempt). See [[declineReason]]'s call site for ordering vs. the authority gates
+ * below.
+ */
+ private[delta] def unsupportedSelectedSchemeReason(
+ uris: Seq[URI],
+ libhdfs: Set[String]): Option[String] = {
+ val schemes = unsupportedSchemes(uris, libhdfs)
+ if (schemes.isEmpty) {
+ None
+ } else {
+ Some(
+ "Native Delta scan does not support selected data file or deletion vector filesystem " +
+ s"scheme(s) ${schemes.mkString(", ")}")
+ }
+ }
+
+ /**
+ * Decline reason when `uris` span more than one object-store authority (scheme plus the raw
+ * authority -- userinfo, host, port -- lowercased so `S3A://Bucket:1234` and
+ * `s3a://bucket:1234` collapse), or `None` when they share one. `file://` paths carry no
+ * authority, so local scans across many directories are unaffected.
+ */
+ private[delta] def multiStoreReason(uris: Seq[URI]): Option[String] = {
+ val authorities = uris.map(uriAuthority).distinct
+ if (authorities.size > 1) {
+ Some(
+ "Native Delta scan does not support data files spanning multiple object stores " +
+ s"(found: ${authorities.sorted.mkString(", ")})")
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes `uri` to a lowercased `scheme://authority` string, keyed on the raw `getAuthority`
+ * rather than the individually-parsed host/port/userinfo fields: `getAuthority` already
+ * includes userinfo, so containers on the same storage account don't collapse together; and
+ * `getHost` (and `getUserInfo`/`getPort`) return `null` for the whole authority when it doesn't
+ * conform to RFC 3986's `reg-name` syntax (e.g. an underscore in a GCS bucket name,
+ * `gs://my_bucket`), silently collapsing distinct buckets into one empty-host key. A `null`
+ * authority normalizes to the empty string.
+ */
+ private[delta] def uriAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ s"$scheme://$authority"
+ }
+
+ /**
+ * The raw userinfo component of `uri`'s authority, or empty when none. Splits at the LAST `@`
+ * rather than using `URI#getUserInfo`: as with [[uriAuthority]], the structured getters return
+ * `null` for the whole authority when it fails RFC 3986 `reg-name` syntax (e.g. an underscore
+ * in a GCS bucket name), hiding a real userinfo component. Never lowercased: userinfo is
+ * case-sensitive.
+ */
+ private[delta] def uriUserInfo(uri: URI): String = {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ if (at >= 0) authority.substring(0, at) else ""
+ }
+
+ /**
+ * Redacts `uri`'s authority to `scheme` + `://` + a literal `***` + `@host[:port]`, for
+ * embedding in a decline reason. NEVER interpolate `uri.getAuthority` or [[uriUserInfo]]
+ * directly into a reason string: doing so would leak credentials embedded as URI userinfo (e.g.
+ * `s3a://AKIA...:secret@bucket`) into the SQL plan's explain output, fallback-reason logging,
+ * or the Spark UI.
+ */
+ private[delta] def redactedAuthority(uri: URI): String = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT)).getOrElse("")
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostPort = if (at >= 0) authority.substring(at + 1) else authority
+ s"$scheme://***@$hostPort"
+ }
+
+ /**
+ * Decline reason when any of `uris` carries userinfo in its authority (e.g. the container in an
+ * abfss:// path), or `None` when none do. The native store cache, `ObjectStoreUrl`, and
+ * DataFusion registry all key on scheme/host/port only, dropping userinfo, so two authorities
+ * differing only in userinfo collide onto the same store handle -- normally a missing-object
+ * error, but silently wrong data if a same-named object exists under both.
+ */
+ private[delta] def userInfoBearingAuthorityReason(uris: Seq[URI]): Option[String] = {
+ val offending = uris.filter(uri => uriUserInfo(uri).nonEmpty).map(redactedAuthority).distinct
+ if (offending.isEmpty) {
+ None
+ } else {
+ Some("Native Delta scan does not support object-store paths whose authority carries " +
+ "userinfo (e.g. the container in an abfss:// path): the native object-store cache, " +
+ "ObjectStoreUrl and DataFusion registry all key on scheme, host and port only, so two " +
+ "containers on one storage account share a single store handle " +
+ s"(found: ${offending.sorted.mkString(", ")})")
+ }
+ }
+
+ /**
+ * String-literal Hadoop conf keys consulted by [[credentialAliasReason]] below. `hadoop-aws` is
+ * NOT on this module's runtime classpath (`spark-hadoop-cloud` is test-scope only), so
+ * `org.apache.hadoop.fs.s3a.Constants` must never be referenced here -- doing so would raise
+ * `NoClassDefFoundError` and take down every Delta scan in a session with no S3 dependency at
+ * all, not just S3 ones.
+ */
+ private val HadoopCredentialProviderPathKey = "hadoop.security.credential.provider.path"
+ private val S3aCredentialProviderPathKey = "fs.s3a.security.credential.provider.path"
+
+ private def s3aBucketProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.security.credential.provider.path"
+
+ /**
+ * The LONG form of [[s3aBucketProviderPathKey]]: `S3AUtils#lookupPassword` resolves per-bucket
+ * overrides through both a long key (`fs.s3a.bucket.B.<full base key>`) and a short key
+ * (`fs.s3a.bucket.B.<base key minus its "fs.s3a." prefix>`) -- see [[s3aCredentialAliases]] for
+ * why both must be covered here too.
+ */
+ private def s3aBucketLongProviderPathKey(bucket: String): String =
+ s"fs.s3a.bucket.$bucket.fs.s3a.security.credential.provider.path"
+
+ /**
+ * The aliases the native S3 credentials provider chain tries per bucket (short bucket form,
+ * then global), PLUS the LONG bucket form Hadoop's `S3AUtils#lookupPassword` also consults.
+ * Hadoop resolves long before short before global, keeping a non-empty long value over the
+ * short/global ones (`S3AUtils#getPassword` returns a non-empty `val` unchanged); native reads
+ * only short+global. So a JCEKS entry set ONLY under the long alias is resolved by Hadoop but
+ * invisible to native -- a shadowed value under ANY alias here means the caller must decline.
+ * List order only affects which alias name appears in the reason.
+ */
+ private def s3aCredentialAliases(bucket: String): Seq[String] =
+ Seq(
+ s"fs.s3a.bucket.$bucket.fs.s3a.access.key",
+ s"fs.s3a.bucket.$bucket.fs.s3a.secret.key",
+ s"fs.s3a.bucket.$bucket.fs.s3a.session.token",
+ s"fs.s3a.bucket.$bucket.access.key",
+ s"fs.s3a.bucket.$bucket.secret.key",
+ s"fs.s3a.bucket.$bucket.session.token",
+ "fs.s3a.access.key",
+ "fs.s3a.secret.key",
+ "fs.s3a.session.token")
+
+ private def nonEmptyConf(hadoopConf: Configuration, key: String): Boolean =
+ Option(hadoopConf.get(key)).exists(_.nonEmpty)
+
+ /**
+ * The lowercase-scheme-checked S3/S3A bucket name from `uri`'s authority (host, minus any
+ * userinfo or port), or `None` when `uri`'s scheme is not `s3`/`s3a`. Parses the raw authority
+ * manually (last `@`, then last `:`) rather than using `URI#getHost`: the same RFC 3986
+ * `reg-name` pitfall as [[uriAuthority]] applies (an underscore, valid in an S3 bucket name,
+ * makes `getHost` return `null` for the whole authority).
+ */
+ private def s3Bucket(uri: URI): Option[String] = {
+ val scheme = Option(uri.getScheme).map(_.toLowerCase(Locale.ROOT))
+ if (scheme.contains("s3") || scheme.contains("s3a")) {
+ val authority = Option(uri.getAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val hostAndPort = if (at >= 0) authority.substring(at + 1) else authority
+ val colon = hostAndPort.lastIndexOf(':')
+ val host = if (colon >= 0) hostAndPort.substring(0, colon) else hostAndPort
+ if (host.isEmpty) None else Some(host)
+ } else {
+ None
+ }
+ }
+
+ /**
+ * The three per-bucket S3A credential base keys the native S3 client's `get_config` (`s3.rs`)
+ * resolves: short bucket key first, then global (native never reads the long bucket key).
+ */
+ private val PlainCredentialBaseKeys: Seq[String] =
+ Seq("fs.s3a.access.key", "fs.s3a.secret.key", "fs.s3a.session.token")
+
+ private def plainValue(hadoopConf: Configuration, key: String): Option[String] =
+ Option(hadoopConf.get(key)).filter(_.nonEmpty)
+
+ /**
+ * The short-bucket-then-global value native's `get_config` (s3.rs) resolves for `baseKey` under
+ * `bucket`. Used by the credential-provider-class gates below, which read general S3A options
+ * the same way native does.
+ */
+ private def effectiveOptionValue(
+ hadoopConf: Configuration,
+ bucket: String,
+ baseKey: String): Option[String] = {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ plainValue(hadoopConf, shortKey).orElse(plainValue(hadoopConf, baseKey))
+ }
+
+ private def plainLongFormCredentialDivergenceReason(bucket: String, longKey: String): String =
+ "Native Delta scan cannot forward long-form bucket credentials for " +
+ s"$bucket ($longKey is set but the native S3 client only reads the short bucket and " +
+ "global keys, so its credentials would differ from Hadoop's)"
+
+ /**
+ * Zero-I/O plain-value divergence check (see [[s3aCredentialAliases]] for the long-before-short
+ * precedence). Native's `get_config` never reads the long form, so it can silently diverge from
+ * Hadoop's effective value. Declines when the long form is set and the two sides disagree
+ * (short set to the SAME value as long passes). Never interpolates a resolved value, only the
+ * key name.
+ *
+ * This applies to credentials ONLY. Hadoop resolves them via `S3AUtils#lookupPassword`, where
+ * the long bucket form (`fs.s3a.bucket.B.fs.s3a.<key>`) WINS when set -- a real, silent
+ * divergence from native, which never reads it. Every OTHER `fs.s3a` option instead flows
+ * through `S3AUtils#propagateBucketOptions`, which strips only ONE `fs.s3a.bucket.B.` prefix
+ * layer: a long-form key there folds into `fs.s3a.fs.s3a.<key>`, a key nothing else in Hadoop
+ * ever reads, so the long form is inert for those options -- Hadoop's own effective value
+ * already reduces to short.orElse(global), matching native. No gate is needed (or present) for
+ * that wider set; see DeltaScanContribSuite's pinned control test for the long-form endpoint
+ * case.
+ */
+ private def plainLongFormCredentialReason(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ PlainCredentialBaseKeys.foldLeft(Option.empty[String]) { (declined, baseKey) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ val shortKey = s"fs.s3a.bucket.$bucket." + baseKey.stripPrefix("fs.s3a.")
+ val longKey = s"fs.s3a.bucket.$bucket.$baseKey"
+ val long = plainValue(hadoopConf, longKey)
+ if (long.isEmpty) {
+ None
+ } else {
+ val short = plainValue(hadoopConf, shortKey)
+ val global = plainValue(hadoopConf, baseKey)
+ val hadoopEffective = long.orElse(short).orElse(global)
+ val nativeEffective = short.orElse(global)
+ if (hadoopEffective != nativeEffective) {
+ Some(plainLongFormCredentialDivergenceReason(bucket, longKey))
+ } else {
+ None
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * String-literal mirror of every credential-provider class name s3.rs's
+ * `build_aws_credential_provider_metadata` and `is_anonymous_credential_provider` recognize
+ * (Hadoop S3A names plus AWS SDK v1/v2 names). `hadoop-aws` is NOT on this module's runtime
+ * classpath (see the note above [[HadoopCredentialProviderPathKey]]), so these are string
+ * literals, never `org.apache.hadoop.fs.s3a.auth.*` or AWS SDK provider `classOf` references.
+ */
+ private val SupportedCredentialProviderClasses: Set[String] = Set(
+ "org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider",
+ "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider",
+ "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider",
+ "com.amazonaws.auth.ContainerCredentialsProvider",
+ "com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper",
+ "software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider",
+ "com.amazonaws.auth.InstanceProfileCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider",
+ "com.amazonaws.auth.EnvironmentVariableCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider",
+ "com.amazonaws.auth.WebIdentityTokenCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider",
+ "com.amazonaws.auth.profile.ProfileCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider",
+ "com.amazonaws.auth.AnonymousAWSCredentials")
+
+ private val AnonymousCredentialProviderClasses: Set[String] = Set(
+ "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
+ "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider",
+ "com.amazonaws.auth.AnonymousAWSCredentials")
+
+ private val HadoopAssumedRoleProviderClass =
+ "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider"
+
+ private val AwsCredentialsProviderKey = "fs.s3a.aws.credentials.provider"
+ private val AssumedRoleCredentialsProviderKey = "fs.s3a.assumed.role.credentials.provider"
+
+ /**
+ * Splits a Hadoop-style comma-separated credential-provider-class list the same way s3.rs's
+ * `parse_credential_provider_names` does: split on comma, trim each entry, drop empties.
+ */
+ private def parseProviderClassNames(value: String): Seq[String] =
+ value.split(",").map(_.trim).filter(_.nonEmpty).toSeq
+
+ private def unsupportedProviderReason(bucket: String, key: String, className: String): String =
+ s"Native Delta scan does not support the credential provider class $className " +
+ s"configured via $key for $bucket (the native S3 client only supports a fixed set of " +
+ "provider classes; an unsupported class would fail at scan execution time, after the " +
+ "scan was already claimed, rather than at planning time)"
+
+ private def mixedAnonymousProviderReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support $key for $bucket naming an anonymous credential " +
+ "provider together with any other provider (the native S3 client rejects this " +
+ "combination at scan execution time)"
+
+ private def anonymousAssumedRoleProviderReason(bucket: String, key: String): String =
+ s"Native Delta scan does not support an anonymous credential provider in $key for " +
+ s"$bucket (the native S3 client does not allow an anonymous provider as the base " +
+ "credentials for an assumed-role chain)"
+
+ private def unsupportedProviderNameReason(
+ bucket: String,
+ key: String,
+ names: Seq[String]): Option[String] =
+ names
+ .find(name => !SupportedCredentialProviderClasses.contains(name))
+ .map(unsupportedProviderReason(bucket, key, _))
+
+ /**
+ * Decline reason when `bucket`'s effective (short-bucket-then-global; see
+ * [[effectiveOptionValue]]) `assumed.role.credentials.provider` names an unsupported class, or
+ * an anonymous one (native's `build_assume_role_credential_provider_metadata` rejects ANY
+ * anonymous entry here, not just a mix). Unset defaults to native's own hardcoded
+ * `[SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider]` fallback, both
+ * always supported, so `None` is safe.
+ */
+ private def assumedRoleProviderClassReason(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ effectiveOptionValue(hadoopConf, bucket, AssumedRoleCredentialsProviderKey) match {
+ case None => None
+ case Some(value) =>
+ val names = parseProviderClassNames(value)
+ unsupportedProviderNameReason(bucket, AssumedRoleCredentialsProviderKey, names).orElse {
+ if (names.exists(AnonymousCredentialProviderClasses.contains)) {
+ Some(anonymousAssumedRoleProviderReason(bucket, AssumedRoleCredentialsProviderKey))
+ } else {
+ None
+ }
+ }
+ }
+ }
+
+ /**
+ * Decline reason when `bucket`'s effective `aws.credentials.provider` names a class native's
+ * `build_aws_credential_provider_metadata` (s3.rs) does not recognize, mixes an anonymous
+ * provider with any other provider (native's `build_credential_provider` rejects this
+ * combination), or -- when `AssumedRoleCredentialProvider` is among the names -- its
+ * `assumed.role.credentials.provider` sub-chain has the same problem. An unset/empty value is
+ * fine: native falls back to its own default AWS SDK provider chain. Checked so an unsupported
+ * class or invalid combination declines at planning time instead of erroring during scan
+ * execution, after the scan was already claimed.
+ */
+ private def providerClassReason(hadoopConf: Configuration, bucket: String): Option[String] = {
+ effectiveOptionValue(hadoopConf, bucket, AwsCredentialsProviderKey).flatMap { value =>
+ val names = parseProviderClassNames(value)
+ unsupportedProviderNameReason(bucket, AwsCredentialsProviderKey, names)
+ .orElse {
+ if (names.length > 1 && names.exists(AnonymousCredentialProviderClasses.contains)) {
+ Some(mixedAnonymousProviderReason(bucket, AwsCredentialsProviderKey))
+ } else {
+ None
+ }
+ }
+ .orElse {
+ if (names.contains(HadoopAssumedRoleProviderClass)) {
+ assumedRoleProviderClassReason(hadoopConf, bucket)
+ } else {
+ None
+ }
+ }
+ }
+ }
+
+ /**
+ * Returns the first reason any bucket among `uris` names an unsupported (or invalidly combined)
+ * credential-provider class, or `None` when every bucket's provider configuration is one native
+ * can build. See [[providerClassReason]].
+ */
+ private[delta] def providerClassGateReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ buckets.foldLeft(Option.empty[String]) { (declined, bucket) =>
+ if (declined.isDefined) declined else providerClassReason(hadoopConf, bucket)
+ }
+ }
+
+ private def s3aScopedProviderPathReason(bucket: String, providerPathKey: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($providerPathKey configures an S3A-scoped Hadoop credential provider that " +
+ "Configuration#getPassword does not consult, so the native S3 client's credentials " +
+ "cannot be verified)"
+
+ private def shadowedCredentialAliasReason(bucket: String, alias: String): String =
+ "Native Delta scan cannot forward Hadoop credential-provider aliases for " +
+ s"$bucket ($alias resolves through $HadoopCredentialProviderPathKey but is not present " +
+ "as a plain configuration value, so the native S3 client would have no credentials)"
+
+ private def unverifiableCredentialProviderReason(bucket: String, error: Throwable): String =
+ "Native Delta scan cannot verify Hadoop credential-provider aliases for " +
+ s"$bucket (reading $HadoopCredentialProviderPathKey raised " +
+ s"${error.getClass.getName}), declining rather than risk missing credentials"
+
+ /**
+ * Compares each [[s3aCredentialAliases]] alias's plain `Configuration#get` value against
+ * `Configuration#getPassword` (providers first, plain conf as fallback); a resolved value that
+ * differs means the alias is invisible or wrong to native's plain-conf extraction, so decline.
+ * Only called once the GLOBAL provider path is established as the sole path for `bucket`.
+ *
+ * Runs inside try/catch: `getPassword` performs real keystore I/O, and a corrupt or unreadable
+ * store must decline this bucket rather than abort planning for the whole session.
+ */
+ private def verifyGlobalProviderAliases(
+ hadoopConf: Configuration,
+ bucket: String): Option[String] = {
+ try {
+ s3aCredentialAliases(bucket).foldLeft(Option.empty[String]) { (declined, alias) =>
+ if (declined.isDefined) {
+ declined
+ } else {
+ val resolved =
+ Option(hadoopConf.getPassword(alias)).map(new String(_)).filter(_.nonEmpty)
+ resolved match {
+ case Some(value) if !Option(hadoopConf.get(alias)).contains(value) =>
+ Some(shadowedCredentialAliasReason(bucket, alias))
+ case _ => None
+ }
+ }
+ }
+ } catch {
+ case e @ (_: IOException | _: RuntimeException) =>
+ Some(unverifiableCredentialProviderReason(bucket, e))
+ }
+ }
+
+ /**
+ * The decline reason for `bucket` alone, or `None` when its credentials are safe to forward.
+ * `globalPathSet`/`s3aPathSet` are hoisted by the caller since they don't vary per bucket.
+ *
+ * Checked FIRST, regardless of provider-path config: [[plainLongFormCredentialReason]] is a
+ * zero-I/O, provider-unrelated check that is decisive on its own.
+ *
+ * Zero-I/O precheck: `None` immediately when none of the four provider-path keys are set -- a
+ * config-map lookup only, no keystore access.
+ *
+ * Arm A: the S3A-scoped provider-path keys point S3A's own resolution at a provider
+ * `Configuration#getPassword` does not consult, so any being set declines with no keystore
+ * read.
+ *
+ * Arm B: only the global path is set, which `getPassword` DOES consult; delegates to
+ * [[verifyGlobalProviderAliases]].
+ */
+ private def bucketCredentialAliasReason(
+ hadoopConf: Configuration,
+ bucket: String,
+ globalPathSet: Boolean,
+ s3aPathSet: Boolean): Option[String] = {
+ plainLongFormCredentialReason(hadoopConf, bucket).orElse {
+ val bucketPathKey = s3aBucketProviderPathKey(bucket)
+ val bucketLongPathKey = s3aBucketLongProviderPathKey(bucket)
+ val bucketPathSet = nonEmptyConf(hadoopConf, bucketPathKey)
+ val bucketLongPathSet = nonEmptyConf(hadoopConf, bucketLongPathKey)
+ if (!globalPathSet && !s3aPathSet && !bucketPathSet && !bucketLongPathSet) {
+ None
+ } else if (s3aPathSet || bucketPathSet || bucketLongPathSet) {
+ val offendingKey =
+ if (s3aPathSet) S3aCredentialProviderPathKey
+ else if (bucketPathSet) bucketPathKey
+ else bucketLongPathKey
+ Some(s3aScopedProviderPathReason(bucket, offendingKey))
+ } else {
+ verifyGlobalProviderAliases(hadoopConf, bucket)
+ }
+ }
+ }
+
+ /**
+ * Returns the first reason a native S3 scan cannot faithfully forward this table's Hadoop
+ * credentials, or `None` when claimable. `uris` is the scan's data-file and DV URIs; only
+ * `s3`/`s3a` authorities matter here (ABFS/WASB mooted by the userinfo gate, GCS out of scope).
+ * `Configuration#getPassword` checks every provider FIRST, falling back to plain conf only when
+ * unset, so a keystore-only or shadowing alias is invisible/wrong to native's plain-conf
+ * extraction; when in doubt, decline. Never interpolates a resolved value, only key names.
+ */
+ private[delta] def credentialAliasReason(
+ hadoopConf: Configuration,
+ uris: Seq[URI]): Option[String] = {
+ val buckets = uris.flatMap(s3Bucket).distinct
+ if (buckets.isEmpty) {
+ return None
Review Comment:
Stopped enumerating prefixes: the gate now declines any fs.gs.* or google.cloud.* key involving auth, which covers the deprecated service.account.auth forms you found and whatever other spellings the connector's prefix machinery accepts. ADC-only still claims. Happy to hear if you see a shape this misses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3876140496
##########
contrib/delta-spark/dev/run-delta-regression.sh:
##########
@@ -0,0 +1,169 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Run Delta Lake's own Spark test suites against a Comet build with the
+# native Delta scan enabled. Clones delta at $DELTA_VERSION into $WORKDIR,
+# injects Comet into the test SparkSession (DeltaSQLCommandTest) and the
+# test classpath (unmanagedJars), then runs the given testOnly selectors.
+#
+# Usage:
+# COMET_JARS=/path/comet-spark.jar,/path/comet-contrib-delta.jar,/path/flatbuffers.jar \
+# ./run-delta-regression.sh <workdir> 'org.apache.spark.sql.delta.DeletionVectorsSuite' [...]
+#
+# Env:
+# DELTA_VERSION delta tag to test against (default 3.3.2)
+# COMET_JARS comma-separated jars added to the test classpath (required)
+# JAVA_HOME JDK for sbt (17 recommended)
+set -euo pipefail
+
+DELTA_VERSION="${DELTA_VERSION:-3.3.2}"
+WORKDIR="${1:?usage: run-delta-regression.sh <workdir> <suite> [...suites]}"
+shift
+[ $# -ge 1 ] || { echo "no suites given" >&2; exit 2; }
+: "${COMET_JARS:?COMET_JARS must list the comet jars}"
+
+# Resolve to an absolute path before the cd below: the log path is built from
+# $WORKDIR after we're already inside the Delta checkout, so a relative
+# argument would otherwise be re-anchored under $DELTA_DIR.
+mkdir -p "$WORKDIR"
+WORKDIR=$(cd "$WORKDIR" && pwd)
+
+IFS=',' read -ra _jars <<< "$COMET_JARS"
+for j in "${_jars[@]}"; do
+ [ -f "$j" ] || { echo "COMET_JARS entry not found: $j" >&2; exit 2; }
Review Comment:
**[P3] Resolve COMET_JARS entries before changing directories**
Could we canonicalize each validated jar path before `cd "$DELTA_DIR"`, as already done for `WORKDIR`? For example, `COMET_JARS=jars/comet.jar` passes this check in the caller's directory, but line 158 exports it unchanged after the directory switch. The injected sbt `file(p)` then searches beneath the Delta checkout and cannot find the jar.
Verified with the unchanged script, a controlled checkout and stub sbt: the relative path passes validation but is missing at the consumer; the absolute-path control remains readable. No actual Delta/sbt build was run by this path-handling probe.
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so its rewrite (via
+ * [[CometScanWithPlanData]]) survives plan copies -- a transient field would be dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning (the CometIcebergNativeScanExec lesson).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node
+ // automatically gets a helper consistent with ITS runtimeFilters, avoiding the #3510 class of
+ // bug where a stored helper field desyncs from rewritten filters. Costs one extra file listing
+ // per executed instance; correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below, and AQE's UI
+ // plan-walk calls `.metrics` on every node MID-PLANNING, including while a DPP subquery is
+ // still an adaptive placeholder or a partition filter holds an unresolved ScalarSubquery (see
+ // `hasUnevaluableSubqueryFilter` below). That's safe ONLY because constructing `scanHelper` is a
+ // cheap case-class build with no file listing, and core's `CometScanExec.metrics` touches only
+ // `wrapped.driverMetrics` (populated by Spark's own planning) plus a static metric-node
+ // constructor -- neither file listing nor subquery resolution. If core's `metrics` ever touches
+ // either, forcing `scanHelper` here would resurrect the AQE mid-planning crashes this invariant
+ // prevents.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element, so checking runtimeFilters
+ // here avoids constructing/forcing the derived scanHelper just to read partitioning. The
+ // InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep in sync. The
+ // ScalarSubquery case is probed rather than treated as permanently unevaluable: Spark exposes no
+ // public finished/updated flag on ExecSubqueryExpression, but `eval()` doubles as one -- it only
+ // reads the cached `result` behind a `require(updated, ...)` guard, while the subquery is
+ // actually run by `updateResult()` (invoked separately during prepare/AQE), never by `eval()`.
+ // Once resolved, outputPartitioning below reports the real perPartitionData.length instead of
+ // staying at zero -- a fused native parent's buildNativeContext requires that count to match.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case s: ScalarSubquery => !isScalarSubqueryResolved(s)
+ case _ => false
+ })
+
+ // `eval()` never triggers the subquery's execution: on a resolved subquery it is a pure cached
+ // read of `result` (verified against bytecode: `Predef.require(updated(), ...)` then a plain
+ // field read), so this probe is safe to call repeatedly, including from AQE's mid-planning plan
+ // walks. Pre-resolution, the ONLY throw is `require`'s `IllegalArgumentException`; catch exactly
+ // that, since anything else escaping is a genuine bug we must not mask as unpartitioned.
+ private def isScalarSubqueryResolved(s: ScalarSubquery): Boolean =
+ try {
+ s.eval()
+ true
+ } catch {
+ case _: IllegalArgumentException => false
+ }
+
+ private def isAdaptivePlaceholder(p: SparkPlan): Boolean = p match {
+ case ReusedSubqueryExec(inner) => isAdaptivePlaceholder(inner)
+ case _: CometSubqueryAdaptiveBroadcastExec => true
+ case _: SubqueryAdaptiveBroadcastExec => true
+ case _ => false
+ }
+
+ override lazy val outputOrdering: Seq[SortOrder] = originalPlan.outputOrdering
+
+ override def dynamicPruningFilters: Seq[Expression] = runtimeFilters
+
+ override def withDynamicPruningFilters(filters: Seq[Expression]): SparkPlan = {
+ // A real copy: runtimeFilters is a constructor field included in equality, so the copy
+ // survives enclosing-block rebuilds, and the derived scanHelper picks up the rewritten
+ // filters automatically.
+ copy(runtimeFilters = filters)
+ }
+
+ /**
+ * Lazy split-mode serialization, mirroring CometNativeScanExec: common data was serialized at
+ * planning; per-partition file lists serialize here, at execution time.
+ */
+ @transient private lazy val serializedPartitionData
+ : (Array[Byte], Array[Array[Byte]], Array[Seq[String]]) = {
+ // Resolve the helper's DPP subqueries: it holds its own InSubqueryExec instances that
+ // Spark's expressions walk does not see (the helper is derived, not a child).
+ scanHelper.partitionFilters.foreach {
+ case DynamicPruningExpression(e: InSubqueryExec) if e.values().isEmpty =>
+ e.updateResult()
+ case _ =>
+ }
+
+ val commonBytes = {
+ val deltaScan = DeltaSparkScanEnvelope.unpack(nativeOp)
+ // Scalar subqueries in dataFilters were unresolved at planning; resolve them now and
+ // append them as pushed filters, as CometNativeScanExec.serializedPartitionData does.
+ val subqueryFilters = org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .resolvedSubqueryFilters(dataFilters, output, requiredSchema, conf)
+ val common = if (subqueryFilters.isEmpty) {
+ deltaScan.getCommon
+ } else {
+ val builder = deltaScan.getCommon.toBuilder
+ subqueryFilters.foreach(builder.addDataFilters)
+ builder.build()
+ }
+ OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaScan.getDeltaCommon)
+ .build()
+ .toByteArray
+ }
+
+ val filePartitions = scanHelper.getFilePartitions()
+
+ val tableRoot = DeltaSparkScanEnvelope.unpack(nativeOp).getDeltaCommon.getTableRoot
+ val perPartitionBytes = filePartitions.map { filePartition =>
+ org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .serializePartition(filePartition, originalPlan, tableRoot)
+ }.toArray
+
+ val perPartitionPaths = filePartitions.map(_.files.map(_.filePath.toString).toSeq).toArray
+
+ (commonBytes, perPartitionBytes, perPartitionPaths)
+ }
+
+ override def commonData: Array[Byte] = serializedPartitionData._1
+
+ override def perPartitionData: Array[Array[Byte]] = serializedPartitionData._2
+
+ def perPartitionFilePaths: Array[Seq[String]] = serializedPartitionData._3
+
+ override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+ val nativeMetrics = CometMetricNode.fromCometPlan(this)
+ val serializedPlan = CometExec.serializeNativePlan(nativeOp)
+
+ new CometExecRDD(
+ sparkContext,
+ Seq.empty,
+ Map(sourceKey -> commonData),
+ Map(sourceKey -> perPartitionData),
+ serializedPlan,
+ perPartitionData.length,
+ output.length,
+ nativeMetrics,
+ Seq.empty,
+ None,
+ Seq.empty,
+ perPartitionFilePaths = perPartitionFilePaths)
Review Comment:
**[P2] Report Delta scan input to Spark task metrics**
Could we register `reportScanInputMetrics` for this RDD and include contrib scans in the fused reporting decision? A fresh, uncached Delta-only read returns this bare `CometExecRDD`, unlike the existing native scans, so no callback copies SQL scan metrics into task input counters. Fused/native-shuffle execution also skips it because [hasScanInput](https://github.com/apache/datafusion-comet/blob/4ca3207af3853f8c6572098e14b53f79ab1d6fea/spark/src/main/scala/org/apache/spark/sql/comet/operators.scala#L828) recognizes only `CometNativeScanExec`.
Task/stage/event-log input bytes and records consequently remain zero even though SQL scan metrics are populated; results are unaffected. Please cover standalone and fused reads with a task-listener regression. Independently source-traced and challenged; a full Spark/Delta/JNI reproduction was blocked by Maven dependency access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-6112362577
@dwsmith1983 could you fix conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5351554528
Thanks @dwsmith1983. On the design topics you flagged, I’d prefer using `CometScanContrib` as the shared interface and agreeing on consistent configuration naming. The separate optional JAR sounds reasonable if it lets users try the feature without rebuilding Comet. We can discuss the packaging details separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4104619235
##########
native/core/src/parquet/parquet_exec.rs:
##########
@@ -84,6 +89,9 @@ pub(crate) fn init_datasource_exec(
encryption_enabled: bool,
use_field_id: bool,
ignore_missing_field_id: bool,
+ rebase_from_file_metadata: bool,
Review Comment:
Fixed in `281e02f01`, the merge with main that landed just after this review. `timestamp_errors.rs`, `schema_errors.rs` and `schema_errors/partition_columns.rs` now pass `false, "CORRECTED", "CORRECTED"`, the same values the neighbouring ordinary-scan fixtures use, and `cargo clippy --all-targets` and the core test build compile again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4166636744
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala:
##########
@@ -0,0 +1,1913 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import java.io.IOException
+import java.net.URI
+import java.util.Locale
+
+import scala.collection.mutable.{ListBuffer, Map => MutableMap}
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.catalyst.expressions.{Alias, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getExistenceDefaultValues
+import org.apache.spark.sql.comet.CometScanExec
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ProjectExec, SparkPlan}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType}
+
+import org.apache.comet.CometConf
+import org.apache.comet.CometConf.COMET_LIBHDFS_SCHEMES
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.parquet.CometParquetUtils
+import org.apache.comet.rules.{CometScanRule, CometScanTypeChecker}
+import org.apache.comet.serde.operator.CometNativeScan
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Claim/decline gates for the native Delta scan. Correctness rule: when in doubt, decline,
+ * Spark's Delta reader handles the scan and results stay correct, just unaccelerated.
+ */
+object DeltaScanSupport {
+
+ /**
+ * Reader features the native path understands; anything else on the protocol declines the
+ * table. `deletionVectors`/`columnMapping` are declined separately below for specific reasons.
+ */
+ private val understoodReaderFeatures: Set[String] =
+ Set("columnMapping", "deletionVectors", "timestampNtz", "v2Checkpoint", "vacuumProtocolCheck")
+
+ /**
+ * Is this exactly Delta's DSv1 parquet format? Compared by class name, not `classOf`: a
+ * `classOf` reference would raise `NoClassDefFoundError` and break every parquet scan when
+ * delta-spark is absent from the classpath.
+ */
+ def isDeltaScan(scanExec: FileSourceScanExec): Boolean =
+ scanExec.relation.fileFormat.getClass.getName ==
+ "org.apache.spark.sql.delta.DeltaParquetFileFormat"
+
+ /**
+ * Claim-time artifacts [[declineReason]] already computes but [[CometDeltaNativeScan.convert]]
+ * also needs -- threaded through by reference (populated only on the claimable path, right
+ * before `declineReason` returns `None`) so a claimed scan does not pay to recompute either:
+ * the Hadoop conf ([[org.apache.spark.sql.internal.SessionState#newHadoopConfWithOptions]] is
+ * not cheap) and the deletion-vector descriptors (base64-decoded, non-trivial only for DV-shape
+ * scans). One instance is created per claim attempt in `DeltaScanContrib` and passed to both
+ * `declineReason` and `convert`.
+ */
+ private[delta] final class DeltaClaimMemo {
+ var hadoopConf: Configuration = _
+ var dvDescriptors: Seq[DeletionVectorDescriptor] = Seq.empty
+ }
+
+ /**
+ * First reason this Delta scan cannot go native, or None when claimable (in which case `memo`
+ * is populated for [[CometDeltaNativeScan.convert]] to reuse). Only called when [[isDeltaScan]]
+ * is true. `scanHelper` is the [[CometScanExec]] built to drive `convert` on a claim, reused
+ * for the multi-store gate below.
+ */
+ def declineReason(
+ plan: SparkPlan,
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ memo: DeltaClaimMemo): Option[String] = {
+ val format = scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+ val protocol = format.protocol
+ val metadata = format.metadata
+ // Name mode is supported via physical-name schemas; id mode needs the field-id path and
+ // stays declined until validated. Hoisted here since several gates below reuse it.
+ val cmMode = metadata.columnMappingMode.name
+ // Descriptor deserialization is expensive, so hoist it into a `lazy val`, forced at most
+ // once in this method; on the claimable path the result is handed to `convert` through
+ // `memo` below, so a claimed scan deserializes the descriptors exactly once end to end.
+ val tableRoot = scanExec.relation.location.rootPaths.head.toString
+ lazy val dvDescriptors: Seq[DeletionVectorDescriptor] =
+ selectedDvDescriptors(scanHelper, tableRoot)
+
+ // Mirrors core's CometScanRule.isSchemaSupported so scan-time type gates (unsigned-small-int
+ // fallback, collation, shredded-variant-struct) apply identically here. Pure in-memory check,
+ // so it runs first, ahead of every I/O-bearing gate below.
+ // Unlike core, a required Variant root stays declined: this path lacks core's Variant gates.
+ val schemaFallbackReasons = new ListBuffer[String]()
+ val typeChecker = CometScanTypeChecker()
+ val requiredSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.requiredSchema, schemaFallbackReasons)
+ val partitionSchemaSupported =
+ typeChecker.isSchemaSupported(scanExec.relation.partitionSchema, schemaFallbackReasons)
+ if (!requiredSchemaSupported || !partitionSchemaSupported) {
+ return Some(
+ "Native Delta scan does not support the schema: " + schemaFallbackReasons.mkString(", "))
+ }
+
+ if (format.isCDCRead) {
+ return Some("Native Delta scan does not support Change Data Feed reads")
+ }
+
+ // Delta's DML machinery (findTouchedFiles) disables reader optimizations and needs real
+ // row indexes from Spark's reader; claiming here would feed NULL indexes into DV construction.
+ if (!format.optimizationsEnabled) {
+ return Some("Native Delta scan does not support reads with reader optimizations disabled")
+ }
+ if (scanExec.requiredSchema.exists(_.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME) ||
+ scanExec.relation.dataSchema.exists(
+ _.name == DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME)) {
+ return Some("Native Delta scan does not support Delta's generated row-index column")
+ }
+
+ if (cmMode != "none" && cmMode != "name") {
+ return Some(s"Native Delta scan does not support column mapping mode $cmMode")
+ }
+ // createPhysicalSchema wholesale-replaces field metadata, silently dropping EXISTS_DEFAULT.
+ if (cmMode == "name" &&
+ getExistenceDefaultValues(scanExec.requiredSchema).exists(_ != null)) {
+ return Some(
+ "Native Delta scan does not support column defaults together with column mapping")
+ }
+ // createPhysicalSchema rewrites nested StructField names too, and the native builder emits the
+ // required schema verbatim as output, so name-sensitive expressions (e.g. to_json) would leak
+ // physical names. Decline until a rename adapter exists.
+ if (cmMode == "name" &&
+ scanExec.requiredSchema.exists(f => containsNestedStruct(f.dataType))) {
+ return Some("Native Delta scan does not support column mapping with nested struct fields")
+ }
+
+ val readerFeatures = protocol.readerFeatureNames
+ val unknownFeatures = readerFeatures -- understoodReaderFeatures
+ if (unknownFeatures.nonEmpty) {
+ return Some(
+ s"Native Delta scan does not support reader feature(s) ${unknownFeatures.mkString(", ")}")
+ }
+
+ // Non-constant metadata columns are generated per-row by Spark's reader and unsupported,
+ // except Delta's DV bookkeeping columns, which the native path emits as constants.
+ val knownColNames =
+ scanExec.relation.dataSchema.map(_.name).toSet ++
+ scanExec.relation.partitionSchema.map(_.name).toSet ++
+ scanExec.fileConstantMetadataColumns.map(_.name).toSet ++
Review Comment:
Fixed in 9951923b9. The Delta scan now declines when `_metadata.file_block_start` or `file_block_length` is used above the scan, the same fallback core has had since #6510. A deletion-vector read carries the whole `_metadata` struct, so the check looks at what the plan consumes rather than what the scan outputs; otherwise every DV read would fall back. The test reads a single-file 5,000-row table with `maxPartitionBytes=4096` through a projection, a filter on `file_block_start`, and `select id, _metadata`, and covers the DV case too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4171441190
##########
native/core/src/parquet/datetime_rebase.rs:
##########
@@ -0,0 +1,3407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Per-file datetime calendar-rebase handling for the parquet scan.
+//!
+//! Spark 2.4 and earlier wrote dates and timestamps in the hybrid Julian + Gregorian calendar;
+//! Spark 3.0+ uses the proleptic Gregorian calendar and records the calendar policy of every
+//! file it writes in the parquet footer's key-value metadata (`org.apache.spark.version`,
+//! `org.apache.spark.legacyDateTime`, `org.apache.spark.legacyINT96`,
+//! `org.apache.spark.timeZone`). Spark's reader resolves the rebase policy from EACH FILE's
+//! writer metadata (`DataSourceUtils.datetimeRebaseSpec` / `int96RebaseSpec`) -- the session's
+//! `spark.sql.parquet.datetimeRebaseModeInRead` conf only applies to files whose metadata does
+//! not decide the policy on its own -- so a reader that ignores the metadata silently returns
+//! values shifted by up to ten days for dates before 1582-10-15 (e.g. `1500-01-01` reads as
+//! `1500-01-10`).
+//!
+//! This module mirrors that per-file resolution: [`resolve_file_rebase_policies`] computes the
+//! date / INT64-timestamp / INT96-timestamp policies from a file's arrow schema metadata (the
+//! parquet key-value pairs survive the parquet -> arrow schema conversion), and
+//! [`wrap_datetime_rebase`] wraps the per-file rewritten expressions' column references in a
+//! [`SparkDatetimeRebaseExpr`] that rebases values exactly where that is possible without the
+//! JVM's historical timezone tables (dates always; timestamps for a fixed UTC writer zone) and
+//! refuses -- rather than silently corrupting -- ancient values it cannot rebase. Nested
+//! columns are rebuilt leaf by leaf (struct / list / map / fixed-size list / dictionary), each
+//! leaf under its own policy, with nulls and offsets preserved. Modern values are always the
+//! identity under every policy: from 1582-10-15 onward for dates, and from
+//! [`LAST_SWITCH_JULIAN_TS_SECONDS`] (1900-01-01T00:00:00Z, Spark's
+//! `RebaseDateTime.lastSwitchJulianTs`) onward for timestamps.
+//!
+//! Spark applies `datetimeRebaseSpec` to INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns
+//! and `int96RebaseSpec` to INT96 columns. The two physical types are indistinguishable in the
+//! arrow schema DataFusion hands the expression adapter (both surface as `Timestamp(us, "UTC")`
+//! after INT96 coercion), so Comet's parquet reader factory stamps the file's INT96 leaf
+//! ordinals -- taken from the parquet footer's own `SchemaDescriptor` -- into the key-value
+//! metadata under [`INT96_LEAVES_METADATA_KEY`] before the arrow schema is derived (see
+//! [`stamp_int96_leaves`] and `eager_page_index_reader_factory.rs`), and the adapter attributes
+//! every timestamp leaf to its spec from that stamp. Without a stamp, the two specs are merged:
+//! agreement decides, disagreement degrades to [`RebasePolicy::CheckAncient`].
+//!
+//! The wrapper sits BENEATH the schema adapter's nested narrowing (the struct -> struct convert
+//! that keeps only the requested children), which is what keeps those ordinals physical -- but
+//! it means the wrapper sees every physical child, requested or not. Spark only ever decodes
+//! the requested nested schema, so [`FileRebasePolicies::restrict_to_requested`] marks the
+//! physical leaves the narrowing drops as the identity: an unrequested ancient `s.ts` never
+//! blocks `select s.d`, exactly as in Spark.
+//!
+//! The same pairing decides a timestamp leaf's policy by the type the query READS it as, since
+//! `ParquetVectorUpdaterFactory.getUpdater` keys on the requested Spark type, not the parquet
+//! annotation: a leaf read as `TIMESTAMP_NTZ` never rebases (INT96 or INT64; Spark 4.x's
+//! `BinaryToSQLTimestampUpdater` / `LongUpdater` consult no mode, and Spark 3.x refuses the
+//! INT96 and adjusted-INT64 pairings outright, which Comet's `allow_timestamp_ltz_to_ntz` gate
+//! reproduces); a leaf read as `TIMESTAMP` rebases under the datetime spec even when the file
+//! declares it `isAdjustedToUTC=false` (`isTimestampTypeMatched` checks the unit only); and a
+//! `DATE` leaf keeps the date policy whether read as `DATE` or, on Spark 4.x, as
+//! `TIMESTAMP_NTZ` (`DateToTimestampNTZWithRebaseUpdater`). A date or timestamp leaf read as a
+//! number (a `DATE` as `INT`, a `TIMESTAMP` of any unit as `BIGINT`, e.g. NANOS under
+//! `spark.sql.legacy.parquet.nanosAsLong`) never rebases: the numeric updaters come first in
+//! `getUpdater` and return the stored value.
+//!
+//! Currently only enabled by the Delta scan arms via
+//! `SparkParquetOptions::rebase_from_file_metadata`, which also carries the session read modes
+//! ([`SessionRebaseModes`], forwarded from the JVM) that decide the policy for files without
+//! Spark writer metadata; the plain NativeScan keeps its documented no-rebase behavior (see
+//! the compatibility guide and issue #5010).
+
+use std::collections::HashMap;
+use std::fmt::{self, Display};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Date32Array, FixedSizeListArray, GenericListArray, MapArray,
+ OffsetSizeTrait, PrimitiveArray, RecordBatch, StructArray,
+};
+use arrow::datatypes::{
+ ArrowPrimitiveType, ArrowTimestampType, DataType, Date32Type, FieldRef, Schema, SchemaRef,
+ TimeUnit, TimestampMicrosecondType, TimestampMillisecondType, TimestampNanosecondType,
+ TimestampSecondType,
+};
+use arrow::error::ArrowError;
+use datafusion::common::tree_node::{Transformed, TreeNode};
+use datafusion::common::{DataFusionError, Result as DataFusionResult};
+use datafusion::physical_expr::expressions::Column;
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::ColumnarValue;
+use parquet::basic::Type as ParquetPhysicalType;
+use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaData};
+use parquet::schema::types::SchemaDescriptor;
+
+use super::name_fold::fold_names;
+use super::parquet_support::field_id;
+
+/// Footer key naming the Spark release that wrote the file; absent for non-Spark writers.
+const SPARK_VERSION_METADATA_KEY: &str = "org.apache.spark.version";
+/// Present (empty value) when the file's dates and INT64 timestamps were written with
+/// `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_DATETIME_KEY: &str = "org.apache.spark.legacyDateTime";
+/// Present (empty value) when the file's INT96 timestamps were written with
+/// `spark.sql.parquet.int96RebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_INT96_KEY: &str = "org.apache.spark.legacyINT96";
+/// The writer session's time zone, stamped alongside either legacy flag.
+const SPARK_TIMEZONE_KEY: &str = "org.apache.spark.timeZone";
+
+/// Key-value metadata entry Comet's parquet reader factory adds to a file's footer metadata
+/// (in memory only, never written back) so the expression adapter can tell INT96 timestamp
+/// columns from INT64 ones after both have been coerced to the same arrow type. Value:
+/// `"<leaf count>:<comma-separated INT96 leaf ordinals>"`, where leaves are the file's
+/// primitive columns in `SchemaDescriptor::columns()` order -- the same depth-first order
+/// parquet-rs assigns arrow leaves, so an arrow-side depth-first walk lines up with it. The
+/// leaf count lets the reader detect a stamp that does not describe the schema it is paired
+/// with (see [`Int96Attribution::from_schema`]).
+pub(crate) const INT96_LEAVES_METADATA_KEY: &str = "comet.int96_leaf_columns";
+
+/// Day of the Gregorian cutover (1582-10-15) as days since the epoch; rebasing is the identity
+/// from this day onward. Same value as Spark's `RebaseDateTime.lastSwitchJulianDay`.
+const LAST_SWITCH_JULIAN_DAY: i32 = -141427;
+
+/// Spark's `RebaseDateTime.lastSwitchJulianTs` (and `lastSwitchGregorianTs`) in seconds since
+/// the epoch: 1900-01-01T00:00:00Z. Spark derives it as the latest switch instant across every
+/// zone in its `julian-gregorian-rebase-micros.json` table (`getLastSwitchTs`, which also
+/// asserts the calendars' difference is zero for every zone from then on): most zones ran on
+/// local mean time before 1900, so the last instant at which rebasing changes a value in ANY
+/// zone is 1900-01-01T00:00:00Z, not the 1582 cutover. `createTimestampRebaseFuncInRead`
+/// under `EXCEPTION` throws exactly for `micros < lastSwitchJulianTs` (after converting
+/// `TIMESTAMP_MILLIS` to micros), and `rebaseJulianToGregorianMicros` is the identity from it
+/// onward in every zone. The value is in seconds so it scales exactly to any timestamp unit.
+pub(crate) const LAST_SWITCH_JULIAN_TS_SECONDS: i64 = -2_208_988_800;
+
+/// The per-century differences between the Julian and proleptic Gregorian calendars, and the
+/// Julian-calendar switch days at which each difference starts to apply. Copied verbatim from
+/// Spark's `RebaseDateTime.julianGregDiffs` / `julianGregDiffSwitchDay` (which Spark generated
+/// from `localRebaseJulianToGregorianDays`); `rebase_julian_to_gregorian_days` must stay
+/// value-for-value equal to Spark's `rebaseJulianToGregorianDays`.
+const JULIAN_GREG_DIFFS: [i32; 14] = [2, 1, 0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, 0];
+const JULIAN_GREG_DIFF_SWITCH_DAY: [i32; 14] = [
+ -719164, -682945, -646420, -609895, -536845, -500320, -463795, -390745, -354220, -317695,
+ -244645, -208120, -171595, -141427,
+];
+
+/// Proleptic-Gregorian days since 1970-01-01 for a nominal civil date, via Howard Hinnant's
+/// `days_from_civil`. `d` may exceed the month's length; the excess rolls into the following
+/// month exactly like `LocalDate.of(y, m, 1).plusDays(d - 1)` in Spark's
+/// `localRebaseJulianToGregorianDays` (how the non-existent proleptic date `1000-02-29`,
+/// valid in the Julian calendar, lands on `1000-03-01`).
+fn days_from_civil(y: i64, m: i64, d: i64) -> i64 {
+ let y = if m <= 2 { y - 1 } else { y };
+ let era = y.div_euclid(400);
+ let yoe = y - era * 400; // [0, 399]
+ let mp = (m + 9) % 12; // [0, 11], March = 0
+ let doy = (153 * mp + 2) / 5 + d - 1;
+ let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
+ era * 146097 + doe - 719468
+}
+
+/// Julian-calendar civil date `(year, month, day)` for a day count since 1970-01-01 that labels
+/// days in the Julian calendar (astronomical year numbering: 1 BCE is year 0). Standard
+/// Julian-day-number conversion (E.G. Richards' algorithm), exact for any day.
+fn julian_day_to_civil(days: i64) -> (i64, i64, i64) {
+ // Integer (noon) Julian Day Number of this civil day: 1970-01-01 is JDN 2440588.
+ let jdn = days + 2_440_588;
+ let f = jdn + 1401;
+ let e = 4 * f + 3;
+ let g = e.rem_euclid(1461) / 4;
+ let h = 5 * g + 2;
+ let day = h.rem_euclid(153) / 5 + 1;
+ let month = (h / 153 + 2).rem_euclid(12) + 1;
+ let year = e.div_euclid(1461) - 4716 + (14 - month) / 12;
+ (year, month, day)
+}
+
+/// Exact port of Spark's `RebaseDateTime.rebaseJulianToGregorianDays`: reinterprets a day count
+/// written in the hybrid Julian + Gregorian calendar as the proleptic Gregorian day count of the
+/// same nominal civil date. Identity for days from 1582-10-15 onward. Days before the tables'
+/// range (before Julian `0001-01-01`) take the calendar-arithmetic path, mirroring Spark's
+/// `localRebaseJulianToGregorianDays` fallback.
+pub(crate) fn rebase_julian_to_gregorian_days(days: i32) -> i32 {
+ if days < JULIAN_GREG_DIFF_SWITCH_DAY[0] {
+ let (y, m, d) = julian_day_to_civil(days as i64);
+ (days_from_civil(y, m, 1) + (d - 1)) as i32
+ } else {
+ // Spark's rebaseDays: linear search from the most recent switch day.
+ let mut i = JULIAN_GREG_DIFF_SWITCH_DAY.len();
+ loop {
+ i -= 1;
+ if i == 0 || days >= JULIAN_GREG_DIFF_SWITCH_DAY[i] {
+ break;
+ }
+ }
+ days + JULIAN_GREG_DIFFS[i]
+ }
+}
+
+/// Timezone strings from `org.apache.spark.timeZone` that denote a fixed zero-offset zone in
+/// both `java.util.TimeZone` and `java.time`. Only for these is timestamp rebasing the pure
+/// nominal-date shift [`SparkDatetimeRebaseExpr::rebase_timestamp_utc`] computes; any other (or
+/// absent) zone needs the JVM's historical timezone tables and stays on the
+/// refuse-ancient-values path.
+const UTC_EQUIVALENT_TIMEZONES: [&str; 6] = ["UTC", "Etc/UTC", "GMT", "Etc/GMT", "Z", "+00:00"];
+
+/// How the writer's session time zone (if recorded) affects timestamp rebasing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum WriterTimeZone {
+ /// A fixed zero-offset zone: rebasing reduces to the exact nominal-date shift.
+ Utc,
+ /// Any other zone, or none recorded (pre-3.0 files): ancient values cannot be rebased
+ /// without the JVM's historical timezone data.
+ OtherOrUnknown,
+}
+
+/// One session-level datetime rebase read mode (a `LegacyBehaviorPolicy` value of
+/// `spark.sql.parquet.datetimeRebaseModeInRead` / `int96RebaseModeInRead`), consulted by
+/// [`resolve_file_rebase_policies`] ONLY for files whose footer metadata does not decide the
+/// policy on its own -- exactly the `getOrElse` fallback in Spark's
+/// `DataSourceUtils.getRebaseSpec`. Files that carry `org.apache.spark.version` ignore these
+/// modes entirely, on every Spark version.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) enum RebaseReadMode {
+ /// Refuse ancient values (Spark raises `SparkUpgradeException`); maps to
+ /// [`RebasePolicy::CheckAncient`]. The default mirrors the conservative posture used
+ /// before the conf was plumbed through (and Spark 3.x's own conf default).
+ #[default]
+ Exception,
+ /// Read values as proleptic Gregorian without rebasing.
+ Corrected,
+ /// Rebase from the hybrid Julian + Gregorian calendar.
+ Legacy,
+}
+
+impl RebaseReadMode {
+ /// Parses a `LegacyBehaviorPolicy` conf value. `SQLConf` validates and upper-cases the
+ /// session conf, but a per-relation `datetimeRebaseMode` option arrives verbatim, so the
+ /// match is case-insensitive. Anything unrecognized -- including the empty string a proto
+ /// producer that predates the field sends -- falls back to [`RebaseReadMode::Exception`],
+ /// which refuses ancient values rather than silently corrupting them.
+ pub(crate) fn from_conf_value(value: &str) -> Self {
+ match value.to_ascii_uppercase().as_str() {
+ "CORRECTED" => RebaseReadMode::Corrected,
+ "LEGACY" => RebaseReadMode::Legacy,
+ _ => RebaseReadMode::Exception,
+ }
+ }
+}
+
+/// The session's effective datetime rebase read modes, one per spec class (INT64
+/// dates/timestamps vs INT96 timestamps), forwarded from the JVM at planning time.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) struct SessionRebaseModes {
+ /// `spark.sql.parquet.datetimeRebaseModeInRead` (or the relation's `datetimeRebaseMode`).
+ pub datetime: RebaseReadMode,
+ /// `spark.sql.parquet.int96RebaseModeInRead` (or the relation's `int96RebaseMode`).
+ pub int96: RebaseReadMode,
+}
+
+/// Calendar policy of one file's date or timestamp columns, resolved from writer metadata the
+/// same way Spark's `DataSourceUtils.getRebaseSpec` resolves it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum RebasePolicy {
+ /// Written in the proleptic Gregorian calendar; values pass through untouched.
+ Corrected,
+ /// Written in the hybrid Julian + Gregorian calendar; values must be rebased.
+ Legacy(WriterTimeZone),
+ /// Policy could not be pinned down (contradictory flags, or a non-Spark writer under the
+ /// `EXCEPTION` read mode): modern values -- identical under either calendar -- pass,
+ /// ancient values raise. Mirrors Spark's `EXCEPTION` behavior (`SparkUpgradeException`).
+ CheckAncient,
+}
+
+/// Which of a file's leaf columns are physically INT96, from the stamp the parquet reader
+/// factory adds under [`INT96_LEAVES_METADATA_KEY`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) enum Int96Attribution {
+ /// No stamp, or a stamp whose leaf count does not match the schema it arrived with: the
+ /// INT64 and INT96 timestamp specs cannot be told apart per column and are merged.
+ Unknown,
+ /// Sorted leaf ordinals (depth-first over the file schema's primitive columns) that are
+ /// INT96; every other timestamp leaf is INT64.
+ Known(Vec<usize>),
+}
+
+impl Int96Attribution {
+ /// Parses the stamp out of `schema`'s metadata and validates its leaf count against the
+ /// schema's own depth-first leaf count, so a stamp that does not describe this schema (a
+ /// crafted footer key, or a cached-metadata mismatch) degrades to [`Self::Unknown`].
+ fn from_schema(schema: &Schema) -> Self {
+ let Some(stamp) = schema.metadata().get(INT96_LEAVES_METADATA_KEY) else {
+ return Int96Attribution::Unknown;
+ };
+ let Some((count, ordinals)) = stamp.split_once(':') else {
+ return Int96Attribution::Unknown;
+ };
+ let schema_leaves: usize = schema
+ .fields()
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ if count.parse::<usize>().ok() != Some(schema_leaves) {
+ return Int96Attribution::Unknown;
+ }
+ let parsed: Option<Vec<usize>> = if ordinals.is_empty() {
+ Some(Vec::new())
+ } else {
+ ordinals
+ .split(',')
+ .map(|o| o.parse::<usize>().ok().filter(|o| *o < schema_leaves))
+ .collect()
+ };
+ match parsed {
+ Some(mut leaves) => {
+ leaves.sort_unstable();
+ Int96Attribution::Known(leaves)
+ }
+ None => Int96Attribution::Unknown,
+ }
+ }
+
+ /// `Some(true)` / `Some(false)` when the leaf is known to be INT96 / INT64, `None` when
+ /// the attribution is unknown.
+ fn is_int96(&self, leaf: usize) -> Option<bool> {
+ match self {
+ Int96Attribution::Unknown => None,
+ Int96Attribution::Known(leaves) => Some(leaves.binary_search(&leaf).is_ok()),
+ }
+ }
+}
+
+/// The [`INT96_LEAVES_METADATA_KEY`] value describing `schema`: its leaf count and the
+/// ordinals of its INT96 primitive columns.
+pub(crate) fn int96_leaf_stamp(schema: &SchemaDescriptor) -> String {
+ let ordinals: Vec<String> = schema
+ .columns()
+ .iter()
+ .enumerate()
+ .filter(|(_, column)| column.physical_type() == ParquetPhysicalType::INT96)
+ .map(|(ordinal, _)| ordinal.to_string())
+ .collect();
+ format!("{}:{}", schema.num_columns(), ordinals.join(","))
+}
+
+/// Returns a copy of `metadata` whose key-value metadata carries the [`int96_leaf_stamp`] of
+/// its own schema, or `None` when it already does (the common case after the first open of a
+/// file, since the caller caches the stamped copy). Any pre-existing entry under the key --
+/// a file cannot legitimately carry one -- is replaced, never trusted. Only the file-level
+/// key-value list changes; row groups and page indexes are carried over as-is. The parquet
+/// API cannot carry a file decryptor, nor `FileMetaData`'s crate-private encryption fields
+/// (encryption algorithm, footer signing key metadata), across this rebuild, so callers must
+/// not stamp opens that supply decryption properties -- and the only consumer, the Delta
+/// scan, declines every encrypted-parquet configuration before planning, so a parquet
+/// modular encryption file never reaches this path with or without those properties.
+pub(crate) fn stamp_int96_leaves(metadata: &ParquetMetaData) -> Option<ParquetMetaData> {
+ let file_metadata = metadata.file_metadata();
+ let stamp = int96_leaf_stamp(file_metadata.schema_descr());
+ let existing = file_metadata
+ .key_value_metadata()
+ .and_then(|kvs| kvs.iter().find(|kv| kv.key == INT96_LEAVES_METADATA_KEY))
+ .and_then(|kv| kv.value.as_deref());
+ if existing == Some(stamp.as_str()) {
+ return None;
+ }
+ let mut key_values: Vec<KeyValue> = file_metadata
+ .key_value_metadata()
+ .map(|kvs| {
+ kvs.iter()
+ .filter(|kv| kv.key != INT96_LEAVES_METADATA_KEY)
+ .cloned()
+ .collect()
+ })
+ .unwrap_or_default();
+ key_values.push(KeyValue::new(INT96_LEAVES_METADATA_KEY.to_string(), stamp));
+ let stamped_file_metadata = FileMetaData::new(
+ file_metadata.version(),
+ file_metadata.num_rows(),
+ file_metadata.created_by().map(str::to_string),
+ Some(key_values),
+ file_metadata.schema_descr_ptr(),
+ file_metadata.column_orders().cloned(),
+ );
+ Some(
+ ParquetMetaData::new(stamped_file_metadata, metadata.row_groups().to_vec())
+ .into_builder()
+ .set_column_index(metadata.column_index().cloned())
+ .set_offset_index(metadata.offset_index().cloned())
+ .build(),
+ )
+}
+
+/// Per-file rebase policies for the three affected column classes, plus the INT96
+/// attribution that selects between the two timestamp specs per leaf.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct FileRebasePolicies {
+ /// `DATE` columns, governed by `org.apache.spark.legacyDateTime` alone.
+ pub date: RebasePolicy,
+ /// INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns, adjusted to UTC or not: the
+ /// datetime spec (same resolution as `date`), as Spark's `ParquetVectorUpdaterFactory`
+ /// selects for INT64 read as `TIMESTAMP`.
+ pub int64_timestamp: RebasePolicy,
+ /// INT96 columns: the INT96 spec (`org.apache.spark.legacyINT96`, min version 3.1.0).
+ pub int96_timestamp: RebasePolicy,
+ /// Which timestamp leaves are INT96. See [`Int96Attribution`].
+ pub int96_leaves: Int96Attribution,
+ /// Sorted depth-first leaf ordinals -- over the physical file schema, the same ordinals
+ /// `int96_leaves` uses -- that the query does not read: nested children the schema
+ /// adapter's struct narrowing drops before any value leaves the scan. Spark never decodes
+ /// them either, so their policy is the identity whatever the file's calendar. Empty until
+ /// [`Self::restrict_to_requested`] runs (every leaf requested).
+ pub unrequested_leaves: Vec<usize>,
+ /// Sorted physical leaf ordinals of timezone-carrying timestamps (INT96, or INT64 with
+ /// `isAdjustedToUTC=true`) the query reads as `TIMESTAMP_NTZ`. Spark decodes those with
+ /// `BinaryToSQLTimestampUpdater` / `LongUpdater`, which never rebase, so their policy is
+ /// the identity whatever the file's calendar. Filled by [`Self::restrict_to_requested`].
+ pub ntz_requested_leaves: Vec<usize>,
+ /// Sorted physical leaf ordinals of timezone-free INT64 timestamps
+ /// (`isAdjustedToUTC=false`) the query reads as `TIMESTAMP`. Spark's INT64 branch checks
+ /// only the unit (`isTimestampTypeMatched`) and hands a `TimestampType` request to
+ /// `LongWithRebaseUpdater` under the datetime spec, adjusted or not, so these leaves take
+ /// the same policy as adjusted INT64 leaves. Filled by [`Self::restrict_to_requested`].
+ pub ltz_requested_leaves: Vec<usize>,
+ /// Sorted physical leaf ordinals of dates and timestamps the query reads as a number
+ /// (`INT`, `BIGINT`, ...). Spark's `ParquetVectorUpdaterFactory` picks the numeric updaters
+ /// before any date or timestamp arm, and none of them rebase, so their policy is the
+ /// identity whatever the file's calendar. Filled by [`Self::restrict_to_requested`].
+ pub numeric_requested_leaves: Vec<usize>,
+}
+
+/// The leaf ordinals [`push_unrequested_leaves`] records while pairing a physical type with
+/// the type the query reads it as. Each list is emitted in depth-first order, so it is already
+/// sorted for the binary searches in [`FileRebasePolicies`].
+#[derive(Debug, Default)]
+struct LeafPairing {
+ unrequested: Vec<usize>,
+ ntz_requested: Vec<usize>,
+ ltz_requested: Vec<usize>,
+ numeric_requested: Vec<usize>,
+}
+
+impl FileRebasePolicies {
+ /// True when some policy is not the plain proleptic-Gregorian pass-through, i.e. when the
+ /// per-column wrap in [`wrap_datetime_rebase`] can install anything at all.
+ pub(crate) fn any_rebase_needed(&self) -> bool {
+ self.date != RebasePolicy::Corrected
+ || self.int64_timestamp != RebasePolicy::Corrected
+ || self.int96_timestamp != RebasePolicy::Corrected
+ }
+
+ /// True when the query reads the leaf as a date or timestamp: it is requested, and not
+ /// as a number.
+ fn is_requested(&self, leaf: usize) -> bool {
+ self.unrequested_leaves.binary_search(&leaf).is_err()
+ && self.numeric_requested_leaves.binary_search(&leaf).is_err()
+ }
+
+ /// The policy of the `Date32` leaf at depth-first ordinal `leaf`: the file's date policy,
+ /// or the identity when the query does not read that leaf.
+ fn date_policy(&self, leaf: usize) -> RebasePolicy {
+ if self.is_requested(leaf) {
+ self.date
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+
+ /// The policy of the timezone-carrying timestamp leaf at depth-first ordinal `leaf`: the
+ /// identity when the query does not read it or reads it as `TIMESTAMP_NTZ` (Spark's NTZ
+ /// updaters never rebase; on Spark 3.x the pairing is refused before any rebase decision,
+ /// which Comet's `allow_timestamp_ltz_to_ntz` gate reproduces); otherwise its physical
+ /// type's spec when the attribution is known, or else the two specs merged -- agreement
+ /// decides, disagreement degrades to [`RebasePolicy::CheckAncient`], which still passes
+ /// every modern value and refuses only ancient ones.
+ fn timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) || self.ntz_requested_leaves.binary_search(&leaf).is_ok() {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ Some(false) => self.int64_timestamp,
+ None if self.int64_timestamp == self.int96_timestamp => self.int64_timestamp,
+ None => RebasePolicy::CheckAncient,
+ }
+ }
+
+ /// The policy of the timezone-free timestamp leaf at depth-first ordinal `leaf` (INT64 with
+ /// `isAdjustedToUTC=false`): the identity unless the query reads it as `TIMESTAMP`, which
+ /// Spark decodes with `LongWithRebaseUpdater` under the datetime spec exactly like an
+ /// adjusted INT64 leaf. The stamp is still consulted so a leaf it names INT96 follows the
+ /// INT96 spec; without a stamp the physical type itself proves INT64, so the two specs are
+ /// not merged.
+ fn tz_free_timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) || self.ltz_requested_leaves.binary_search(&leaf).is_err() {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ _ => self.int64_timestamp,
+ }
+ }
+
+ /// These policies with every physical leaf the query does not read marked the identity,
+ /// and every timestamp leaf whose requested type differs from its physical one in timezone
+ /// presence recorded, so [`leaf_policies`] can pick the policy Spark's
+ /// `ParquetVectorUpdaterFactory.getUpdater` picks for the REQUESTED type.
+ /// `requested` pairs each top-level field of `physical_schema` (by position) with the type
+ /// of the logical field the schema adapter narrows it to -- `None` for a column without a
+ /// logical counterpart, whose leaves are left as they are (no expression reads it anyway).
+ /// Nested children pair the way the adapter's struct convert selects them (see
+ /// [`push_unrequested_leaves`]); the INT96 attribution is untouched, since the ordinals
+ /// stay physical. `requested` is parallel to the schema's fields; should a caller pass a
+ /// shorter slice, the trailing columns simply keep every leaf (the safe direction).
+ pub(crate) fn restrict_to_requested(
+ mut self,
+ physical_schema: &Schema,
+ requested: &[Option<&DataType>],
+ case_sensitive: bool,
+ use_field_id: bool,
+ ) -> Self {
+ debug_assert_eq!(requested.len(), physical_schema.fields().len());
+ let matching = FieldMatching {
+ case_sensitive,
+ use_field_id,
+ };
+ let mut next_leaf = 0;
+ let mut pairing = LeafPairing::default();
+ for (field, requested) in physical_schema.fields().iter().zip(requested) {
+ match requested {
+ Some(logical) => push_unrequested_leaves(
+ field.data_type(),
+ logical,
+ &mut next_leaf,
+ matching,
+ &mut pairing,
+ ),
+ None => next_leaf += leaf_count(field.data_type()),
+ }
+ }
+ // Emitted in depth-first order, so already sorted for the binary searches.
+ self.unrequested_leaves = pairing.unrequested;
+ self.ntz_requested_leaves = pairing.ntz_requested;
+ self.ltz_requested_leaves = pairing.ltz_requested;
+ self.numeric_requested_leaves = pairing.numeric_requested;
+ self
+ }
+}
+
+/// The field-matching rules of the schema adapter's nested narrowing
+/// (`parquet_convert_struct_to_struct`): names fold per `case_sensitive`, and Parquet field ids
+/// select fields when `use_field_id` is set.
+#[derive(Debug, Clone, Copy)]
+struct FieldMatching {
+ case_sensitive: bool,
+ use_field_id: bool,
+}
+
+/// Appends to `out.unrequested` the depth-first leaf ordinals of `physical` (counting from
+/// `next_leaf`, which advances past every leaf of `physical`) that reading it as `requested`
+/// drops, and records in `out.ntz_requested` / `out.ltz_requested` the timestamp leaves whose
+/// requested type has the opposite timezone presence (a timezone-carrying leaf read as
+/// `TIMESTAMP_NTZ`, a timezone-free leaf read as `TIMESTAMP`); the unit is irrelevant to
+/// either, as it is to Spark's `isTimestampTypeMatched`. A date leaf read as `TINYINT`,
+/// `SMALLINT`, `INT`, `BIGINT` or `DOUBLE`, and a timestamp leaf read as `BIGINT`, go to
+/// `out.numeric_requested`: Spark's INT32 and INT64 numeric updaters for those pairings never
+/// rebase. Spark accepts a date as `BIGINT` or `DOUBLE` only on 4.x (3.x refuses the read);
+/// the identity is still the right policy for it.
+///
+/// Recurses through exactly the pairings `parquet_convert_array` narrows, and no others: a
+/// struct child is dropped only when NO requested child selects it by either rule the struct
+/// convert uses -- folded name, or Parquet field id when ids are in play -- and an ambiguous
+/// child (several requested children select it) is kept; `List` pairs with `List` by element
+/// type, and `Map` with a `Map` of the same key ordering by its entries, positionally. Any
+/// other pairing -- a `LargeList` / `FixedSizeList` / dictionary, a map whose ordering
+/// differs, or a shape mismatch -- is handed to arrow's cast or passed through whole by the
+/// convert, so it keeps every leaf under its physical type's policy. Keeping a superset of
+/// what the narrowing reads is always safe (a spurious check at worst); dropping a leaf the
+/// narrowing reads would skip its rebase, so every doubt resolves to "requested". Timestamp
+/// leaves inside those pass-through shapes are never recorded either, so they keep the
+/// physical rule (a spurious check for an NTZ request, no rebase for a `TIMESTAMP` request of
+/// a timezone-free leaf); Spark's requested schemas never take those arrow shapes.
+fn push_unrequested_leaves(
+ physical: &DataType,
+ requested: &DataType,
+ next_leaf: &mut usize,
+ matching: FieldMatching,
+ out: &mut LeafPairing,
+) {
+ match (physical, requested) {
+ (DataType::Timestamp(_, Some(_)), DataType::Timestamp(_, None)) => {
+ out.ntz_requested.push(*next_leaf);
+ *next_leaf += 1;
+ }
+ (DataType::Timestamp(_, None), DataType::Timestamp(_, Some(_))) => {
+ out.ltz_requested.push(*next_leaf);
+ *next_leaf += 1;
+ }
+ (
+ DataType::Date32,
+ DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Int64
+ | DataType::Float64,
+ )
+ | (DataType::Timestamp(_, _), DataType::Int64) => {
+ out.numeric_requested.push(*next_leaf);
+ *next_leaf += 1;
+ }
+ (DataType::Struct(physical_fields), DataType::Struct(requested_fields)) => {
+ let names: Vec<&str> = physical_fields
+ .iter()
+ .chain(requested_fields.iter())
+ .map(|f| f.name().as_str())
+ .collect();
+ // A fold failure means the names could not be compared at all; keeping every leaf
+ // requested is the safe superset, the same as the pass-through pairings below.
+ let Ok(folded) = fold_names(&names, matching.case_sensitive) else {
+ *next_leaf += leaf_count(physical);
+ return;
+ };
+ let (physical_folded, requested_folded) = folded.split_at(physical_fields.len());
+ for (i, child) in physical_fields.iter().enumerate() {
+ let child_id = if matching.use_field_id {
+ field_id(child)
+ } else {
+ None
+ };
+ let mut selectors = requested_fields.iter().enumerate().filter(|(j, r)| {
+ requested_folded[*j] == physical_folded[i]
+ || (child_id.is_some() && field_id(r) == child_id)
+ });
+ match (selectors.next(), selectors.next()) {
+ (None, _) => {
+ let n = leaf_count(child.data_type());
+ out.unrequested.extend(*next_leaf..*next_leaf + n);
+ *next_leaf += n;
+ }
+ (Some((_, requested_child)), None) => push_unrequested_leaves(
+ child.data_type(),
+ requested_child.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (Some(_), Some(_)) => *next_leaf += leaf_count(child.data_type()),
+ }
+ }
+ }
+ (DataType::List(physical_item), DataType::List(requested_item)) => push_unrequested_leaves(
+ physical_item.data_type(),
+ requested_item.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (
+ DataType::Map(physical_entries, physical_sorted),
+ DataType::Map(requested_entries, requested_sorted),
+ ) if physical_sorted == requested_sorted => {
+ match (physical_entries.data_type(), requested_entries.data_type()) {
+ (DataType::Struct(physical_kv), DataType::Struct(requested_kv))
+ if physical_kv.len() == requested_kv.len() =>
+ {
+ for (p, r) in physical_kv.iter().zip(requested_kv.iter()) {
+ push_unrequested_leaves(
+ p.data_type(),
+ r.data_type(),
+ next_leaf,
+ matching,
+ out,
+ );
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+}
+
+/// The writer time zone recorded in `metadata`, classified for timestamp rebasing. Mirrors the
+/// `Option(lookupFileMeta(SPARK_TIMEZONE_METADATA_KEY))` lookup Spark's `getRebaseSpec` performs
+/// for every LEGACY resolution, conf-fallback included; Spark substitutes the JVM default zone
+/// when the key is absent (`RebaseSpec.timeZone`), which is unavailable natively, so an absent or
+/// non-UTC zone classifies as [`WriterTimeZone::OtherOrUnknown`] (dates still rebase fully --
+/// the day rebase is zone-free -- while ancient timestamps refuse rather than guess).
+fn writer_time_zone(metadata: &HashMap<String, String>) -> WriterTimeZone {
+ match metadata.get(SPARK_TIMEZONE_KEY) {
+ Some(tz) if UTC_EQUIVALENT_TIMEZONES.contains(&tz.as_str()) => WriterTimeZone::Utc,
+ _ => WriterTimeZone::OtherOrUnknown,
+ }
+}
+
+/// One spec resolution, mirroring Spark's `DataSourceUtils.getRebaseSpec` exactly: a Spark
+/// version below `min_version` (lexicographic comparison, same as the Scala `String.<`) or a
+/// present legacy flag means LEGACY; a Spark version at/after `min_version` without the flag
+/// means CORRECTED; no Spark version at all falls back to `conf_mode`, the session read conf
+/// forwarded from the JVM (`getRebaseSpec`'s `modeByConfig` fallback, its ONLY use of the
+/// conf): CORRECTED passes values through, LEGACY rebases (with the writer zone from the
+/// file's `org.apache.spark.timeZone` key, same lookup as the metadata-driven LEGACY path),
+/// and EXCEPTION refuses ancient values as [`RebasePolicy::CheckAncient`].
+fn resolve_spec(
+ metadata: &HashMap<String, String>,
+ min_version: &str,
+ legacy_key: &str,
+ conf_mode: RebaseReadMode,
+) -> RebasePolicy {
+ match metadata.get(SPARK_VERSION_METADATA_KEY) {
+ None => match conf_mode {
+ RebaseReadMode::Corrected => RebasePolicy::Corrected,
+ RebaseReadMode::Legacy => RebasePolicy::Legacy(writer_time_zone(metadata)),
+ RebaseReadMode::Exception => RebasePolicy::CheckAncient,
+ },
+ Some(version) => {
+ if version.as_str() < min_version || metadata.contains_key(legacy_key) {
+ RebasePolicy::Legacy(writer_time_zone(metadata))
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+ }
+}
+
+/// Resolves the per-file rebase policies from a file's arrow schema: the parquet footer's
+/// key-value pairs in its metadata decide the specs (the datetime spec uses min version
+/// `3.0.0` and the INT96 spec `3.1.0`, matching `DataSourceUtils.datetimeRebaseSpec` /
+/// `int96RebaseSpec`; `session_modes` supplies the per-spec conf fallback for files without
+/// Spark writer metadata), and the reader factory's INT96 stamp -- validated against the
+/// schema's leaf structure -- attributes each timestamp leaf to its spec.
+pub(crate) fn resolve_file_rebase_policies(
+ physical_file_schema: &Schema,
+ session_modes: SessionRebaseModes,
+) -> FileRebasePolicies {
+ let metadata = physical_file_schema.metadata();
+ let datetime_spec = resolve_spec(
+ metadata,
+ "3.0.0",
+ SPARK_LEGACY_DATETIME_KEY,
+ session_modes.datetime,
+ );
+ let int96_spec = resolve_spec(
+ metadata,
+ "3.1.0",
+ SPARK_LEGACY_INT96_KEY,
+ session_modes.int96,
+ );
+ FileRebasePolicies {
+ date: datetime_spec,
+ int64_timestamp: datetime_spec,
+ int96_timestamp: int96_spec,
+ int96_leaves: Int96Attribution::from_schema(physical_file_schema),
+ unrequested_leaves: Vec::new(),
+ ntz_requested_leaves: Vec::new(),
+ ltz_requested_leaves: Vec::new(),
+ numeric_requested_leaves: Vec::new(),
+ }
+}
+
+/// Number of primitive leaves `dt` contains in a depth-first walk -- the same count and order
+/// parquet-rs uses when it maps the file's `SchemaDescriptor` columns onto the arrow schema, so
+/// arrow-side leaf ordinals line up with [`int96_leaf_stamp`]'s.
+fn leaf_count(dt: &DataType) -> usize {
+ match dt {
+ DataType::Struct(fields) => fields.iter().map(|f| leaf_count(f.data_type())).sum(),
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_count(f.data_type()),
+ DataType::Dictionary(_, value) => leaf_count(value),
+ DataType::RunEndEncoded(_, value) => leaf_count(value.data_type()),
+ DataType::Union(fields, _) => fields.iter().map(|(_, f)| leaf_count(f.data_type())).sum(),
+ _ => 1,
+ }
+}
+
+/// Appends the policy of every leaf of `dt`, in depth-first order, to `out`, consuming leaf
+/// ordinals from `next_leaf` (exactly [`leaf_count`] of them). Only `Date32` and timestamps
+/// have a policy to apply, and only when the query reads the leaf; a timestamp leaf's policy
+/// follows the type the query reads it as (see [`FileRebasePolicies::timestamp_policy`] and
+/// [`FileRebasePolicies::tz_free_timestamp_policy`]), and every other leaf is the identity
+/// ([`RebasePolicy::Corrected`]).
+fn leaf_policies(
+ dt: &DataType,
+ next_leaf: &mut usize,
+ policies: &FileRebasePolicies,
+ out: &mut Vec<RebasePolicy>,
+) {
+ match dt {
+ DataType::Date32 => {
+ out.push(policies.date_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Timestamp(_, Some(_)) => {
+ out.push(policies.timestamp_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Timestamp(_, None) => {
+ out.push(policies.tz_free_timestamp_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Struct(fields) => {
+ for f in fields {
+ leaf_policies(f.data_type(), next_leaf, policies, out);
+ }
+ }
+ // Mirrors `leaf_count` variant for variant, so a rebase-affected leaf inside a nested
+ // type `rebase_array` cannot rebuild (views, run-end, union -- never produced from a
+ // parquet schema) still gets its real policy and makes `rebase_array` refuse loudly
+ // instead of being stamped the identity.
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_policies(f.data_type(), next_leaf, policies, out),
+ DataType::Dictionary(_, value) => leaf_policies(value, next_leaf, policies, out),
+ DataType::RunEndEncoded(_, value) => {
+ leaf_policies(value.data_type(), next_leaf, policies, out)
+ }
+ DataType::Union(fields, _) => {
+ for (_, f) in fields.iter() {
+ leaf_policies(f.data_type(), next_leaf, policies, out);
+ }
+ }
+ _ => {
+ *next_leaf += 1;
+ out.push(RebasePolicy::Corrected);
Review Comment:
Fixed in cca449728. An INT32 leaf requested as DATE now takes the file's date policy, matching `IntegerWithRebaseUpdater`, and the stored days are rebased before the schema adapter casts them. Narrower integer storage (INT_8, INT_16, unsigned) has no native cast to DATE, so those reads still fail at the cast rather than return a date. The regression is your shape: a metadata-free `int32 v` behind a `(v DATE)` catalog table, `CONVERT TO DELTA ... NO STATISTICS`, read under LEGACY (1500-01-01), CORRECTED (1500-01-10) and EXCEPTION.
Under EXCEPTION the native scan refuses the row as Spark does, but the error is a `CometNativeException` rather than `SparkUpgradeException`. That holds for every native rebase refusal in this PR, so I filed #6558 to map them to Spark's error class in a separate change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3886372102
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -193,11 +688,16 @@ fn remap_physical_schema(
}
// Block accidental name match for ID-bearing logical fields whose ID is missing
- // from the file. Mirrors Spark's `generateFakeColumnName` in `matchIdField`.
+ // from the file. Mirrors Spark's `generateFakeColumnName` in `matchIdField`. Runs
+ // regardless of `case_sensitive` (field-ID matching is independent of Spark's
+ // casing config), so `case_tables` may be absent here even when it would be
+ // populated for a genuinely case-insensitive scan -- `names_equal_ignore_case_java`
+ // degrades to `str::to_lowercase` in that case, which is fine for this defensive,
+ // over-approximating check (see `java_lowercase`).
if should_match_by_id
&& unmatched_id_logical_names
.iter()
- .any(|name| name.eq_ignore_ascii_case(field.name()))
+ .any(|name| names_equal_ignore_case_java(name, field.name(), case_tables))
Review Comment:
Fixed, the shield compares exactly under case sensitivity and only fires for fields the name pass didn't claim, matching clipParquetGroupFields. Auditing the neighboring comparisons turned up four more divergences from Spark's semantics (a stray ID-less column could win an exact-name lookup for an ID-matched field, the fake name could collide with a real column, and two duplicate-handling mismatches), all fixed with tests against real files.
One thing from that audit worth your eyes separately: validateIcebergFileScanTasks keys on a hardcoded scheme list including gcs, oss, wasb, and wasbs, which the native url parser doesn't recognize, so those Iceberg tables get claimed and then fail at execution. Left untouched here since it's the Iceberg path, but it's the same class of drift this PR kept hitting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5760329511
@andygrove @viirya main is merged in as of today. Is this a candidate for 1.1.0, or should it wait for the release after?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4012778373
##########
spark/pom.xml:
##########
@@ -222,6 +222,27 @@ under the License.
</dependencies>
<profiles>
+ <profile>
+ <!-- Only the contrib build needs the test-jar: contrib/delta-spark reuses CometTestBase
+ and the shared test utilities in its suites. Binding it here keeps the jar out of
+ every other profile's install and out of the release upload. -->
+ <id>delta</id>
Review Comment:
### Correctness
[P2] Enable the Delta profile when installing the prerequisite test JAR
Moving this execution into `delta` means the prerequisite installs in `delta_contrib_test.yml` (lines 113 and 168) no longer produce Spark's `-tests.jar`: both install `common,spark` without `delta`. The following contrib-only `test` invocation requires that test JAR through `contrib/delta-spark/pom.xml:108–114`, but does not rebuild Spark. If no matching test artifact is available, the contrib build fails dependency resolution. An older repository or cache artifact can instead supply stale test classes. These commands never produce this checkout's current test JAR. Please activate `delta` for the prerequisite installs and the README's matching command, or use a build sequence that produces the current test JAR. The parent POM supplies no alternative execution; this is a new build regression distinct from the release-policy discussion in #5882.
##########
.github/workflows/delta_contrib_test.yml:
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Delta Contrib Tests
+
+# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
+# live in the umbrella workflow.
+on:
+ workflow_call:
+
+permissions:
+ contents: read
+
+env:
+ RUST_VERSION: stable
+ RUST_BACKTRACE: 1
+ # Force GNU ld on Linux: rust-lld cannot resolve -ljvm against the Zulu JDK
+ # layout installed by setup-java (same rationale as pr_build_linux.yml).
+ RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
+ # The container's default locale is POSIX, which makes the JVM's file-path encoder
+ # reject non-ASCII partition directory names the suites create.
+ LANG: "C.UTF-8"
+ LC_ALL: "C.UTF-8"
+
+jobs:
+
+ contrib-delta:
+ name: Delta contrib (Spark ${{ matrix.profile.spark }})
+ runs-on: ubuntu-24.04
+ container:
+ image: amd64/rust
+ strategy:
+ matrix:
+ profile:
+ - spark: "3.5"
+ java_version: "17"
+ - spark: "4.0"
+ java_version: "17"
+ - spark: "4.1"
+ java_version: "17"
+ # spark-4.2 is intentionally absent: the contrib profile is dormant
+ # until a Delta release supports Spark 4.2.
+ fail-fast: false
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ env.RUST_VERSION }}
+ jdk-version: ${{ matrix.profile.java_version }}
+
+ - name: Cache Maven dependencies
+ uses: actions/cache@v6
+ with:
+ path: |
+ ~/.m2/repository
+ /root/.m2/repository
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-${{ matrix.profile.spark }}
+ restore-keys: |
+ ${{ runner.os }}-java-maven-
+
+ - name: Restore Cargo cache
+ uses: actions/cache/restore@v6
+ with:
+ path: |
+ ~/.cargo/registry
+ ~/.cargo/git
+ native/target
+ key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
+ restore-keys: |
+ ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-
+
+ - name: Build native library (CI profile)
+ run: |
+ cd native
+ # `delta` is in the default feature set, so no feature flag is needed.
+ cargo build --profile ci
+ env:
+ # Must match the flags spark_sql_test_reusable.yml builds with:
+ # cargo folds RUSTFLAGS into its fingerprints, so any divergence
+ # would make the shared cargo cache restore without ever hitting.
+ RUSTFLAGS: "-Ctarget-cpu=x86-64-v3 -Clink-arg=-fuse-ld=bfd"
+
+ # No cache save: this workflow never runs on a push to main, so the restore above
+ # falls back to the cache build_linux saves there.
+
+ - name: Stage native library at release path
+ run: |
+ # Maven's -Prelease profile (activated below) expects libcomet.so
+ # under native/target/release/; --profile ci builds it under
+ # native/target/ci/ instead (same as the other native-building
+ # workflows), so copy it into place.
+ mkdir -p native/target/release
+ cp native/target/ci/libcomet.so native/target/release/libcomet.so
+
+ - name: Install Comet core jars
+ run: |
+ ./mvnw -B -q -Prelease -Pspark-${{ matrix.profile.spark }} install -pl common,spark -DskipTests -Dspotless.check.skip=true
+
+ - name: Run Delta contrib test suites
+ run: |
+ SPARK_HOME=$(pwd) COMET_CONF_DIR=$(pwd)/conf ./mvnw -B -Prelease -Pspark-${{ matrix.profile.spark }},delta test -pl contrib/delta-spark
+
+ # The container jobs above have no Docker socket, so CometDeltaS3Suite cancels itself
+ # there. This job runs it on the plain runner, where Testcontainers can start MinIO.
+ contrib-delta-s3:
+ name: Delta contrib S3 suite (Spark 3.5, MinIO)
+ runs-on: ubuntu-24.04
+ steps:
+ - uses: actions/checkout@v7
+
+ - name: Setup Rust & Java toolchain
+ uses: ./.github/actions/setup-builder
Review Comment:
### Correctness
[P2] Use runner-compatible dependency installation for the MinIO job
This new job runs directly on `ubuntu-24.04`, but `setup-builder/action.yaml:35–37` executes `apt-get update` and `apt-get install` without `sudo`. That action works inside the other jobs' root container; the hosted Linux runner requires elevation for those commands ([GitHub runner privileges](https://docs.github.com/en/actions/reference/runners/github-hosted-runners#administrative-privileges)). Setup therefore fails before reaching the native build or `CometDeltaS3Suite`. Please use a setup path that elevates package installation on the plain runner while retaining Docker access. This is verified from the workflow/action wiring; the current workflow has not executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4006983830
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,2096 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::mem::size_of;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::{ParquetAccessPlan, RowGroupAccess};
+use datafusion::execution::memory_pool::{MemoryConsumer, MemoryReservation};
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::path::Path;
+use object_store::{ObjectStore, ObjectStoreExt};
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::{PageIndexPolicy, ParquetMetaData};
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ // A corrupt footer can report a negative row count. `num_rows as u64` would otherwise
+ // wrap it into a huge positive value, silently corrupting every row-group boundary
+ // computed from `group_start`/`group_end` below (and therefore which deleted row indexes
+ // land in which row group) instead of failing loudly.
+ if num_rows < 0 {
+ return Err(GeneralError(format!(
+ "Parquet footer reports a negative row count ({num_rows}) for row group {idx}"
+ )));
+ }
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
Review Comment:
> `checked_add` with the existing error style would match the rest.
Done; three row groups of `i64::MAX` rows now fail with the overflow error. The on-disk fixture writes two framed blobs back to back and a sixth file reads the second one through the `offset..offset+framed_len` slice, and a new access-plan test deletes the last row of group 0 and the first row of group 1, asserting the skip lands at the tail of one group and the head of the next with nothing crossing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5521710321
Thanks for the breadth here. Two of these were defects worth fixing in the PR, done and pushed in 48e5d2902; the rest are filed as tracking issues per your suggestion. Point by point:
**Calendar handling: fixed in-PR.** Your repro was real and it led somewhere interesting: the regular native scan has no rebase handling at all today (that is documented as a known limitation and tracked in #5010, ParquetRebaseDatetimeSuite is disabled for Comet), so there was nothing to inherit. The Delta arm now resolves calendar policy per file on the native side: the footer's writer metadata (spark version, legacyDateTime, legacyINT96, timeZone) survives into the Arrow schema the expression adapter sees per file, and it resolves date and timestamp policies the way DataSourceUtils.getRebaseSpec does. Dates rebase exactly (Spark's julianGregDiffs table ported verbatim, with calendar arithmetic before its range). Timestamps rebase exactly for UTC writer zones. Anything it cannot rebase faithfully (non UTC writer zone, missing metadata, contradictory flags) refuses ancient values with an error naming the column instead of returning shifted data; modern values always pass. Your exa
ct 1500-01-01 and ancient INT96 cases are regression tests now, RED before the fix with your 1500-01-10 shift reproduced, plus mixed writer metadata and a temporal predicate case. The mechanism is gated to the Delta arm so the regular scan's documented behavior is unchanged, pinned by a control test.
**Assumed-role session policies: fixed in-PR.** fs.s3a.assumed.role.policy now declines admission, global and bucket scoped, resolved exactly like its consumer (AssumedRoleCredentialProvider reads it with getTrimmed on the propagated conf and sends it in the AssumeRole request, verified against 3.3.4 and 3.4.1 bytecode; native never forwards it). The reason text names the key without echoing the policy document. Unset stays admitted, pinned.
**Tracking issues filed** for the rest: DV file splitting #5655, compressed DV decoding #5656 (a candidate patch preserving sub bitmap containers already exists from a community contributor and can be adapted once this lands), overlapping bitmap and footer reads #5657, shared cloud compatibility helper #5658, credential scoping #5659, v2 checkpoint depth #5660, capability table #5661.
**Benchmark on this head** (48e5d2902 exactly, no extra commits; Apple M5, JDK 17, spark-3.5 profile, local FS, 120M rows, 6 files of ~490MB, zstd, medians of 5 warm runs per fresh session; results bit identical across all modes and verified against closed form expectations):
| pattern | deleted | stock | comet fallback | native |
|---|---|---|---|---|
| none | 0 | 5.31s | 5.25s | 1.13s |
| sparse (0.1% scattered) | 120K | 7.76s | 5.35s | 1.52s |
| contiguous (20%) | 24M | 5.76s | 2.77s | 1.22s |
| alternating (50%) | 60M | 4.76s | 2.53s | 2.51s |
Native output_rows confirms DV application inside the scan (119,880,000 on sparse). Preparation cost measured on the real exported DVs through the PR's own decode and access plan code: decode is negligible everywhere (CRC plus roaring, up to 0.36ms per file on the 2.5MB alternating DVs); the cost center is selector expansion for alternating deletes, 61 to 93ms and ~400MB true peak per file, about 0.5s CPU across the table. Contiguous deletes cost near zero allocation since fully deleted row groups become skips.
Two operational notes the alternating case surfaced. First, the default dv.maxDeletedRowsPerFile cap (1M) declines contiguous and alternating up front and falls back cleanly; the numbers show that guard picks the better path for alternating (native only ties fallback there while needing a ~1.6GB reservation per file). Second, raising the cap without sizing the off heap pool fails tasks hard at the reservation, which is the documented reserve before build behavior doing its job; 18g admits it. The reservation is about 4x the measured true peak, deliberately pessimistic.
Suite batteries at this head pass 227/227 on both spark-3.5 and spark-4.0, MinIO suite live. The remaining gap from your list is live object store validation beyond MinIO, which I do not have infrastructure for here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "ErikBPF (via GitHub)" <gi...@apache.org>.
ErikBPF commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3920223308
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -76,6 +77,583 @@ fn schema_has_field_ids(schema: &SchemaRef) -> bool {
schema.fields().iter().any(|f| parse_field_id(f).is_some())
}
+// ---------------------------------------------------------------------------------------------
+// JVM-shipped case tables: reproduce the PLANNING JVM's `String.toLowerCase(Locale.ROOT)`,
+// which Spark's Parquet footer field matching is built on.
+//
+// The data arrives on `NativeScanCommon` (populated by `JvmCaseTables.scala` when
+// case_sensitive = false), generated from the very JVM that plans the query, so the native
+// matcher is correct BY CONSTRUCTION for whatever JDK runs Spark. The one contextual mapping
+// (`Locale.ROOT` has exactly one: Greek capital sigma U+03A3) cannot be a per-codepoint table
+// entry, so its condition is ported as an algorithm over a shipped per-codepoint
+// classification -- see `JvmCaseTables::lowercase`.
+// ---------------------------------------------------------------------------------------------
+
+// Sigma-scan classes: the wire contract shared with `JvmCaseTables.scala`. The classes are
+// the UAX#29-style word-break classes (ALetter, Numeric, MidLetter, MidNum, MidNumLet,
+// Extend, Format) as the PLANNING JVM's legacy break iterator actually realizes them --
+// probed per codepoint from `BreakIterator.isBoundary` on the JVM side -- plus classes for
+// its pre-UAX#29 extensions (the danda and the supplementary-plane behaviors of its UTF-16
+// DFA), with cased variants split out via the JDK's `isCased`. Any codepoint outside every
+// shipped range -- and any class value this build does not know -- is a word boundary: the
+// sigma context scan stops there, which is also the safe reading for class values added by a
+// NEWER JVM-side generator.
+const CLASS_ALETTER_CASED: u8 = 1;
+const CLASS_ALETTER: u8 = 2;
+const CLASS_NUMERIC: u8 = 3;
+const CLASS_MID_LETTER: u8 = 4;
+const CLASS_MID_NUM: u8 = 5;
+const CLASS_MID_NUM_LET: u8 = 6;
+/// Cased supplementary char: attaches to the preceding word and closes it (and forms a word
+/// of its own at raw text start).
+const CLASS_SUPP_CASED: u8 = 7;
+/// U+0964/U+0965: word-terminal, chains only into digits.
+const CLASS_DANDA: u8 = 8;
+/// U+0345, the one cased combining mark: cased only when its run is attached to a word.
+const CLASS_EXTEND_CASED: u8 = 9;
+/// Cased digit-base (Nl Roman numerals): joins like CLASS_ALETTER_CASED when reached
+/// directly, but bridges only mid-num punctuation, never mid-letter.
+const CLASS_NUMERIC_CASED: u8 = 10;
+/// Non-cased Mn/Me marks: riders that attach only to genuine letter/digit bases.
+const CLASS_EXTEND: u8 = 11;
+/// Word-forming non-cased supplementary letter: a genuine letter-base that closes the word
+/// immediately after itself.
+const CLASS_SUPP_LETTER: u8 = 12;
+/// Cf format characters: fully transparent (WB4-style) -- deleted from the sequence before
+/// the scans run, so a pure-format rider chain bridges mid punctuation ("AΣ-<ZWJ>b" is one
+/// word exactly like "AΣ-b").
+const CLASS_FORMAT: u8 = 13;
+/// Supplementary chars that attach to the preceding word but never form one themselves
+/// (supplementary combining marks, tag characters): a cased mark riding on one belongs to
+/// the sigma's word only when the run hangs off a real base (`supp_mn_anchor`).
+const CLASS_SUPP_MN: u8 = 14;
+/// Word-forming supplementary digit: like CLASS_SUPP_LETTER except a riding cased mark
+/// carries only across mid-num (digit-context) punctuation, never mid-letter.
+const CLASS_SUPP_NUM: u8 = 15;
+/// Not on the wire: the absence of a class.
+const CLASS_BOUNDARY: u8 = 0;
+
+const CAPITAL_SIGMA: char = '\u{03A3}';
+const SMALL_SIGMA: char = '\u{03C3}';
+const SMALL_FINAL_SIGMA: char = '\u{03C2}';
+
+/// What a supplementary-mark run ultimately hangs off (see `supp_mn_anchor`).
+#[derive(PartialEq, Eq, Clone, Copy)]
+enum SuppMnAnchor {
+ None,
+ Letter,
+ Digit,
+}
+
+/// The planning JVM's case data, parsed once per scan from `NativeScanCommon` and attached to
+/// [`SparkParquetOptions`]. Two tables:
+///
+/// - `lower`: every codepoint the JVM lowercases non-identically, with its full (possibly
+/// multi-char, e.g. U+0130 -> "i" + U+0307) replacement; codepoints absent here lowercase
+/// to themselves;
+/// - `class_ranges`: sorted, disjoint `(start, end, class)` codepoint ranges holding the
+/// word-break classification the JVM probed from its own `BreakIterator`.
+///
+/// `lowercase` applies Java's algorithm over that data: per codepoint, U+03A3 takes its
+/// contextual final/non-final form via the ported `isFinalCased` condition -- word-boundary
+/// based (the JDK's legacy break-iterator word rules), NOT the Unicode-standard Final_Sigma
+/// case-ignorable skip, so e.g. "A1Σ" lowers to "a1ς" -- and every other codepoint takes its
+/// table replacement. `JvmCaseTables.mirrorLowercase` on the Scala side is the line-for-line
+/// mirror of this function over the same generated data; the JVM-side parity suite proves the
+/// pair equal to the running JDK's `String.toLowerCase(Locale.ROOT)` across the full codepoint
+/// space (calibrated to zero mismatches on JDK 17, 21, and 25).
+#[derive(Debug)]
+pub struct JvmCaseTables {
+ /// Non-identity lowercase mappings: codepoint -> full replacement string.
+ lower: HashMap<char, String>,
+ /// Sorted, disjoint (start, end, class) inclusive codepoint ranges for the sigma scan.
+ class_ranges: Vec<(u32, u32, u8)>,
+ /// Precomputed content hash so `SparkParquetOptions`'s derived `Hash` stays cheap.
+ fingerprint: u64,
+}
+
+impl PartialEq for JvmCaseTables {
+ fn eq(&self, other: &Self) -> bool {
+ self.fingerprint == other.fingerprint
+ && self.class_ranges == other.class_ranges
+ && self.lower == other.lower
+ }
+}
+
+impl Eq for JvmCaseTables {}
+
+impl Hash for JvmCaseTables {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ // Equal contents always produce the equal (deterministically computed) fingerprint,
+ // so hashing only the fingerprint is consistent with `PartialEq`.
+ state.write_u64(self.fingerprint);
+ }
+}
+
+fn is_letter_base(cls: u8) -> bool {
+ cls == CLASS_ALETTER_CASED
+ || cls == CLASS_ALETTER
+ || cls == CLASS_SUPP_CASED
+ || cls == CLASS_SUPP_LETTER
+}
+
+fn is_digit_base(cls: u8) -> bool {
+ cls == CLASS_NUMERIC || cls == CLASS_NUMERIC_CASED
+}
+
+impl JvmCaseTables {
+ /// Parse the proto representation: `lower_cp`/`lower_repl` are index-aligned, and
+ /// `class_ranges` holds (start, end, class) triples. Malformed input (length mismatch,
+ /// trailing partial triple, out-of-range codepoints) is dropped entry-by-entry rather
+ /// than rejected: every dropped entry degrades one codepoint to identity/boundary
+ /// behavior instead of failing the scan.
+ pub fn from_proto(lower_cp: &[u32], lower_repl: &[String], class_ranges: &[u32]) -> Self {
+ let mut lower = HashMap::with_capacity(lower_cp.len());
+ for (cp, repl) in lower_cp.iter().zip(lower_repl.iter()) {
+ if let Some(c) = char::from_u32(*cp) {
+ lower.insert(c, repl.clone());
+ }
+ }
+ let ranges: Vec<(u32, u32, u8)> = class_ranges
+ .as_chunks::<3>()
+ .0
+ .iter()
+ .filter(|t| t[0] <= t[1] && t[1] <= 0x10FFFF && u8::try_from(t[2]).is_ok())
+ .map(|t| (t[0], t[1], t[2] as u8))
+ .collect();
+
+ let mut hasher = DefaultHasher::new();
+ for (start, end, class) in &ranges {
+ hasher.write_u32(*start);
+ hasher.write_u32(*end);
+ hasher.write_u8(*class);
+ }
+ let mut lower_sorted: Vec<(&char, &String)> = lower.iter().collect();
+ lower_sorted.sort_by_key(|(c, _)| **c);
+ for (c, repl) in lower_sorted {
+ hasher.write_u32(*c as u32);
+ hasher.write(repl.as_bytes());
+ }
+
+ Self {
+ lower,
+ class_ranges: ranges,
+ fingerprint: hasher.finish(),
+ }
+ }
+
+ /// Sigma-scan class of `c`; `CLASS_BOUNDARY` when no shipped range covers it.
+ fn class_of(&self, c: char) -> u8 {
+ let cp = c as u32;
+ let mut lo = 0usize;
+ let mut hi = self.class_ranges.len();
+ while lo < hi {
+ let mid = (lo + hi) / 2;
+ let (start, end, class) = self.class_ranges[mid];
+ if cp < start {
+ hi = mid;
+ } else if cp > end {
+ lo = mid + 1;
+ } else {
+ return class;
+ }
+ }
+ CLASS_BOUNDARY
+ }
+
+ /// First position at or beyond `start` (stepping by `step`, i.e. -1 backward / +1
+ /// forward) whose class is not CLASS_EXTEND. Returns `None` if the scan runs off the
+ /// array without finding one.
+ fn skip_extends(&self, cps: &[char], start: isize, step: isize) -> Option<usize> {
+ let mut k = start;
+ while k >= 0 && (k as usize) < cps.len() {
+ if self.class_of(cps[k as usize]) != CLASS_EXTEND {
+ return Some(k as usize);
+ }
+ k += step;
+ }
+ None
+ }
+
+ /// As [`Self::skip_extends`], but also skips CLASS_EXTEND_CASED, reporting whether one
+ /// was walked: a cased mark (U+0345) crossed while looking for a base is itself cased
+ /// whenever the landing validates the run.
+ fn skip_extends_tracking_cased(
+ &self,
+ cps: &[char],
+ start: isize,
+ step: isize,
+ ) -> (Option<usize>, bool) {
+ let mut k = start;
+ let mut saw_cased = false;
+ while k >= 0 && (k as usize) < cps.len() {
+ let cls = self.class_of(cps[k as usize]);
+ if cls != CLASS_EXTEND && cls != CLASS_EXTEND_CASED {
+ return (Some(k as usize), saw_cased);
+ }
+ if cls == CLASS_EXTEND_CASED {
+ saw_cased = true;
+ }
+ k += step;
+ }
+ (None, saw_cased)
+ }
+
+ /// What the supplementary-mark run at `k` (CLASS_SUPP_MN) ultimately hangs off, walking
+ /// down through further marks and supplementary chars: a letter-flavored base, a
+ /// digit-flavored base, or nothing word-forming. A cased mark riding the run belongs to
+ /// the sigma's word only per this anchor.
+ fn supp_mn_anchor(&self, cps: &[char], k: usize) -> SuppMnAnchor {
+ let mut m = k as isize - 1;
+ while m >= 0 {
+ let cls = self.class_of(cps[m as usize]);
+ if cls != CLASS_SUPP_MN && cls != CLASS_EXTEND && cls != CLASS_EXTEND_CASED {
+ break;
+ }
+ m -= 1;
+ }
+ if m < 0 {
+ return SuppMnAnchor::None;
+ }
+ match self.class_of(cps[m as usize]) {
+ CLASS_ALETTER_CASED | CLASS_ALETTER | CLASS_SUPP_CASED | CLASS_SUPP_LETTER => {
+ SuppMnAnchor::Letter
+ }
+ CLASS_NUMERIC | CLASS_NUMERIC_CASED | CLASS_SUPP_NUM => SuppMnAnchor::Digit,
+ _ => SuppMnAnchor::None,
+ }
+ }
+
+ /// Backward half of the ported `isFinalCased`: is there a cased letter before position
+ /// `i` within the sigma's word? Runs over the FORMAT-FILTERED sequence; `leading_format`
+ /// says whether format chars were filtered off the raw text start.
+ fn scan_back_finds_cased(&self, cps: &[char], i: usize, leading_format: bool) -> bool {
+ let mut last_letter = true; // the sigma itself is a letter
+ let mut j = i as isize - 1;
+ while j >= 0 {
+ match self.class_of(cps[j as usize]) {
+ CLASS_ALETTER_CASED | CLASS_NUMERIC_CASED => return true,
+ CLASS_ALETTER => {
+ last_letter = true;
+ j -= 1;
+ }
+ CLASS_NUMERIC => {
+ last_letter = false;
+ j -= 1;
+ }
+ CLASS_EXTEND => {
+ // Non-cased marks attach only to a real base below them; anything else
+ // (mid punctuation, danda, boundary, text start) leaves the run
+ // unattached.
+ let Some(k) = self.skip_extends(cps, j, -1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ let is_continuer = b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_EXTEND_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_NUM;
+ if !is_continuer {
+ return false;
+ }
+ j = k as isize;
+ }
+ CLASS_SUPP_CASED => {
+ // Closes the preceding word, so the scan stops -- except at RAW text
+ // start (no filtered-out leading format chars), where the DFA keeps it
+ // joined to what follows.
+ return j == 0 && !leading_format;
+ }
+ CLASS_SUPP_LETTER | CLASS_SUPP_MN | CLASS_SUPP_NUM => {
+ // Attach/close and never themselves cased; nothing beyond is reachable.
+ return false;
+ }
+ CLASS_EXTEND_CASED => {
+ // Cased combining mark (U+0345): cased when its run hangs off a base --
+ // a BMP letter/digit, a word-forming supplementary char (which closes a
+ // word right below the mark, merging the mark into the sigma's
+ // segment), or an ANCHORED supplementary mark.
+ let (Some(k), _) = self.skip_extends_tracking_cased(cps, j - 1, -1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ if b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_NUM
+ {
+ return true;
+ }
+ if b == CLASS_SUPP_MN {
+ return self.supp_mn_anchor(cps, k) != SuppMnAnchor::None;
+ }
+ return false;
+ }
+ CLASS_DANDA => {
+ // Backward across a danda: the word part before it must end in letters
+ // (grammar: letters, optional danda, then number+word chains) -- or
+ // carry a riding cased mark on a word-forming base, or be a cased
+ // supplementary char at text start -- and the danda itself chains only
+ // into digits after it.
+ if last_letter {
+ return false;
+ }
+ let (Some(k), saw_cased_mark) =
+ self.skip_extends_tracking_cased(cps, j - 1, -1)
+ else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ if b == CLASS_ALETTER_CASED {
+ return true;
+ }
+ if b == CLASS_SUPP_CASED {
+ return saw_cased_mark || (k == 0 && !leading_format);
+ }
+ if b == CLASS_SUPP_LETTER {
+ return saw_cased_mark;
+ }
+ if b == CLASS_SUPP_MN {
+ return saw_cased_mark
+ && self.supp_mn_anchor(cps, k) == SuppMnAnchor::Letter;
+ }
+ if b != CLASS_ALETTER {
+ return false;
+ }
+ if saw_cased_mark {
+ return true;
+ }
+ last_letter = true;
+ j = k as isize;
+ }
+ cls @ (CLASS_MID_LETTER | CLASS_MID_NUM | CLASS_MID_NUM_LET) => {
+ // `<mid-letter><let>` / `<mid-num><digit>` require a genuine
+ // letter/digit base before the punctuation; scanning backward
+ // legitimately walks marks-then-base (marks trail their base). A cased
+ // mark walked over rides whatever the punctuation hangs off, including
+ // a context-matching anchored supplementary mark or supplementary
+ // digit.
+ let mw_ok = cls == CLASS_MID_LETTER || cls == CLASS_MID_NUM_LET;
+ let mn_ok = cls == CLASS_MID_NUM || cls == CLASS_MID_NUM_LET;
+ let (Some(real_pos), saw_cased_mark) =
+ self.skip_extends_tracking_cased(cps, j - 1, -1)
+ else {
+ return false;
+ };
+ let b = self.class_of(cps[real_pos]);
+ if last_letter
+ && mw_ok
+ && saw_cased_mark
+ && b == CLASS_SUPP_MN
+ && self.supp_mn_anchor(cps, real_pos) == SuppMnAnchor::Letter
+ {
+ return true;
+ }
+ if !last_letter
+ && mn_ok
+ && saw_cased_mark
+ && (b == CLASS_SUPP_NUM
+ || (b == CLASS_SUPP_MN
+ && self.supp_mn_anchor(cps, real_pos) == SuppMnAnchor::Digit))
+ {
+ return true;
+ }
+ let bridge_valid = (last_letter && mw_ok && is_letter_base(b))
+ || (!last_letter && mn_ok && is_digit_base(b));
+ if !bridge_valid {
+ return false;
+ }
+ if saw_cased_mark {
+ return true;
+ }
+ j = real_pos as isize;
+ }
+ _ => return false,
+ }
+ }
+ false
+ }
+
+ /// Forward half of the ported `isFinalCased`: is there a cased letter after position `i`
+ /// within the sigma's word? Runs over the FORMAT-FILTERED sequence.
+ fn scan_fwd_finds_cased(&self, cps: &[char], i: usize) -> bool {
+ let mut last_letter = true;
+ let mut j = i + 1;
+ while j < cps.len() {
+ match self.class_of(cps[j]) {
+ CLASS_ALETTER_CASED | CLASS_NUMERIC_CASED => return true,
+ CLASS_ALETTER => {
+ last_letter = true;
+ j += 1;
+ }
+ CLASS_NUMERIC => {
+ last_letter = false;
+ j += 1;
+ }
+ CLASS_EXTEND => {
+ // A mark run trailing the anchor is properly attached in text order, so
+ // the run stays open past it, including onto mid punctuation on its far
+ // side.
+ let Some(k) = self.skip_extends(cps, j as isize, 1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ let is_continuer = b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_EXTEND_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_MN
+ || b == CLASS_SUPP_NUM
+ || b == CLASS_DANDA
+ || b == CLASS_MID_LETTER
+ || b == CLASS_MID_NUM
+ || b == CLASS_MID_NUM_LET;
+ if !is_continuer {
+ return false;
+ }
+ j = k;
+ }
+ CLASS_SUPP_CASED | CLASS_EXTEND_CASED => {
+ // Attaches to the current word, so the scan sees it (cased).
+ return true;
+ }
+ CLASS_SUPP_LETTER | CLASS_SUPP_MN | CLASS_SUPP_NUM => {
+ // Attach to the current word and close it; never themselves cased, and
+ // nothing beyond is reachable.
+ return false;
+ }
+ // The danda attaches only to a word part that ends in letters (reached
+ // after digits the word is already closed) and continues only into a digit
+ // -- unless that digit is itself cased (a Roman numeral), which resolves
+ // the scan immediately.
+ CLASS_DANDA if !last_letter => return false,
+ CLASS_DANDA
+ if j + 1 < cps.len() && self.class_of(cps[j + 1]) == CLASS_NUMERIC_CASED =>
+ {
+ return true;
+ }
+ CLASS_DANDA if j + 1 < cps.len() && self.class_of(cps[j + 1]) == CLASS_NUMERIC => {
+ last_letter = false;
+ j += 2;
+ }
+ cls @ (CLASS_MID_LETTER | CLASS_MID_NUM | CLASS_MID_NUM_LET) => {
+ // `<mid-letter><let>` / `<mid-num><digit>` require a genuine
+ // letter/digit base IMMEDIATELY after the punctuation -- unlike the
+ // backward scan, marks here are never skipped past: a mark directly
+ // after the punctuation is attached to the punctuation, not a base, so
+ // it blocks the bridge. (Format chars are already filtered out, which
+ // is what lets "AΣ-<ZWJ>b" bridge exactly like "AΣ-b".)
+ let mw_ok = cls == CLASS_MID_LETTER || cls == CLASS_MID_NUM_LET;
+ let mn_ok = cls == CLASS_MID_NUM || cls == CLASS_MID_NUM_LET;
+ if j + 1 >= cps.len() {
+ return false;
+ }
+ let b = self.class_of(cps[j + 1]);
+ if (last_letter && mw_ok && is_letter_base(b))
+ || (!last_letter && mn_ok && is_digit_base(b))
+ {
+ j += 1;
+ } else {
+ return false;
+ }
+ }
+ _ => return false,
+ }
+ }
+ false
+ }
+
+ /// Lowercase `s` the way the planning JVM's `String.toLowerCase(Locale.ROOT)` does.
+ pub fn lowercase(&self, s: &str) -> String {
+ let raw: Vec<char> = s.chars().collect();
+ // Built lazily on the first sigma: the format-filtered sequence the scans run over
+ // (WB4-style: the legacy break iterator's `<ignore>` class loops on every DFA
+ // state), the raw->filtered index map, and whether format chars led the raw text.
+ let mut filtered: Option<(Vec<char>, Vec<usize>, bool)> = None;
+ let mut out = String::with_capacity(s.len());
+ for (i, &c) in raw.iter().enumerate() {
+ if c == CAPITAL_SIGMA {
+ // The condition consults the ORIGINAL neighbors, exactly as the JDK scans
+ // `src`, not the partially-lowered output. The final/non-final target chars
+ // are Unicode-stable (pinned in `ConditionalSpecialCasing`'s entry table).
+ let (f, idx, leading_format) = filtered.get_or_insert_with(|| {
+ let mut f = Vec::with_capacity(raw.len());
+ let mut idx = vec![0usize; raw.len()];
+ for (k, &rc) in raw.iter().enumerate() {
+ idx[k] = f.len();
+ if self.class_of(rc) != CLASS_FORMAT {
+ f.push(rc);
+ }
+ }
+ let leading_format = self.class_of(raw[0]) == CLASS_FORMAT;
+ (f, idx, leading_format)
+ });
+ let fi = idx[i];
+ let is_final = self.scan_back_finds_cased(f, fi, *leading_format)
+ && !self.scan_fwd_finds_cased(f, fi);
+ out.push(if is_final {
+ SMALL_FINAL_SIGMA
+ } else {
+ SMALL_SIGMA
+ });
+ } else if let Some(repl) = self.lower.get(&c) {
+ out.push_str(repl);
+ } else {
+ out.push(c);
+ }
+ }
+ out
+ }
+}
+
+/// Lowercase `s` for case-insensitive field matching. With tables (populated whenever
+/// `case_sensitive = false`, shared by the core Parquet scan and the Delta contrib scan) this
+/// reproduces the planning JVM's `String.toLowerCase(Locale.ROOT)` exactly.
+///
+/// Without tables, fall back to Rust's `str::to_lowercase`, which agrees with Java on all
+/// simple mappings and differs only where the two Unicode snapshots or the sigma context
+/// diverge. This is a real, live path, not just a defensive default: the Iceberg native scan
+/// (`SparkPhysicalExprAdapterFactory::new(_, None)`) defaults `case_sensitive = false` and
+/// always reaches this fallback for its schema name remap, alongside
+/// `parquet_convert_struct_to_struct`'s general struct-cast matching and Rust-only unit tests
+/// that construct `SparkParquetOptions` directly.
+pub(crate) fn java_lowercase(s: &str, tables: Option<&JvmCaseTables>) -> String {
Review Comment:
Update: [apache/datafusion-comet#5602](https://github.com/apache/datafusion-comet/pull/5602) had already merged before this reply and supersedes the staged patch. Current `main` includes the ASCII fast path, schema-wide pre-folding, exact JVM folding for non-ASCII names, and caching. No additional patch is needed here; I am closing the fork PR to avoid duplicate work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "ErikBPF (via GitHub)" <gi...@apache.org>.
ErikBPF commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5518974053
Correction on the NYC pruning observation above: I traced the predicate shape before proposing a fix.
The source Parquet column is annotated `Timestamp(isAdjustedToUTC=false, timeUnit=microseconds)`, so Spark 3.5 correctly infers `TIMESTAMP_NTZ`. My workload used ordinary `TIMESTAMP` bounds. Spark therefore analyzed each predicate as `cast(tpep_pickup_datetime AS timestamp) ...`; a direct Spark 3.5 plan probe confirms that using `TIMESTAMP_NTZ` bounds removes the cast.
Comet did carry the filter into the native scan, but it becomes Comet’s Spark-compatible custom cast. DataFusion 55 pruning recognizes its own `CastExpr`/`TryCastExpr`, not that custom expression, so the zero page-index/row-group pruning is expected for this mixed LTZ/NTZ predicate. It is a broader cast-pruning limitation, not evidence of a Delta/DV-specific gap.
A type-matched workload rerun would measure available Delta/file/Parquet pruning; I would not propose a timezone-sensitive cast rewrite from the current evidence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5522958178
Both P2s addressed in 053ec14b4, plus one related bug your cases flushed out.
Session read modes: verified the exact getRebaseSpec semantics against the 3.5.9 and 4.0.4 sources first. The conf applies only when org.apache.spark.version is absent from the footer; with a version present, policy is purely the version comparison plus the legacy flag. The serde now resolves the effective modes the way ParquetFileFormat does (ParquetOptions over relation options and session conf, so the 3.5 EXCEPTION and 4.0 CORRECTED defaults come from the session rather than being hardcoded), carries them in two new DeltaSparkScanCommon fields, and the native resolver uses them in the version-absent arm. Your two cases are regression tests built on raw parquet-mr files with no Spark metadata, converted to Delta: corrected 1500-01-01 and the pre-1970 microsecond timestamp now read verbatim natively under CORRECTED, rebase under LEGACY, and fail loudly under EXCEPTION. One nuance mirrored deliberately: conf LEGACY resolves the writer zone from file metadata; Spark falls back to t
he JVM default zone, which native cannot replicate, so non UTC zones keep the refuse-ancient posture for timestamps while dates rebase fully since they are zone free.
Nested types: the wrapper now checks the applicable leaf policies before rejecting, recursing through struct, list, map, and dictionary. Your STRUCT with a DATE leaf under corrected dates plus legacy INT96 stays native and correct, tested with modern and null values; a nested column with a genuinely affected leaf still declines.
The bonus bug: with no pushed predicate and identical logical and physical schemas, the DataFusion opener skips the expression adapter entirely, which is exactly the shape of a metadata-free file, so even the refusal path was silently bypassed. Rebase-enabled scans now stamp a schema marker that keeps the adapter engaged, with a test pinning the ancient-value refusal on that exact shape.
Rebase expression benchmark as requested (M5, release, real module code): per row the wrapper costs 0.7 to 2.2ns depending on type and policy, 0.26 to 0.29ns for the check-only path, against effectively zero unwrapped. End to end on 20M rows the full-scan cost is noise (0.225 to 0.235s across corrected, legacy modern, legacy ancient; Spark fallback 0.36 to 0.47s). The honest cost is selective predicates: the opaque wrapper disables pruning on wrapped columns, and on a sorted table a point predicate decoded 2,627,592 rows against 40,000 with pruning live, a 65x read amplification that warm cache hides but cold or remote storage will not. That is the operational tradeoff of the legacy path; correct results either way, counts identical across all modes.
Batteries at this head: 231/231 on both spark-3.5 and spark-4.0, 327 native lib tests, clippy and fmt clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4042643424
##########
docs/source/user-guide/latest/delta.md:
##########
@@ -0,0 +1,62 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Delta Lake (experimental)
+
+Comet can execute DSv1 Delta Lake table scans natively. Reads planned by
+delta-spark run through Comet's native Parquet scan, inheriting row-group
+pruning, page-index pruning, and filter pushdown, with deletion vectors
+applied inside the scan.
+
+Support is experimental and explicitly opt-in. Two things are required:
+
+1. The `comet-contrib-delta-spark` contrib jar on the classpath, alongside
+ `delta-spark`. It is never bundled into `comet-spark`.
+2. `spark.comet.scan.delta.enabled=true`. The default is `false`, so
+ the jar alone does nothing.
+
+Unsupported tables and features fall back to Spark's reader. See the
+[contrib module README](https://github.com/apache/datafusion-comet/blob/main/contrib/delta-spark/README.md)
+for the supported Spark/Delta version matrix and build instructions.
+
+Unlike the core native scan, the Delta scan resolves each data file's datetime
+calendar-rebase policy from the file's own writer metadata
+(`org.apache.spark.legacyDateTime` and friends), the same way Spark's reader
+does, selecting the `datetimeRebaseModeInRead` spec for dates and INT64
+timestamps and the `int96RebaseModeInRead` spec for INT96 timestamps, at any
+nesting depth: dates written with the legacy hybrid Julian/Gregorian calendar
+are rebased exactly, timestamps are rebased exactly when the file records a
+fixed UTC writer time zone, and ancient values whose calendar cannot be
+applied natively (non-UTC legacy writer zones, or files that do not declare a
+policy under the `EXCEPTION` read mode) raise an error rather than silently
+returning shifted values. Modern values are unaffected: dates from 1582-10-15
+onward, and timestamps from 1900-01-01T00:00:00Z onward (Spark's own
+rebase cutoff). Disable `spark.comet.scan.delta.enabled` for such tables to
+read them through Spark.
+
+## Configuration
+
+<!--BEGIN:CONFIG_TABLE[delta]-->
Review Comment:
> Could the guard cover every entry in `DeltaScanConf.all` including defaults, and fail instead of cancel on a missing file? Or drop the markers so nobody reads the table as generated.
Both, in d707d8d. The markers are gone and a comment says the table is hand-maintained. The guard renders every `DeltaScanConf` entry the way GenerateDocs would, key, doc and default, asserts each row verbatim, flags rows that match no entry, and fails when the page cannot be found. It fails on a changed default and passes on the current page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4035920775
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so its rewrite (via
+ * [[CometScanWithPlanData]]) survives plan copies -- a transient field would be dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning (the CometIcebergNativeScanExec lesson).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node
+ // automatically gets a helper consistent with ITS runtimeFilters, avoiding the #3510 class of
+ // bug where a stored helper field desyncs from rewritten filters. Costs one extra file listing
+ // per executed instance; correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below, and AQE's UI
+ // plan-walk calls `.metrics` on every node MID-PLANNING, including while a DPP subquery is
+ // still an adaptive placeholder or a partition filter holds an unresolved ScalarSubquery (see
+ // `hasUnevaluableSubqueryFilter` below). That's safe ONLY because constructing `scanHelper` is a
+ // cheap case-class build with no file listing, and core's `CometScanExec.metrics` touches only
+ // `wrapped.driverMetrics` (populated by Spark's own planning) plus a static metric-node
+ // constructor -- neither file listing nor subquery resolution. If core's `metrics` ever touches
+ // either, forcing `scanHelper` here would resurrect the AQE mid-planning crashes this invariant
+ // prevents.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element, so checking runtimeFilters
+ // here avoids constructing/forcing the derived scanHelper just to read partitioning. The
+ // InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep in sync. The
+ // ScalarSubquery case is probed rather than treated as permanently unevaluable: Spark exposes no
+ // public finished/updated flag on ExecSubqueryExpression, but `eval()` doubles as one -- it only
+ // reads the cached `result` behind a `require(updated, ...)` guard, while the subquery is
+ // actually run by `updateResult()` (invoked separately during prepare/AQE), never by `eval()`.
+ // Once resolved, outputPartitioning below reports the real perPartitionData.length instead of
+ // staying at zero -- a fused native parent's buildNativeContext requires that count to match.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case s: ScalarSubquery => !isScalarSubqueryResolved(s)
+ case _ => false
+ })
+
+ // `eval()` never triggers the subquery's execution: on a resolved subquery it is a pure cached
+ // read of `result` (verified against bytecode: `Predef.require(updated(), ...)` then a plain
+ // field read), so this probe is safe to call repeatedly, including from AQE's mid-planning plan
+ // walks. Pre-resolution, the ONLY throw is `require`'s `IllegalArgumentException`; catch exactly
+ // that, since anything else escaping is a genuine bug we must not mask as unpartitioned.
+ private def isScalarSubqueryResolved(s: ScalarSubquery): Boolean =
+ try {
+ s.eval()
+ true
+ } catch {
+ case _: IllegalArgumentException => false
+ }
+
+ private def isAdaptivePlaceholder(p: SparkPlan): Boolean = p match {
+ case ReusedSubqueryExec(inner) => isAdaptivePlaceholder(inner)
+ case _: CometSubqueryAdaptiveBroadcastExec => true
+ case _: SubqueryAdaptiveBroadcastExec => true
+ case _ => false
+ }
+
+ override lazy val outputOrdering: Seq[SortOrder] = originalPlan.outputOrdering
+
+ override def dynamicPruningFilters: Seq[Expression] = runtimeFilters
+
+ override def withDynamicPruningFilters(filters: Seq[Expression]): SparkPlan = {
+ // A real copy: runtimeFilters is a constructor field included in equality, so the copy
+ // survives enclosing-block rebuilds, and the derived scanHelper picks up the rewritten
+ // filters automatically.
+ copy(runtimeFilters = filters)
+ }
+
+ /**
+ * Lazy split-mode serialization, mirroring CometNativeScanExec: common data was serialized at
+ * planning; per-partition file lists serialize here, at execution time.
+ */
+ @transient private lazy val serializedPartitionData
+ : (Array[Byte], Array[Array[Byte]], Array[Seq[String]]) = {
+ // Resolve the helper's DPP subqueries: it holds its own InSubqueryExec instances that
+ // Spark's expressions walk does not see (the helper is derived, not a child).
+ scanHelper.partitionFilters.foreach {
+ case DynamicPruningExpression(e: InSubqueryExec) if e.values().isEmpty =>
+ e.updateResult()
+ case _ =>
+ }
+
+ val commonBytes = {
+ val deltaScan = DeltaSparkScanEnvelope.unpack(nativeOp)
+ // Scalar subqueries in dataFilters were unresolved at planning; resolve them now and
+ // append them as pushed filters, as CometNativeScanExec.serializedPartitionData does.
+ // has_data_filters follows their presence, not the serialized count: a filter that fails
+ // to serialize still keeps native on the safe timestamp conversion for a filtered scan.
+ val resolved = org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .resolvedSubqueryFilters(dataFilters, output, requiredSchema, conf)
+ val common = if (!resolved.hasResolvedFilters) {
+ deltaScan.getCommon
+ } else {
+ val builder = deltaScan.getCommon.toBuilder
+ builder.setHasDataFilters(true)
+ resolved.protos.foreach(builder.addDataFilters)
+ builder.build()
+ }
+ OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaScan.getDeltaCommon)
+ .build()
+ .toByteArray
+ }
+
+ val filePartitions = scanHelper.getFilePartitions()
+
+ val tableRoot = DeltaSparkScanEnvelope.unpack(nativeOp).getDeltaCommon.getTableRoot
+ val perPartitionBytes = filePartitions.map { filePartition =>
+ org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .serializePartition(filePartition, originalPlan, tableRoot)
+ }.toArray
+
+ val perPartitionPaths = filePartitions.map(_.files.map(_.filePath.toString).toSeq).toArray
+
+ (commonBytes, perPartitionBytes, perPartitionPaths)
+ }
+
+ override def commonData: Array[Byte] = serializedPartitionData._1
+
+ override def perPartitionData: Array[Array[Byte]] = serializedPartitionData._2
+
+ def perPartitionFilePaths: Array[Seq[String]] = serializedPartitionData._3
+
+ override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+ val nativeMetrics = CometMetricNode.fromCometPlan(this)
+ val serializedPlan = CometExec.serializeNativePlan(nativeOp)
+
+ new CometExecRDD(
+ sparkContext,
+ Seq.empty,
+ Map(sourceKey -> commonData),
+ Map(sourceKey -> perPartitionData),
+ serializedPlan,
+ perPartitionData.length,
Review Comment:
> Could you pass `PlanDataInjector.planFingerprint(serializedPlan)` here, adapt the provider to prepare the common protobuf once, and compile the Delta contrib for both Scala lines?
Done in adc17c286. The scan passes the fingerprint, and `DeltaPlanDataInjector` now parses the common `DeltaSparkScan` once in `prepareCommon` and only merges the partition's file list in `inject`, matching the built-in injectors. `DeltaPlanDataInjectorSuite` pins the contract, including reuse and stale replacement through core's `prepareShared`. The contrib test-compiles under `-Pspark-3.5,delta` (Scala 2.12) and `-Pspark-4.0,delta` (Scala 2.13), and the three Delta suites pass locally. The follow-up commit drops redundant `s` interpolators in `CometDeltaNativeScanSuite` that the semantic scalafix check flags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "schenksj (via GitHub)" <gi...@apache.org>.
schenksj commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5375555814
> > Hey @sunchao / @parthchandra — are you looking to move this work over to @dwsmith1983’s series?
> > We’re 2 PRs into the 10-PR series now that the contrib modules have been merged. The series fully implements all of the Delta protocols, with all 10k+ Delta test cases passing.
> > I’m fine either way. I won’t have a huge amount of time to work on this over the next couple of months, so it could go faster with a different attendant, but the PRs are ready to roll.
> > You can see the series here: https://github.com/schenksj/datafusion-comet/pulls (PRs #5–13).
>
> I see value in both (even though it is extra work to maintain both paths) and in principle agree with @sunchao. Ideally, we want to converge these two. Logged an issue based on an AI generated convergence path - #5411
Thanks guys. I'm concerned that having 2 will create a lot of confusion when it comes to support.. Even enabling and disabling various scan features is too much to understand for most of the expert data engineers I work with every day.
I'm happy to move forward initially in parallel, though like I mentioned before my time to work with this is going to be pretty sparse for the next couple of months.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3835740268
##########
native/core/src/execution/planner/delta_spark_scan.rs:
##########
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! JVM-planned Delta handler for the generic `OpStruct::ContribScan` dispatcher, feature-gated
+//! behind `delta`.
+//!
+//! delta-spark has already done log replay, snapshot resolution, and partition pruning by the
+//! time the scan reaches Comet, so the envelope carries a concrete file list (plus deletion
+//! vector descriptors) and the read path reuses the exact same shared parquet scan builder as
+//! `NativeScan` -- inheriting row-group stats pruning, page-index pruning, and filter pushdown.
+//! Sibling of the kernel-planned handler in `delta_scan.rs`; the two claim different
+//! `type_url`s within the same `ContribScan` envelope.
+
+use std::collections::HashMap;
+
+use datafusion_comet_proto::spark_operator::{
+ ContribScan, DeltaSparkScan, Operator, SparkFilePartition,
+};
+use prost::Message;
+
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::execution::planner::PhysicalPlanner;
+use crate::execution::planner::PlanCreationResult;
+
+/// Type name the JVM-planned Delta contrib claims within the `ContribScan` envelope. The
+/// contrib jar packs a `DeltaSparkScan` with a `type_url` of
+/// `type.googleapis.com/comet.contrib.delta_spark.DeltaSparkScan`; dispatch keys on the
+/// contrib-owned suffix, same convention as the kernel path's `delta_scan.rs`.
+const DELTA_SPARK_SCAN_TYPE_NAME: &str = "comet.contrib.delta_spark.DeltaSparkScan";
+
+/// Contrib entry point for the `OpStruct::ContribScan` dispatcher. Returns `Some(result)` when
+/// the envelope carries a JVM-planned Delta scan, or `None` when the `type_url` belongs to some
+/// other contrib.
+pub(crate) fn try_plan_contrib_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ contrib: &ContribScan,
+) -> Option<PlanCreationResult> {
+ if !contrib.type_url.ends_with(DELTA_SPARK_SCAN_TYPE_NAME) {
+ return None;
+ }
+ Some(
+ DeltaSparkScan::decode(contrib.value.as_slice())
+ .map_err(|e| {
+ GeneralError(format!(
+ "Failed to decode DeltaSparkScan from contrib_scan: {e}"
+ ))
+ })
+ .and_then(|scan| plan_delta_spark_scan(planner, spark_plan, &scan)),
+ )
+}
+
+fn plan_delta_spark_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ scan: &DeltaSparkScan,
+) -> PlanCreationResult {
+ // Delta data files are plain parquet; the read path deliberately reuses
+ // the same shared parquet scan builder as NativeScan so Delta inherits
+ // row-group stats pruning, page-index pruning, and filter pushdown. Only
+ // the file list arrives in Delta-specific form. Note delta_common's
+ // column_mapping_mode is informational in M1: the actual field-id
+ // matching switch is common.use_field_id, same as the Iceberg path.
+ let common = scan
+ .common
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing common data".into()))?;
+
+ let delta_partition = scan
+ .file_partition
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing file_partition".into()))?;
+
+ let spark_partition = SparkFilePartition {
+ partitioned_file: delta_partition
+ .partitioned_file
+ .iter()
+ .map(|f| {
+ f.file
+ .clone()
+ .ok_or_else(|| GeneralError("DeltaSparkPartitionedFile missing inner file".into()))
+ })
+ .collect::<Result<Vec<_>, _>>()?,
+ };
+
+ let (object_store_url, mut files) =
+ planner.prepare_scan_store_and_files(common, &spark_partition)?;
Review Comment:
Done. The scan now declines any selected data file or DV URI whose authority carries userinfo, which covers every abfss container@account path, so these scans stay on Spark until the shared cache preserves container identity. This replaced the per-scan collision tracking (it was a strict subset); the native check stays as a last line of defense and its error messages now redact userinfo too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5375342431
> Hey @sunchao / @parthchandra — are you looking to move this work over to @dwsmith1983’s series?
>
> We’re 2 PRs into the 10-PR series now that the contrib modules have been merged. The series fully implements all of the Delta protocols, with all 10k+ Delta test cases passing.
>
> I’m fine either way. I won’t have a huge amount of time to work on this over the next couple of months, so it could go faster with a different attendant, but the PRs are ready to roll.
>
> You can see the series here: https://github.com/schenksj/datafusion-comet/pulls (PRs #5–13).
I see value in both (even though it is extra work to maintain both paths) and in principle agree with @sunchao. Ideally, we want to converge these two. Logged an issue based on an AI generated convergence path - https://github.com/apache/datafusion-comet/issues/5411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3831203484
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ val common = commonBuilder.build()
+ val tableRoot = relation.location.rootPaths.head.toString
+ val deltaCommon = OperatorOuterClass.DeltaSparkScanCommon
+ .newBuilder()
+ .setTableRoot(tableRoot)
+ .setColumnMappingMode(columnMappingMode(scanExec))
+ .setSourceKey(DeltaPlanDataInjector.sourceKey(tableRoot, common))
+ .build()
+ val deltaScan = OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaCommon)
+ Operator
+ .newBuilder()
+ .setPlanId(scanExec.id)
+ .setContribScan(DeltaSparkScanEnvelope.pack(deltaScan.build()))
+ .build()
+ }
+ }
+
+ /**
+ * Harvest subquery-bearing predicates for this scan from its covering FilterExec. Spark 3.x
+ * strips them from a scan's `dataFilters` at planning (`FileSourceStrategy` routes them to the
+ * post-scan filter only), while Spark 4.x keeps them in `dataFilters`. Collecting them here at
+ * claim time gives the execution-time resolve-and-push path the same inputs on every Spark
+ * version; the dedup keeps Spark 4.x from carrying duplicates.
+ */
+ def subqueryFiltersFromParent(
+ plan: org.apache.spark.sql.execution.SparkPlan,
+ scanExec: FileSourceScanExec): Seq[org.apache.spark.sql.catalyst.expressions.Expression] = {
+ import org.apache.spark.sql.catalyst.expressions.{PlanExpression, SubqueryExpression}
+ // Nearest FilterExec above the scan (the DV shape interposes nodes between them, so do
+ // not require a direct parent-child edge). Safety comes from the reference guard, not the
+ // plan walk: only conjuncts expressed directly over the scan's own output attributes
+ // survive, so filters from other branches or over aliased projections contribute nothing.
+ val filtersAboveScan = plan.collect {
+ case f: org.apache.spark.sql.execution.FilterExec if f.find(_ eq scanExec).isDefined => f
+ }
+ filtersAboveScan.lastOption
+ .map { f =>
+ splitConjunctivePredicates(f.condition)
+ .filter(_.references.subsetOf(scanExec.outputSet))
Review Comment:
Fixed. The spine walk now requires every projection expression to be deterministic before crossing a ProjectExec, mirroring PushPredicateThroughNonJoin's guard, and intermediate filters get the same check on their condition as belt and braces (near unreachable today since the harvest picks the lowest qualifying filter, but it makes the invariant explicit). Your exact query is a regression test with rowFilterPushdown enabled: result is ID 1, the scan stays native, and the test asserts the predicate was not harvested. One deliberate non-change: a nondeterministic conjunct in the same FilterExec still lets its deterministic siblings push, matching FileSourceStrategy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3874383692
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = relation.partitionSchema,
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme (finding 8).
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
Review Comment:
Good case. The comparator resolved both sides against the original conf, but real S3A propagates bucket options before substituting, so a reference can pick up a bucket override that native never sees. The Hadoop side of the comparison now runs on a propagated view, replicated from S3AUtils since hadoop-aws isn't on the runtime classpath, with a test that runs the real S3AUtils next to the replica to catch drift. Your shape declines now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5419675562
On the unsigned small integer safety fallback:
Fixed. The claim path now runs core's CometScanTypeChecker over the required and partition schemas before any other gate, so the unsigned-small-int safety fallback, plus the collation and variant checks, apply identically to Delta scans. Regressions: a SMALLINT Delta table declines with the safety check on (answers match Spark) and reads natively with it off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3861288117
##########
contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/CometDeltaNativeScan.scala:
##########
@@ -0,0 +1,539 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.contrib.delta
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.comet.{CometScanExec, DeltaPlanDataInjector}
+import org.apache.spark.sql.delta.DeltaParquetFileFormat
+import org.apache.spark.sql.delta.RowIndexFilterType
+import org.apache.spark.sql.delta.actions.DeletionVectorDescriptor
+import org.apache.spark.sql.execution.{FileSourceScanExec, ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.datasources.{FilePartition, PartitionedFile}
+import org.apache.spark.sql.types.{ByteType, LongType, MetadataBuilder, StructField, StructType}
+
+import org.apache.comet.objectstore.NativeConfig
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.{exprToProto, serializeDataType}
+import org.apache.comet.serde.operator.{literalToProto, partition2Proto, schema2Proto, CometNativeScan}
+import org.apache.comet.shims.ShimFileFormat
+
+/**
+ * Serde for the native Delta scan. Two shapes:
+ * - Plain reads reuse core's `NativeScanCommon` builder wholesale.
+ * - Deletion-vector reads: Delta's planner appends `__delta_internal_is_row_deleted` (tinyint)
+ * and Spark's row-index temp column (bigint) to the read schema and filters on is_row_deleted
+ * above the scan. The native reader applies the DV as a row selection, so surviving rows are
+ * by construction not deleted: both internal columns are emitted as per-file constants (0),
+ * the parquet read schema is stripped to the real data columns, and the DV descriptor ships
+ * per file for the native side to fetch and decode.
+ */
+object CometDeltaNativeScan
+ extends Logging
+ with org.apache.spark.sql.catalyst.expressions.PredicateHelper {
+
+ val IsRowDeletedColumn: String = DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME
+ val RowIndexColumn: String = ShimFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME
+
+ private[delta] val internalColumnNames: Set[String] = Set(IsRowDeletedColumn, RowIndexColumn)
+
+ // Prefix for the internal columns' slots in the partition schema, mirroring core's
+ // _comet_metadata_ prefix rationale: DataFusion matches partition columns by name.
+ private val deltaConstFieldPrefix = "_comet_delta_"
+
+ def isDvShape(scanExec: FileSourceScanExec): Boolean =
+ scanExec.requiredSchema.exists(f => internalColumnNames.contains(f.name))
+
+ private def deltaFormat(scanExec: FileSourceScanExec): DeltaParquetFileFormat =
+ scanExec.relation.fileFormat.asInstanceOf[DeltaParquetFileFormat]
+
+ private def columnMappingMode(scanExec: FileSourceScanExec): String =
+ deltaFormat(scanExec).metadata.columnMappingMode.name
+
+ /**
+ * Under column mapping, parquet files store physical column names (stable UUIDs / ids), so the
+ * schemas passed to the native parquet reader must be physical. Positions and structure are
+ * preserved, so all positional output binding and projection are unaffected. The scan's
+ * internal DV columns are not part of the table schema and must be stripped before calling
+ * this.
+ */
+ private def toPhysical(scanExec: FileSourceScanExec, schema: StructType): StructType = {
+ val format = deltaFormat(scanExec)
+ if (format.metadata.columnMappingMode.name == "none") {
+ schema
+ } else {
+ // Name mode matches file columns by physical NAME. createPhysicalSchema also stamps
+ // parquet.field.id metadata, but files written before the column-mapping upgrade have
+ // no field ids and would fail the reader's id expectations, strip the ids so the
+ // reader stays purely name-based (id mode, when enabled, will keep them).
+ stripFieldIds(org.apache.spark.sql.delta.DeltaColumnMapping
+ .createPhysicalSchema(schema, format.metadata.schema, format.metadata.columnMappingMode))
+ }
+ }
+
+ private def stripFieldIds(schema: StructType): StructType = {
+ import org.apache.spark.sql.types._
+ def stripType(dt: DataType): DataType = dt match {
+ case s: StructType => stripFieldIds(s)
+ case a: ArrayType => a.copy(elementType = stripType(a.elementType))
+ case m: MapType =>
+ m.copy(keyType = stripType(m.keyType), valueType = stripType(m.valueType))
+ case other => other
+ }
+ StructType(schema.fields.map { f =>
+ val metadata = new MetadataBuilder()
+ .withMetadata(f.metadata)
+ .remove("parquet.field.id")
+ // Sibling key Delta stamps on array/map fields under IcebergCompat/Uniform.
+ .remove("parquet.field.nested.ids")
+ .build()
+ f.copy(dataType = stripType(f.dataType), metadata = metadata)
+ })
+ }
+
+ /**
+ * Build the planning-time `DeltaScan` operator (common data only; file partitions are injected
+ * lazily at execution). Returns None when an output data type cannot be serialized or the plan
+ * shape is not one we can translate faithfully.
+ */
+ def convert(scanExec: FileSourceScanExec, scanHelper: CometScanExec): Option[Operator] = {
+ val relation = scanExec.relation
+
+ val firstFileUri = scanHelper.selectedPartitions
+ .flatMap(_.files.headOption)
+ .headOption
+ .map(_.getPath.toUri)
+
+ val hadoopConf = relation.sparkSession.sessionState
+ .newHadoopConfWithOptions(relation.options)
+
+ val tableRootPath = relation.location.rootPaths.head
+ val tableRoot = tableRootPath.toString
+
+ val commonOpt = if (!isDvShape(scanExec)) {
+ // Under column mapping (name mode) the parquet reader must see physical names;
+ // positions are preserved so output binding and projection stay untouched.
+ CometNativeScan.buildNativeScanCommon(
+ source = scanExec.simpleStringWithNodeId(),
+ output = scanExec.output,
+ requiredSchema = toPhysical(scanExec, scanExec.requiredSchema),
+ dataSchema = toPhysical(scanExec, relation.dataSchema),
+ partitionSchema = toPhysical(scanExec, relation.partitionSchema),
+ fileConstantMetadataColumns = scanExec.fileConstantMetadataColumns,
+ dataFilters = scanHelper.supportedDataFilters,
+ firstFileUri = firstFileUri,
+ hadoopConf = hadoopConf,
+ conf = scanExec.conf)
+ } else {
+ buildDvScanCommon(scanExec, scanHelper, firstFileUri, hadoopConf)
+ }
+
+ commonOpt.map { commonBuilder =>
+ // Union object-store options over every authority a partition of this scan may need a
+ // store for, not just the first data file's scheme.
+ val dvDescriptors = DeltaScanSupport.selectedDvDescriptors(scanHelper, tableRoot)
+ commonBuilder.putAllObjectStoreOptions(
+ mergedObjectStoreOptions(
+ hadoopConf,
+ storeUris(dvDescriptors, tableRootPath, firstFileUri)).asJava)
+
+ val common = commonBuilder.build()
+ val deltaCommon = OperatorOuterClass.DeltaSparkScanCommon
+ .newBuilder()
+ .setTableRoot(tableRoot)
+ .setColumnMappingMode(columnMappingMode(scanExec))
+ .setSourceKey(DeltaPlanDataInjector.sourceKey(tableRoot, common))
+ .build()
+ val deltaScan = OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaCommon)
+ Operator
+ .newBuilder()
+ .setPlanId(scanExec.id)
+ .setContribScan(DeltaSparkScanEnvelope.pack(deltaScan.build()))
+ .build()
+ }
+ }
+
+ /**
+ * One representative store URI per distinct object-store authority this scan's partitions may
+ * need options for: the data-file authority (`firstFileUri`), the table root unconditionally
+ * (UUID-relative DV sidecars resolve against it, and it is cheap to include even when absent),
+ * and every distinct on-disk DV authority from `descriptors`. Inline DVs are filtered out --
+ * they carry no external URI, only embedded bytes. Deduping by authority (rather than by full
+ * URI) keeps this O(distinct authorities) instead of O(files): a table with N deletion-vector
+ * files on the same external store previously produced ~N distinct URIs here, each
+ * independently fed into `mergedObjectStoreOptions`'s `extractObjectStoreOptions` walk over
+ * `hadoopConf`. Candidates are deduped keeping the FIRST URI seen per authority, so callers can
+ * rely on `firstFileUri`/the table root winning over any DV path that happens to share their
+ * authority. Factored out of [[convert]] so the URI-assembly logic (in particular the
+ * `storageType` filter and `absolutePath` resolution) is directly unit-testable with hand-built
+ * [[DeletionVectorDescriptor]] fixtures, without a Spark session or real selected files (a
+ * `file://` scan alone can't exercise a foreign-authority DV).
+ */
+ private[delta] def storeUris(
+ descriptors: Seq[DeletionVectorDescriptor],
+ tableRootPath: Path,
+ firstFileUri: Option[java.net.URI]): Seq[java.net.URI] = {
+ val dvAuthorityUris = descriptors
+ .filter(_.storageType != DeletionVectorDescriptor.INLINE_DV_MARKER)
+ .map(_.absolutePath(tableRootPath).toUri)
+ val candidates = firstFileUri.toSeq ++ Seq(tableRootPath.toUri) ++ dvAuthorityUris
+ val byAuthority = scala.collection.mutable.LinkedHashMap.empty[String, java.net.URI]
+ candidates.foreach(uri =>
+ byAuthority.getOrElseUpdate(DeltaScanSupport.uriAuthority(uri), uri))
+ byAuthority.values.toSeq
+ }
+
+ /**
+ * Unions `NativeConfig.extractObjectStoreOptions` over every `uris` authority. Safe to simply
+ * union rather than pick one: the extracted keys are scheme-disjoint prefixes (`fs.s3a.*` vs
+ * `fs.azure.*`, ...), so options for different schemes never collide, and re-extracting the
+ * same scheme from two URIs is idempotent. Factored out of [[convert]] so it is directly
+ * unit-testable without a Spark session.
+ */
+ private[delta] def mergedObjectStoreOptions(
+ hadoopConf: org.apache.hadoop.conf.Configuration,
+ uris: Seq[java.net.URI]): Map[String, String] =
+ uris.foldLeft(Map.empty[String, String]) { (merged, uri) =>
+ merged ++ NativeConfig.extractObjectStoreOptions(hadoopConf, uri)
+ }
+
+ /**
+ * Harvest subquery-bearing predicates for this scan from its covering FilterExec. Spark 3.x
+ * strips them from a scan's `dataFilters` at planning (`FileSourceStrategy` routes them to the
+ * post-scan filter only), while Spark 4.x keeps them in `dataFilters`. Collecting them here at
+ * claim time gives the execution-time resolve-and-push path the same inputs on every Spark
+ * version; the dedup keeps Spark 4.x from carrying duplicates.
+ *
+ * Safety comes from the plan walk, not just the reference guard: a filter is only harvested
+ * when every operator between it and the scan commutes with pushing the predicate into the scan
+ * (see `spineToScan`). Reference containment alone proves the predicate is expressible over the
+ * scan's output, not that moving it there is semantics-preserving -- an intervening LIMIT/TopN
+ * (or Sort, Aggregate, Window, join, ...) can change which rows the predicate would have
+ * applied to, so those stop the walk and the filter is left where Spark placed it.
+ */
+ def subqueryFiltersFromParent(
+ plan: org.apache.spark.sql.execution.SparkPlan,
+ scanExec: FileSourceScanExec): Seq[org.apache.spark.sql.catalyst.expressions.Expression] = {
+ import org.apache.spark.sql.catalyst.expressions.{PlanExpression, SubqueryExpression}
+ import org.apache.spark.sql.execution.{FilterExec, ProjectExec, SparkPlan}
+
+ // Whether every node on the path from `node` down to `scanExec` is one that pushdown can
+ // safely cross: a deterministic ProjectExec is 1:1 on rows (an Alias mints a new exprId, so
+ // it cannot alias over the scan's own output attributes, which is what the reference guard
+ // below requires) and a deterministic FilterExec only removes rows, so moving a predicate
+ // expressed over the scan's output through either preserves the query's semantics. A
+ // nondeterministic projection breaks that: a deterministic conjunct does not commute with it,
+ // because pushing the predicate into the scan changes which rows survive to have
+ // nondeterministic expressions (e.g. monotonically_increasing_id()) evaluated over them,
+ // changing the result -- so both guards require `.deterministic`, mirroring Spark's own
+ // PushPredicateThroughNonJoin/CollapseProject rules. Anything else (LIMIT/TopN, Sort,
+ // Aggregate, Window, joins, Union, Sample, ...) can reorder or drop rows in ways that make
+ // "push the predicate down to the scan" change the result, so an unrecognized node stops the
+ // walk: the filter is left uncollected (missed pruning only, never a correctness issue).
+ def spineToScan(node: SparkPlan): Boolean = node match {
+ case n if n eq scanExec => true
+ case p: ProjectExec if p.projectList.forall(_.deterministic) => spineToScan(p.child)
+ case f: FilterExec if f.condition.deterministic => spineToScan(f.child)
+ case _ => false
+ }
+
+ // Nearest FilterExec whose spine down to the scan is Project/Filter-only (the DV shape
+ // interposes such nodes between them, so do not require a direct parent-child edge).
+ val filtersAboveScan = plan.collect {
+ case f: FilterExec if spineToScan(f.child) => f
+ }
+ filtersAboveScan.lastOption
+ .map { f =>
+ splitConjunctivePredicates(f.condition)
+ .filter(_.deterministic)
+ .filter(_.references.subsetOf(scanExec.outputSet))
+ .filter(p =>
+ SubqueryExpression.hasSubquery(p) || p.exists(_.isInstanceOf[PlanExpression[_]]))
+ .filterNot(p => scanExec.dataFilters.exists(_.semanticEquals(p)))
+ }
+ .getOrElse(Seq.empty)
+ }
+
+ /**
+ * Resolve scalar-subquery data filters at execution time and serialize them for native
+ * pushdown, mirroring `CometNativeScanExec.serializedPartitionData`. `supportedDataFilters`
+ * excludes PlanExpressions at planning time (subquery results do not exist yet), so these
+ * bounds reach the native reader only through this path. Filters that fail to serialize are
+ * skipped: Spark keeps a covering FilterExec above the scan, so this is missed pruning only,
+ * never a correctness issue.
+ *
+ * Known core-parity limitation: when the scan is fused under a parent native operator,
+ * `ensureSubqueriesResolved` has already called `updateResult()` on these subqueries and this
+ * path calls it again (Spark's ScalarSubquery.updateResult re-executes unconditionally). Core's
+ * CometNativeScanExec has the identical double-execution; benign for Delta (the subquery's
+ * snapshot is pinned at analysis), wasteful for expensive subqueries. Fix belongs in core.
+ */
+ def resolvedSubqueryFilters(
+ dataFilters: Seq[org.apache.spark.sql.catalyst.expressions.Expression],
+ output: Seq[org.apache.spark.sql.catalyst.expressions.Attribute],
+ requiredSchema: StructType,
+ conf: org.apache.spark.sql.internal.SQLConf)
+ : Seq[org.apache.comet.serde.ExprOuterClass.Expr] = {
+ if (!conf.getConf(org.apache.spark.sql.internal.SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ return Seq.empty
+ }
+ val subqueryFilters = dataFilters.filter(_.exists(_.isInstanceOf[ExecScalarSubquery]))
+ if (subqueryFilters.isEmpty) {
+ return Seq.empty
+ }
+ // Same binding guard as the DV shape's plan-time filters: references limited to the
+ // data-column prefix of the output, where positions agree between the output and the
+ // native read schema. For the plain shape strippedLen is the full required schema.
+ // Guard BEFORE updateResult so discarded filters never execute their subqueries.
+ val strippedLen = requiredSchema.count(f => !internalColumnNames.contains(f.name))
+ val dataColIds = output.take(strippedLen).map(_.exprId).toSet
+ val pushableFilters =
+ subqueryFilters.filter(_.references.forall(r => dataColIds.contains(r.exprId)))
+ pushableFilters.foreach(_.foreach {
+ case s: ExecScalarSubquery => s.updateResult()
+ case _ =>
+ })
+ pushableFilters
+ .flatMap { filter =>
+ // MergeScalarSubqueries can fuse several scalar subqueries into one struct-returning
+ // subquery accessed via GetStructField; fold that whole subtree to a literal (a bare
+ // GetStructField-over-Literal would not serialize).
+ val resolved = filter.transform {
+ case g @ org.apache.spark.sql.catalyst.expressions
+ .GetStructField(_: ExecScalarSubquery, _, _) =>
+ Literal.create(g.eval(null), g.dataType)
+ case s: ExecScalarSubquery =>
+ Literal.create(s.eval(null), s.dataType)
+ }
+ val proto = exprToProto(resolved, output)
+ if (proto.isEmpty) {
+ logWarning(s"Could not serialize resolved scalar subquery filter: $resolved")
+ }
+ proto
+ }
+ }
+
+ /**
+ * DV shape common builder. Layout invariants (declined by DeltaScanSupport when violated): scan
+ * output = requiredSchema attrs (data columns, then the internal columns as a suffix) followed
+ * by partition and constant-metadata columns. The parquet read schema strips the internal
+ * columns; they are appended to the partition schema as per-file constants, so the projection
+ * vector routes them from the constants block.
+ */
+ private def buildDvScanCommon(
+ scanExec: FileSourceScanExec,
+ scanHelper: CometScanExec,
+ firstFileUri: Option[java.net.URI],
+ hadoopConf: org.apache.hadoop.conf.Configuration)
+ : Option[OperatorOuterClass.NativeScanCommon.Builder] = {
+ val relation = scanExec.relation
+ val output = scanExec.output
+ val requiredSchema = scanExec.requiredSchema
+
+ val commonBuilder = OperatorOuterClass.NativeScanCommon.newBuilder()
+ commonBuilder.setSource(scanExec.simpleStringWithNodeId())
+
+ val scanTypes = output.flatMap(attr => serializeDataType(attr.dataType))
+ if (scanTypes.length != output.length) {
+ return None
+ }
+ commonBuilder.addAllFields(scanTypes.asJava)
+
+ val strippedRequired =
+ StructType(requiredSchema.filterNot(f => internalColumnNames.contains(f.name)))
+ val strippedLen = strippedRequired.length
+ val requiredLen = requiredSchema.length
+
+ // Keep only data filters that bind identically in the output and the native
+ // (strippedRequired ++ partitionFields) index spaces: references limited to the first
+ // strippedLen output attributes. Internal-column filters (is_row_deleted = 0) are
+ // trivially true after native DV application, and Spark's Filter above the scan
+ // re-evaluates everything anyway.
+ if (scanExec.conf.getConf(
+ org.apache.spark.sql.internal.SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ val dataColIds = output.take(strippedLen).map(_.exprId).toSet
+ val filterProtos = scanHelper.supportedDataFilters
+ .filter(_.references.forall(r => dataColIds.contains(r.exprId)))
+ .flatMap(f => exprToProto(f, output))
+ commonBuilder.addAllDataFilters(filterProtos.asJava)
+ }
+
+ // Constant metadata columns and real partition columns follow the required schema in the
+ // output, exactly like the plain shape.
+ val constantMetadataFields = scanExec.fileConstantMetadataColumns.map(attr =>
+ StructField(
+ s"${CometNativeScan.constantMetadataFieldPrefix}${attr.name}",
+ attr.dataType,
+ attr.nullable))
+ val internalFields = requiredSchema.fields.toSeq
+ .filter(f => internalColumnNames.contains(f.name))
+ .map(f => StructField(s"$deltaConstFieldPrefix${f.name}", f.dataType, f.nullable))
Review Comment:
Fixed. The internal bookkeeping constants are now allocated collision-free against the physical data and partition namespaces, and while in there we gave the file-metadata constants the same protection in the shared core builder, so plain reads are covered too. Your colliding-column shape reproduced the 7 to 0 corruption locally and is covered in the differential suite end to end plus at the unit level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5371428219
Hi @schenksj , I think your series implements Delta native scan based on the `delta-kernel-rs` while the PR here uses the JVM based `delta-spark` for planning, so they are different while both are based on the same contrib groundwork.
I think your series is pretty valuable and should be continued to push forward. At some point we should compare feature coverage and performance between the two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5322185992
Hi @dwsmith1983 Thanks for putting this together! We are also actively looking at Delta support for Comet, and it'd be great if we can collaborate on this effort!
Since #4952 is already approved and close to landing, what do you think about using it as the shared foundation for this work? Ideally, the same `contrib` infrastructure could support both JVM-planned Delta scans and the Rust Kernel-based approach, with this PR providing the JVM-planned path. We have related work in progress, so it would be good to converge on one implementation.
In addition, would it also make sense to land this in smaller pieces, for easier review and iterating? For example:
- Basic native Delta reads, including time travel and fallback for unsupported features
- Column mapping and schema evolution
- Deletion vectors
- Row tracking
- Change Data Feed
Starting to support this in Spark 4 & Delta 4 would be a useful first milestone. Curious how you see the relationship between the two PRs and whether that direction makes sense to you. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3826299516
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -0,0 +1,587 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Delta Lake deletion-vector decoding and translation into DataFusion
+//! [`ParquetAccessPlan`]s (feature = "delta").
+//!
+//! Wire formats implemented here (from delta-spark's `DeletionVectorStore` /
+//! `RoaringBitmapArray`, v3.3.2):
+//! - On-disk DV file: 1 version byte at the start of the file; at
+//! `descriptor.offset`: `[i32 BE size][data: size bytes][i32 BE CRC32(data)]`.
+//! - `data`: `[i32 LE magic]` then either
+//! - magic 1681511376 ("native"): `[i32 LE count]`, then per bitmap
+//! `[i32 LE size][standard 32-bit RoaringBitmap]`, keys implicit (index);
+//! - magic 1681511377 ("portable", the spec's 64-bit extension): `[i64 LE
+//! count]`, then per bitmap `[i32 LE key][standard 32-bit RoaringBitmap]`
+//! with keys ascending -- exactly [`RoaringTreemap`]'s serialized form.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::datasource::listing::PartitionedFile;
+use datafusion::datasource::physical_plan::parquet::metadata::DFParquetMetadata;
+use datafusion::datasource::physical_plan::parquet::ParquetAccessPlan;
+use datafusion::execution::runtime_env::RuntimeEnv;
+use futures::{StreamExt, TryStreamExt};
+use object_store::ObjectStoreExt;
+use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
+use parquet::file::metadata::PageIndexPolicy;
+use roaring::{RoaringBitmap, RoaringTreemap};
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::parquet::parquet_support::prepare_object_store_with_configs;
+use datafusion_comet_proto::spark_operator::DeltaSparkDvDescriptor;
+
+const NATIVE_MAGIC: i32 = 1681511376;
+const PORTABLE_MAGIC: i32 = 1681511377;
+
+/// Unframe a DV blob read from `descriptor.offset` of a DV file:
+/// `[i32 BE size][data][i32 BE crc]`. Verifies both the size against the
+/// descriptor's `size_in_bytes` and the CRC32 checksum.
+pub fn unframe_dv_blob(blob: &[u8], expected_size: usize) -> Result<&[u8], ExecutionError> {
+ if blob.len() < 8 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob too short: {} bytes",
+ blob.len()
+ )));
+ }
+ let size = i32::from_be_bytes(blob[0..4].try_into().unwrap());
+ if size < 0 || size as usize != expected_size {
+ return Err(GeneralError(format!(
+ "Deletion vector size mismatch: file says {size}, descriptor says {expected_size}"
+ )));
+ }
+ let end = 4 + size as usize;
+ if blob.len() < end + 4 {
+ return Err(GeneralError(format!(
+ "Deletion vector blob truncated: need {} bytes, have {}",
+ end + 4,
+ blob.len()
+ )));
+ }
+ let data = &blob[4..end];
+ let expected_crc = i32::from_be_bytes(blob[end..end + 4].try_into().unwrap());
+ let actual_crc = crc32fast::hash(data) as i32;
+ if expected_crc != actual_crc {
+ return Err(GeneralError(
+ "Deletion vector checksum mismatch".to_string(),
+ ));
+ }
+ Ok(data)
+}
+
+/// Deserialize the magic-prefixed RoaringBitmapArray into a 64-bit treemap of
+/// deleted row indexes.
+pub fn deserialize_dv_bitmap(data: &[u8]) -> Result<RoaringTreemap, ExecutionError> {
+ if data.len() < 4 {
+ return Err(GeneralError(
+ "Deletion vector bitmap too short for magic number".to_string(),
+ ));
+ }
+ let magic = i32::from_le_bytes(data[0..4].try_into().unwrap());
+ let rest = &data[4..];
+ match magic {
+ PORTABLE_MAGIC => RoaringTreemap::deserialize_from(rest)
+ .map_err(|e| GeneralError(format!("Invalid portable deletion vector bitmap: {e}"))),
+ NATIVE_MAGIC => {
+ if rest.len() < 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap missing count".to_string(),
+ ));
+ }
+ let count = i32::from_le_bytes(rest[0..4].try_into().unwrap());
+ if count < 0 {
+ return Err(GeneralError(format!(
+ "Invalid RoaringBitmapArray length ({count} < 0)"
+ )));
+ }
+ let mut pos = 4usize;
+ let mut treemap = RoaringTreemap::new();
+ for key in 0..count as u64 {
+ if rest.len() < pos + 4 {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let size = i32::from_le_bytes(rest[pos..pos + 4].try_into().unwrap());
+ pos += 4;
+ if size < 0 || rest.len() < pos + size as usize {
+ return Err(GeneralError(
+ "Native deletion vector bitmap truncated".to_string(),
+ ));
+ }
+ let bitmap = RoaringBitmap::deserialize_from(&rest[pos..pos + size as usize])
+ .map_err(|e| {
+ GeneralError(format!("Invalid deletion vector sub-bitmap: {e}"))
+ })?;
+ pos += size as usize;
+ for value in bitmap {
+ treemap.insert((key << 32) | value as u64);
+ }
+ }
+ Ok(treemap)
+ }
+ other => Err(GeneralError(format!(
+ "Unexpected RoaringBitmapArray magic number {other}"
+ ))),
+ }
+}
+
+/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
+/// row groups become `Skip`, untouched groups stay `Scan`, and partially
+/// deleted groups get a `RowSelection` selecting the complement of the deleted
+/// rows. Page-index pruning later INTERSECTS with these selections, so DV
+/// skips and page skips compose.
+pub fn build_access_plan(
+ row_group_row_counts: &[i64],
+ deleted: &RoaringTreemap,
+) -> Result<ParquetAccessPlan, ExecutionError> {
+ let mut plan = ParquetAccessPlan::new_all(row_group_row_counts.len());
+ // Single sweep over the (sorted) deleted row indexes, bucketing by row group.
+ let mut deleted_iter = deleted.iter().peekable();
+ let mut group_start = 0u64;
+ for (idx, &num_rows) in row_group_row_counts.iter().enumerate() {
+ let num_rows = num_rows as u64;
+ let group_end = group_start + num_rows;
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut cursor = group_start;
+ let mut deleted_in_group = 0u64;
+ while let Some(&row) = deleted_iter.peek() {
+ if row >= group_end {
+ break;
+ }
+ deleted_iter.next();
+ deleted_in_group += 1;
+ if row > cursor {
+ selectors.push(RowSelector::select((row - cursor) as usize));
+ }
+ // Merge runs of consecutive deleted rows into one skip.
+ match selectors.last_mut() {
+ Some(last) if last.skip => last.row_count += 1,
+ _ => selectors.push(RowSelector::skip(1)),
+ }
+ cursor = row + 1;
+ }
+ if deleted_in_group == num_rows && num_rows > 0 {
+ plan.skip(idx);
+ } else if deleted_in_group > 0 {
+ if group_end > cursor {
+ selectors.push(RowSelector::select((group_end - cursor) as usize));
+ }
+ plan.scan_selection(idx, RowSelection::from(selectors));
+ }
+ group_start = group_end;
+ }
+ // A deleted index beyond the file's total row count means the DV does not
+ // belong to this file (stale or corrupted metadata); silently dropping it
+ // would under-apply deletions.
+ if let Some(&row) = deleted_iter.peek() {
+ return Err(GeneralError(format!(
+ "Deletion vector marks row {row} but the file only has {group_start} rows"
+ )));
+ }
+ Ok(plan)
+}
+
+/// One data file plus everything needed to apply its deletion vector. The
+/// file's size comes from `file.object_meta.size` (built by the planner from
+/// the proto's `file_size`).
+pub struct DvScanFile {
+ pub file: PartitionedFile,
+ /// Full URL of the data file (proto `file_path`).
+ pub file_path: String,
+ pub dv: Option<DeltaSparkDvDescriptor>,
+}
+
+/// Upper bound on concurrent DV-blob and footer fetches per partition. Both
+/// are small ranged reads, so a modest fan-out hides object-store latency
+/// without flooding the store client.
+const DV_FETCH_CONCURRENCY: usize = 8;
+
+/// Called via `block_on` at plan-creation time on the executor task: DV blobs
+/// are small ranged reads and footers are needed to learn row-group
+/// boundaries. Files are fetched concurrently (bounded by
+/// [`DV_FETCH_CONCURRENCY`]) with input order preserved. Footer fetches go
+/// through the scan's shared FileMetadataCache, so the scan's subsequent open
+/// of the same file is served from cache. That reuse relies on each input
+/// [`PartitionedFile`] being returned as-is (only `with_extension` applied),
+/// never rebuilt: the cache entry is keyed by this exact `object_meta` and the
+/// scan later looks it up through the same struct.
+pub async fn attach_access_plans(
+ runtime_env: Arc<RuntimeEnv>,
+ object_store_options: &HashMap<String, String>,
+ files: Vec<DvScanFile>,
+) -> Result<Vec<PartitionedFile>, ExecutionError> {
+ futures::stream::iter(files)
+ .map(|scan_file| {
+ attach_access_plan(Arc::clone(&runtime_env), object_store_options, scan_file)
+ })
+ .buffered(DV_FETCH_CONCURRENCY)
+ .try_collect()
+ .await
+}
+
+/// Resolve one file's deletion vector into an attached [`ParquetAccessPlan`];
+/// files without a DV pass through untouched.
+async fn attach_access_plan(
+ runtime_env: Arc<RuntimeEnv>,
+ object_store_options: &HashMap<String, String>,
+ scan_file: DvScanFile,
+) -> Result<PartitionedFile, ExecutionError> {
+ let DvScanFile {
+ file,
+ file_path,
+ dv,
+ } = scan_file;
+ let dv = match dv {
+ Some(dv) => dv,
+ None => return Ok(file),
+ };
+ if dv.size_in_bytes < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative size {}",
+ dv.size_in_bytes
+ )));
+ }
+
+ let data: Vec<u8> = if let Some(inline) = dv.inline_data {
+ inline
+ } else if let Some(dv_path) = &dv.absolute_path {
+ let offset = dv
+ .offset
+ .ok_or_else(|| GeneralError("On-disk deletion vector missing offset".into()))?;
+ if offset < 0 {
+ return Err(GeneralError(format!(
+ "Deletion vector for {file_path} has negative offset {offset}"
+ )));
+ }
+ let offset = offset as u64;
+ // [i32 BE size][data: size_in_bytes][i32 BE crc]
+ let framed_len = 4 + dv.size_in_bytes as u64 + 4;
+ let (dv_url, dv_store_path) = prepare_object_store_with_configs(
+ Arc::clone(&runtime_env),
+ dv_path.clone(),
+ object_store_options,
+ )?;
+ let store = runtime_env.object_store(&dv_url)?;
+ let blob = store
+ .get_range(&dv_store_path, offset..offset + framed_len)
+ .await
+ .map_err(|e| GeneralError(format!("Failed to read deletion vector {dv_path}: {e}")))?;
+ unframe_dv_blob(&blob, dv.size_in_bytes as usize)?.to_vec()
+ } else {
+ return Err(GeneralError(
+ "Deletion vector descriptor has neither inline data nor a path".into(),
+ ));
+ };
+ let deleted = deserialize_dv_bitmap(&data)
+ .map_err(|e| GeneralError(format!("Invalid deletion vector for {file_path}: {e}")))?;
Review Comment:
Fixed as suggested: decoded cardinality is validated against the descriptor right after decode, mirroring `StoredBitmap.validateCardinality`, with negative cardinality rejected alongside the existing negative-size check. The mismatch test sits next to the framing/CRC cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5535608903
@andygrove could you take another look when you have a chance? Since your last review the PR was restructured along the lines you asked for: the case folding is gone in favor of #5602, the core fixes went out as #5653 (now merged), and the field id semantics are in #5654. The branch is a single commit on current main, sunchao approved at the previous head, and CI is green apart from the runs still waiting on authorization. Your changes-requested predates the restructure, so I wanted to make sure it is not holding things up by accident.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3939357361
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so its rewrite (via
+ * [[CometScanWithPlanData]]) survives plan copies -- a transient field would be dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning (the CometIcebergNativeScanExec lesson).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node
+ // automatically gets a helper consistent with ITS runtimeFilters, avoiding the #3510 class of
+ // bug where a stored helper field desyncs from rewritten filters. Costs one extra file listing
+ // per executed instance; correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below, and AQE's UI
+ // plan-walk calls `.metrics` on every node MID-PLANNING, including while a DPP subquery is
+ // still an adaptive placeholder or a partition filter holds an unresolved ScalarSubquery (see
+ // `hasUnevaluableSubqueryFilter` below). That's safe ONLY because constructing `scanHelper` is a
+ // cheap case-class build with no file listing, and core's `CometScanExec.metrics` touches only
+ // `wrapped.driverMetrics` (populated by Spark's own planning) plus a static metric-node
+ // constructor -- neither file listing nor subquery resolution. If core's `metrics` ever touches
+ // either, forcing `scanHelper` here would resurrect the AQE mid-planning crashes this invariant
+ // prevents.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element, so checking runtimeFilters
+ // here avoids constructing/forcing the derived scanHelper just to read partitioning. The
+ // InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep in sync. The
+ // ScalarSubquery case is probed rather than treated as permanently unevaluable: Spark exposes no
+ // public finished/updated flag on ExecSubqueryExpression, but `eval()` doubles as one -- it only
+ // reads the cached `result` behind a `require(updated, ...)` guard, while the subquery is
+ // actually run by `updateResult()` (invoked separately during prepare/AQE), never by `eval()`.
+ // Once resolved, outputPartitioning below reports the real perPartitionData.length instead of
+ // staying at zero -- a fused native parent's buildNativeContext requires that count to match.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case s: ScalarSubquery => !isScalarSubqueryResolved(s)
+ case _ => false
+ })
+
+ // `eval()` never triggers the subquery's execution: on a resolved subquery it is a pure cached
+ // read of `result` (verified against bytecode: `Predef.require(updated(), ...)` then a plain
+ // field read), so this probe is safe to call repeatedly, including from AQE's mid-planning plan
+ // walks. Pre-resolution, the ONLY throw is `require`'s `IllegalArgumentException`; catch exactly
+ // that, since anything else escaping is a genuine bug we must not mask as unpartitioned.
+ private def isScalarSubqueryResolved(s: ScalarSubquery): Boolean =
+ try {
+ s.eval()
+ true
+ } catch {
+ case _: IllegalArgumentException => false
+ }
+
+ private def isAdaptivePlaceholder(p: SparkPlan): Boolean = p match {
+ case ReusedSubqueryExec(inner) => isAdaptivePlaceholder(inner)
+ case _: CometSubqueryAdaptiveBroadcastExec => true
+ case _: SubqueryAdaptiveBroadcastExec => true
+ case _ => false
+ }
+
+ override lazy val outputOrdering: Seq[SortOrder] = originalPlan.outputOrdering
+
+ override def dynamicPruningFilters: Seq[Expression] = runtimeFilters
+
+ override def withDynamicPruningFilters(filters: Seq[Expression]): SparkPlan = {
+ // A real copy: runtimeFilters is a constructor field included in equality, so the copy
+ // survives enclosing-block rebuilds, and the derived scanHelper picks up the rewritten
+ // filters automatically.
+ copy(runtimeFilters = filters)
+ }
+
+ /**
+ * Lazy split-mode serialization, mirroring CometNativeScanExec: common data was serialized at
+ * planning; per-partition file lists serialize here, at execution time.
+ */
+ @transient private lazy val serializedPartitionData
+ : (Array[Byte], Array[Array[Byte]], Array[Seq[String]]) = {
+ // Resolve the helper's DPP subqueries: it holds its own InSubqueryExec instances that
+ // Spark's expressions walk does not see (the helper is derived, not a child).
+ scanHelper.partitionFilters.foreach {
+ case DynamicPruningExpression(e: InSubqueryExec) if e.values().isEmpty =>
+ e.updateResult()
+ case _ =>
+ }
+
+ val commonBytes = {
+ val deltaScan = DeltaSparkScanEnvelope.unpack(nativeOp)
+ // Scalar subqueries in dataFilters were unresolved at planning; resolve them now and
+ // append them as pushed filters, as CometNativeScanExec.serializedPartitionData does.
+ val subqueryFilters = org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .resolvedSubqueryFilters(dataFilters, output, requiredSchema, conf)
+ val common = if (subqueryFilters.isEmpty) {
+ deltaScan.getCommon
+ } else {
+ val builder = deltaScan.getCommon.toBuilder
+ builder.setHasDataFilters(true)
Review Comment:
### Correctness
[P2] Preserve runtime filter presence when serialization fails
`resolvedSubqueryFilters` returns only successfully serialized protos, so this condition leaves `has_data_filters=false` when every resolved filter is rejected. The core scan sets the bit before serialization. With this rebase, the difference enables `checked_timestamp_overflow` through `build_parquet_scan_plan`.
A source-derived case is a non-DV Spark 4.0 Delta scan with only `ts <=> (SELECT max(bound) FROM bounds)` and `spark.comet.expression.EqualNullSafe.enabled=false`. Use nullable `ts` stored as TIMESTAMP_MILLIS, positive overflowing values (e.g. 9,223,372,036,854,776), Parquet min/max statistics, no Delta file statistics, CORRECTED read modes, and a persisted bounds table returning the epoch. The subquery stays unresolved at planning and null-safe equality adds no separate `IsNotNull(ts)` signal. Spark resolves and pushes the equality, pruning the row group; Delta drops the proto and presence bit, then raises during millis-to-micros conversion. The covering Spark filter cannot prevent that earlier error.
Please carry the pre-serialization presence bit out of the helper, including when its proto list is empty, and add a regression for this branch. I traced this case through the source; I did not execute the full query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3939434593
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so its rewrite (via
+ * [[CometScanWithPlanData]]) survives plan copies -- a transient field would be dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning (the CometIcebergNativeScanExec lesson).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node
+ // automatically gets a helper consistent with ITS runtimeFilters, avoiding the #3510 class of
+ // bug where a stored helper field desyncs from rewritten filters. Costs one extra file listing
+ // per executed instance; correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below, and AQE's UI
+ // plan-walk calls `.metrics` on every node MID-PLANNING, including while a DPP subquery is
+ // still an adaptive placeholder or a partition filter holds an unresolved ScalarSubquery (see
+ // `hasUnevaluableSubqueryFilter` below). That's safe ONLY because constructing `scanHelper` is a
+ // cheap case-class build with no file listing, and core's `CometScanExec.metrics` touches only
+ // `wrapped.driverMetrics` (populated by Spark's own planning) plus a static metric-node
+ // constructor -- neither file listing nor subquery resolution. If core's `metrics` ever touches
+ // either, forcing `scanHelper` here would resurrect the AQE mid-planning crashes this invariant
+ // prevents.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element, so checking runtimeFilters
+ // here avoids constructing/forcing the derived scanHelper just to read partitioning. The
+ // InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep in sync. The
+ // ScalarSubquery case is probed rather than treated as permanently unevaluable: Spark exposes no
+ // public finished/updated flag on ExecSubqueryExpression, but `eval()` doubles as one -- it only
+ // reads the cached `result` behind a `require(updated, ...)` guard, while the subquery is
+ // actually run by `updateResult()` (invoked separately during prepare/AQE), never by `eval()`.
+ // Once resolved, outputPartitioning below reports the real perPartitionData.length instead of
+ // staying at zero -- a fused native parent's buildNativeContext requires that count to match.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case s: ScalarSubquery => !isScalarSubqueryResolved(s)
+ case _ => false
+ })
+
+ // `eval()` never triggers the subquery's execution: on a resolved subquery it is a pure cached
+ // read of `result` (verified against bytecode: `Predef.require(updated(), ...)` then a plain
+ // field read), so this probe is safe to call repeatedly, including from AQE's mid-planning plan
+ // walks. Pre-resolution, the ONLY throw is `require`'s `IllegalArgumentException`; catch exactly
+ // that, since anything else escaping is a genuine bug we must not mask as unpartitioned.
+ private def isScalarSubqueryResolved(s: ScalarSubquery): Boolean =
+ try {
+ s.eval()
+ true
+ } catch {
+ case _: IllegalArgumentException => false
+ }
+
+ private def isAdaptivePlaceholder(p: SparkPlan): Boolean = p match {
+ case ReusedSubqueryExec(inner) => isAdaptivePlaceholder(inner)
+ case _: CometSubqueryAdaptiveBroadcastExec => true
+ case _: SubqueryAdaptiveBroadcastExec => true
+ case _ => false
+ }
+
+ override lazy val outputOrdering: Seq[SortOrder] = originalPlan.outputOrdering
+
+ override def dynamicPruningFilters: Seq[Expression] = runtimeFilters
+
+ override def withDynamicPruningFilters(filters: Seq[Expression]): SparkPlan = {
+ // A real copy: runtimeFilters is a constructor field included in equality, so the copy
+ // survives enclosing-block rebuilds, and the derived scanHelper picks up the rewritten
+ // filters automatically.
+ copy(runtimeFilters = filters)
+ }
+
+ /**
+ * Lazy split-mode serialization, mirroring CometNativeScanExec: common data was serialized at
+ * planning; per-partition file lists serialize here, at execution time.
+ */
+ @transient private lazy val serializedPartitionData
+ : (Array[Byte], Array[Array[Byte]], Array[Seq[String]]) = {
+ // Resolve the helper's DPP subqueries: it holds its own InSubqueryExec instances that
+ // Spark's expressions walk does not see (the helper is derived, not a child).
+ scanHelper.partitionFilters.foreach {
+ case DynamicPruningExpression(e: InSubqueryExec) if e.values().isEmpty =>
+ e.updateResult()
+ case _ =>
+ }
+
+ val commonBytes = {
+ val deltaScan = DeltaSparkScanEnvelope.unpack(nativeOp)
+ // Scalar subqueries in dataFilters were unresolved at planning; resolve them now and
+ // append them as pushed filters, as CometNativeScanExec.serializedPartitionData does.
+ val subqueryFilters = org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .resolvedSubqueryFilters(dataFilters, output, requiredSchema, conf)
+ val common = if (subqueryFilters.isEmpty) {
+ deltaScan.getCommon
+ } else {
+ val builder = deltaScan.getCommon.toBuilder
+ builder.setHasDataFilters(true)
Review Comment:
Good catch, and thanks for tracing it through the overflow change. Fixed in 637b44671. The helper now returns the presence alongside the protos, with presence true as soon as pushdown is enabled and a data filter contains a scalar subquery, before the binding guard and regardless of whether serialization succeeds. The exec site sets `has_data_filters` from that flag and appends whatever protos survived, possibly none, which is what the core scan does for its resolved filters.
Two regressions in `CometDeltaNativeScanSuite`: one pins the proto shape (`has_data_filters` true with zero data filters when `EqualNullSafe` is disabled), and one is your case end to end: a raw parquet-mr file with `ts` as TIMESTAMP_MILLIS holding 9223372036854776, converted with `CONVERT TO DELTA ... NO STATISTICS` so Delta's own stats pass does not read the value, a persisted bounds table returning the epoch, CORRECTED read modes, and `ts <=> (SELECT max(bound) FROM bounds)`. Before the fix it failed with `Arithmetic overflow: Overflow happened on: 9223372036854776 * 1000`; now it returns the empty result and the plan carries the Delta native scan.
One note on versions: Spark 3.5 does not push the resolved subquery literal itself (`FileSourceStrategy` excludes filters with subqueries and `pushedDownFilters` never resolves them), so Spark throws the overflow there while Comet now prunes, the same tradeoff #5177 accepted for filtered scans. The answer comparison against Spark is therefore gated to 4.0 and later, as in the core suite, while the native side is asserted on every profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5647289180
I filed #5882 to settle how the contrib jar is built, versioned and published, since the release scripts never pass `-Pdelta` today and the publish script uploads whatever `install` produces (which is also where the test-jar question lands). It also covers why Delta is packaged differently from Iceberg, which compiles against nothing and ships inside `comet-spark`, and whether that should stay the case. I would rather decide that on purpose in the issue than settle it by default here, so the two packaging points in my review (the `-Pdelta` release line and the unconditional test-jar) can be resolved there rather than blocking this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3986708322
##########
native/core/src/execution/planner/delta_spark_scan.rs:
##########
@@ -0,0 +1,792 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! JVM-planned Delta handler for the generic `OpStruct::ContribScan` dispatcher, feature-gated
+//! behind `delta`.
+//!
+//! delta-spark has already done log replay, snapshot resolution, and partition pruning by the
+//! time the scan reaches Comet, so the envelope carries a concrete file list (plus deletion
+//! vector descriptors) and the read path reuses the exact same shared parquet scan builder as
+//! `NativeScan` -- inheriting row-group stats pruning, page-index pruning, and filter pushdown.
+//! Sibling of the kernel-planned handler in `delta_scan.rs`; the two claim different
+//! `type_url`s within the same `ContribScan` envelope.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use datafusion::execution::object_store::ObjectStoreUrl;
+use object_store::path::Path;
+use object_store::ObjectStore;
+use url::Url;
+
+use datafusion_comet_proto::spark_operator::{
+ ContribScan, DeltaSparkScan, Operator, SparkFilePartition, SparkPartitionedFile,
+};
+use prost::Message;
+
+use crate::execution::operators::ExecutionError;
+use crate::execution::operators::ExecutionError::GeneralError;
+use crate::execution::planner::PhysicalPlanner;
+use crate::execution::planner::PlanCreationResult;
+use crate::parquet::objectstore::s3_blob_fs_support::normalize_object_store_url;
+use crate::parquet::parquet_support::{
+ hash_object_store_configs, object_store_url_key, prepare_object_store_with_config_hash,
+};
+
+/// Type name the JVM-planned Delta contrib claims within the `ContribScan` envelope. The
+/// contrib jar packs a `DeltaSparkScan` with a `type_url` of
+/// `type.googleapis.com/comet.contrib.delta_spark.DeltaSparkScan`; dispatch keys on the
+/// contrib-owned suffix, same convention as the kernel path's `delta_scan.rs`.
+const DELTA_SPARK_SCAN_TYPE_NAME: &str = "comet.contrib.delta_spark.DeltaSparkScan";
+
+/// Contrib entry point for the `OpStruct::ContribScan` dispatcher. Returns `Some(result)` when
+/// the envelope carries a JVM-planned Delta scan, or `None` when the `type_url` belongs to some
+/// other contrib.
+pub(crate) fn try_plan_contrib_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ contrib: &ContribScan,
+) -> Option<PlanCreationResult> {
+ if !contrib.type_url.ends_with(DELTA_SPARK_SCAN_TYPE_NAME) {
+ return None;
+ }
+ Some(
+ DeltaSparkScan::decode(contrib.value.as_slice())
+ .map_err(|e| {
+ GeneralError(format!(
+ "Failed to decode DeltaSparkScan from contrib_scan: {e}"
+ ))
+ })
+ .and_then(|scan| plan_delta_spark_scan(planner, spark_plan, &scan)),
+ )
+}
+
+fn plan_delta_spark_scan(
+ planner: &PhysicalPlanner,
+ spark_plan: &Operator,
+ scan: &DeltaSparkScan,
+) -> PlanCreationResult {
+ // Delta data files are plain parquet; the read path deliberately reuses
+ // the same shared parquet scan builder as NativeScan so Delta inherits
+ // row-group stats pruning, page-index pruning, and filter pushdown. Only
+ // the file list arrives in Delta-specific form. Note delta_common's
+ // column_mapping_mode is informational in M1: the actual field-id
+ // matching switch is common.use_field_id, same as the Iceberg path.
+ let common = scan
+ .common
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing common data".into()))?;
+
+ let delta_partition = scan
+ .file_partition
+ .as_ref()
+ .ok_or_else(|| GeneralError("DeltaSparkScan missing file_partition".into()))?;
+
+ let spark_partition = SparkFilePartition {
+ partitioned_file: delta_partition
+ .partitioned_file
+ .iter()
+ .map(|f| {
+ f.file.clone().ok_or_else(|| {
+ GeneralError("DeltaSparkPartitionedFile missing inner file".into())
+ })
+ })
+ .collect::<Result<Vec<_>, _>>()?,
+ };
+
+ // Defense-in-depth against a stale or bypassed JVM gate: DeltaScanSupport.declineReason
+ // (multiStoreReason) already declines data files spanning multiple object-store authorities
+ // at planning time, but prepare_scan_store_and_files below resolves this whole partition's
+ // ObjectStoreUrl from the FIRST file only and then strips every other file down to its bare
+ // object-store path -- a file that actually lives under a different authority would
+ // silently read through the first file's store handle. Checked here rather than inside
+ // prepare_scan_store_and_files itself, which is shared with plain NativeScan and out of
+ // scope for this Delta-specific invariant.
+ check_same_object_store_authority(&spark_partition.partitioned_file)?;
+
+ let (object_store_url, mut files) =
+ planner.prepare_scan_store_and_files(common, &spark_partition)?;
+
+ // Translate deletion vectors into per-file ParquetAccessPlans so deleted
+ // rows are skipped inside the reader (composing, by intersection, with
+ // page-index pruning). Fetching the bitmaps and footers is async I/O;
+ // create_plan runs on the JNI task thread outside the tokio context, so
+ // block_on here is safe and keeps the scan a plain DataSourceExec.
+ if delta_partition
+ .partitioned_file
+ .iter()
+ .any(|f| f.dv.is_some())
+ {
+ let object_store_options: HashMap<String, String> = common
+ .object_store_options
+ .iter()
+ .map(|(k, v)| (k.clone(), v.clone()))
+ .collect();
+ let runtime_env = planner.session_ctx.runtime_env();
+ // `object_store_options` is the same map for every file this partition resolves a store
+ // for, so its hash is loop-invariant too: computed once here rather than once per file
+ // inside `prepare_object_store_with_configs`.
+ let object_store_config_hash = hash_object_store_configs(&object_store_options);
+
+ // Resolve every object store this partition's files touch -- the data
+ // files' shared authority (check_same_object_store_authority above
+ // has already verified every data file in this partition resolves to
+ // the same authority, so prepare_scan_store_and_files's
+ // first-file-only resolution is safe here) plus any on-disk deletion
+ // vector's authority, which may legitimately differ from the data
+ // files' and carries its own resolved store -- before entering the
+ // async DV runtime below. This MUST happen here, on the JNI thread
+ // outside the tokio runtime:
+ // constructing a cold S3 store issues its own internal
+ // Handle::block_on calls (credential-provider / bucket-region
+ // resolution), which panics when nested inside the
+ // get_runtime().block_on(...) a few lines down. See
+ // delta_dv::attach_access_plans's doc comment for the invariant this
+ // maintains -- the async path never builds a store.
+ let mut resolved_stores: HashMap<ObjectStoreUrl, Arc<dyn ObjectStore>> = HashMap::new();
+ // Tracks, per resolved ObjectStoreUrl, the userinfo (and raw URL, for the error
+ // message) of the first URL that resolved to it. This closure is the ONE place in this
+ // scan that sees both data-file and deletion-vector URLs together, so the
+ // store-identity collision check (see check_store_identity's doc comment) lives here
+ // rather than as an extension of check_same_object_store_authority above, which sees
+ // only data files and would hard-error the legitimate cross-bucket DV shape.
+ let mut store_identities: HashMap<ObjectStoreUrl, (String, String)> = HashMap::new();
+ let mut resolve_store =
+ |url: String| -> Result<(Path, Arc<dyn ObjectStore>), ExecutionError> {
+ let parsed_url = Url::parse(&url).map_err(|e| {
+ GeneralError(format!(
+ "Error parsing URL {}: {e}",
+ redacted_url_display(&url)
+ ))
+ })?;
+ let user_info = url_user_info(&parsed_url);
+ // The same s3a/alias rewrite the shared resolution path applies first, so the
+ // cheap key and cached-hit path below agree with what
+ // prepare_object_store_with_config_hash registers under. Re-parses the already
+ // parsed string only so the parse error above stays redacted.
+ let normalized = normalize_object_store_url(&url, &object_store_options)?;
+
+ // Cheap, I/O-free cache key: no config hashing, no global object-store-cache
+ // lock, no runtime_env registration. Checked against the LOCAL `resolved_stores`
+ // map below before ever paying for the expensive resolution path -- most files
+ // in a partition share the same authority as an already-resolved file.
+ let (url_key, _is_hdfs_scheme) = object_store_url_key(&normalized);
+ let parsed_url = normalized.url;
+ let store_url = ObjectStoreUrl::parse(url_key)?;
+ check_store_identity(&store_url, &user_info, &url, &mut store_identities)?;
+ if let Some(store) = resolved_stores.get(&store_url) {
Review Comment:
### Performance
**[P2] Preserve local store-cache hits with isolated registration URLs**
For a partition containing deletion vectors, `object_store_url_key` still produces the physical key (for example, `s3://bucket`), but [`prepare_object_store_with_config_hash`](https://github.com/apache/datafusion-comet/blob/3f86d7ec4865ec5c59bd9560a2f37686932e034e/native/core/src/parquet/parquet_support.rs#L775-L797) now returns `s3+comet-<hash>-native://bucket`. Line 207 inserts that returned key, so the next data-file or external-DV reference to the same bucket always misses this lookup. Every reference then takes the global cache read lock, registers the store again in the runtime and looks it up again. The intended once-per-store local memoization is lost. Native `file://` is the exception because its registration key is unchanged.
Please use the same backend-aware identity for both lookup and insertion, preserving the distinction between native and Hadoop stores, and add a repeated-resolution regression. A source-extracted closure probe with in-memory URI/runtime doubles confirmed 16 slow-path calls for 16 same-store references, compared with one under the old registration-key control. The global cache still reuses the backend, so this is repeated planning work rather than evidence of additional storage I/O.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4084966658
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so its rewrite (via
+ * [[CometScanWithPlanData]]) survives plan copies -- a transient field would be dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning (the CometIcebergNativeScanExec lesson).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node
+ // automatically gets a helper consistent with ITS runtimeFilters, avoiding the #3510 class of
+ // bug where a stored helper field desyncs from rewritten filters. Costs one extra file listing
+ // per executed instance; correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below, and AQE's UI
+ // plan-walk calls `.metrics` on every node MID-PLANNING, including while a DPP subquery is
+ // still an adaptive placeholder or a partition filter holds an unresolved ScalarSubquery (see
+ // `hasUnevaluableSubqueryFilter` below). That's safe ONLY because constructing `scanHelper` is a
+ // cheap case-class build with no file listing, and core's `CometScanExec.metrics` touches only
+ // `wrapped.driverMetrics` (populated by Spark's own planning) plus a static metric-node
+ // constructor -- neither file listing nor subquery resolution. If core's `metrics` ever touches
+ // either, forcing `scanHelper` here would resurrect the AQE mid-planning crashes this invariant
+ // prevents.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element, so checking runtimeFilters
+ // here avoids constructing/forcing the derived scanHelper just to read partitioning. The
+ // InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep in sync. The
+ // ScalarSubquery case is probed rather than treated as permanently unevaluable: Spark exposes no
+ // public finished/updated flag on ExecSubqueryExpression, but `eval()` doubles as one -- it only
+ // reads the cached `result` behind a `require(updated, ...)` guard, while the subquery is
+ // actually run by `updateResult()` (invoked separately during prepare/AQE), never by `eval()`.
+ // Once resolved, outputPartitioning below reports the real perPartitionData.length instead of
+ // staying at zero -- a fused native parent's buildNativeContext requires that count to match.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case s: ScalarSubquery => !isScalarSubqueryResolved(s)
+ case _ => false
+ })
+
+ // `eval()` never triggers the subquery's execution: on a resolved subquery it is a pure cached
+ // read of `result` (verified against bytecode: `Predef.require(updated(), ...)` then a plain
+ // field read), so this probe is safe to call repeatedly, including from AQE's mid-planning plan
+ // walks. Pre-resolution, the ONLY throw is `require`'s `IllegalArgumentException`; catch exactly
+ // that, since anything else escaping is a genuine bug we must not mask as unpartitioned.
+ private def isScalarSubqueryResolved(s: ScalarSubquery): Boolean =
+ try {
+ s.eval()
+ true
+ } catch {
+ case _: IllegalArgumentException => false
+ }
+
+ private def isAdaptivePlaceholder(p: SparkPlan): Boolean = p match {
+ case ReusedSubqueryExec(inner) => isAdaptivePlaceholder(inner)
+ case _: CometSubqueryAdaptiveBroadcastExec => true
+ case _: SubqueryAdaptiveBroadcastExec => true
+ case _ => false
+ }
+
+ override lazy val outputOrdering: Seq[SortOrder] = originalPlan.outputOrdering
+
+ override def dynamicPruningFilters: Seq[Expression] = runtimeFilters
+
+ override def withDynamicPruningFilters(filters: Seq[Expression]): SparkPlan = {
+ // A real copy: runtimeFilters is a constructor field included in equality, so the copy
+ // survives enclosing-block rebuilds, and the derived scanHelper picks up the rewritten
+ // filters automatically.
+ copy(runtimeFilters = filters)
+ }
+
+ /**
+ * Lazy split-mode serialization, mirroring CometNativeScanExec: common data was serialized at
+ * planning; per-partition file lists serialize here, at execution time.
+ */
+ @transient private lazy val serializedPartitionData
+ : (Array[Byte], Array[Array[Byte]], Array[Seq[String]]) = {
+ // Resolve the helper's DPP subqueries: it holds its own InSubqueryExec instances that
+ // Spark's expressions walk does not see (the helper is derived, not a child).
+ scanHelper.partitionFilters.foreach {
+ case DynamicPruningExpression(e: InSubqueryExec) if e.values().isEmpty =>
+ e.updateResult()
+ case _ =>
+ }
+
+ val commonBytes = {
+ val deltaScan = DeltaSparkScanEnvelope.unpack(nativeOp)
+ // Scalar subqueries in dataFilters were unresolved at planning; resolve them now and
+ // append them as pushed filters, as CometNativeScanExec.serializedPartitionData does.
+ // has_data_filters follows their presence, not the serialized count: a filter that fails
+ // to serialize still keeps native on the safe timestamp conversion for a filtered scan.
+ val resolved = org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .resolvedSubqueryFilters(dataFilters, output, requiredSchema, conf)
+ val common = if (!resolved.hasResolvedFilters) {
+ deltaScan.getCommon
+ } else {
+ val builder = deltaScan.getCommon.toBuilder
+ builder.setHasDataFilters(true)
+ resolved.protos.foreach(builder.addDataFilters)
+ builder.build()
+ }
+ OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaScan.getDeltaCommon)
+ .build()
+ .toByteArray
+ }
+
+ val filePartitions = scanHelper.getFilePartitions()
+
+ val tableRoot = DeltaSparkScanEnvelope.unpack(nativeOp).getDeltaCommon.getTableRoot
+ val perPartitionBytes = filePartitions.map { filePartition =>
+ org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .serializePartition(filePartition, originalPlan, tableRoot)
+ }.toArray
+
+ val perPartitionPaths = filePartitions.map(_.files.map(_.filePath.toString).toSeq).toArray
+
+ (commonBytes, perPartitionBytes, perPartitionPaths)
+ }
+
+ override def commonData: Array[Byte] = serializedPartitionData._1
+
+ override def perPartitionData: Array[Array[Byte]] = serializedPartitionData._2
+
+ def perPartitionFilePaths: Array[Seq[String]] = serializedPartitionData._3
+
+ override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+ val nativeMetrics = CometMetricNode.fromCometPlan(this)
+ val serializedPlan = CometExec.serializeNativePlan(nativeOp)
+
+ new CometExecRDD(
+ sparkContext,
+ Seq.empty,
+ Map(sourceKey -> commonData),
+ Map(sourceKey -> perPartitionData),
+ serializedPlan,
+ PlanDataInjector.planFingerprint(serializedPlan),
+ perPartitionData.length,
+ output.length,
+ nativeMetrics,
+ Seq.empty,
+ None,
+ Seq.empty,
+ perPartitionFilePaths = perPartitionFilePaths,
+ reportScanInputMetrics = true)
+ }
+
+ override def doCanonicalize(): CometDeltaNativeScanExec = {
+ val canonOriginal = if (originalPlan != null) {
+ val stripped = originalPlan.copy(partitionFilters =
+ CometScanUtils.filterUnusedDynamicPruningExpressions(originalPlan.partitionFilters))
+ stripped.doCanonicalize()
+ } else {
+ null
+ }
+ CometDeltaNativeScanExec(
+ nativeOp,
+ output.map(QueryPlan.normalizeExpressions(_, output)),
+ requiredSchema,
+ QueryPlan.normalizePredicates(
+ CometScanUtils.filterUnusedDynamicPruningExpressions(runtimeFilters),
+ output),
+ QueryPlan.normalizePredicates(dataFilters, output),
+ relation,
+ canonOriginal,
+ SerializedPlan(None),
+ "")
+ }
+
+ override def stringArgs: Iterator[Any] = Iterator(output, runtimeFilters)
+
+ override def equals(obj: Any): Boolean = obj match {
+ case other: CometDeltaNativeScanExec =>
+ this.originalPlan == other.originalPlan &&
+ this.serializedPlanOpt == other.serializedPlanOpt &&
+ this.runtimeFilters == other.runtimeFilters &&
+ this.dataFilters == other.dataFilters
+ case _ => false
+ }
+
+ override def hashCode(): Int =
+ java.util.Objects.hash(originalPlan, serializedPlanOpt, runtimeFilters, dataFilters)
+
+ private val driverMetricKeys =
+ Set(
+ "numFiles",
+ "filesSize",
+ "numPartitions",
+ "metadataTime",
+ "staticFilesNum",
+ "staticFilesSize",
+ "pruningTime")
+
+ // Forces `scanHelper` (see its doc above for why that -- and reading `.metrics` off it -- is
+ // safe even when AQE calls `.metrics` mid-planning against an unresolved DPP/scalar subquery).
+ override lazy val metrics: Map[String, SQLMetric] = {
+ CometMetricNode.nativeScanMetrics(session.sparkContext) ++
+ scanHelper.metrics.filter { case (k, _) => driverMetricKeys.contains(k) }
+ }
+}
+
+object CometDeltaNativeScanExec {
+
+ /** File-planning helper: reuses CometScanExec's listing/splitting/DPP machinery. */
+ def planningHelper(
Review Comment:
Confirmed, and the comment and the issue were wrong. There was nothing in the module or the docs that said DV'd files are not split, only the issue text. The `planningHelper` comment now says a claimed scan is split like any other file and what each split does today: fetch and decode the whole deletion vector, read the footer, build the whole-file plan and reserve memory for the whole file, while the reader keeps only the row groups that start inside the split.
New test "deletion vectors: one file split into many byte ranges reads natively": one file of 20000 rows written with 16 KB row groups and 4 KB pages, deletion vectors on, a scattered delete that touches every row group plus a deleted tail, read with `spark.sql.files.maxPartitionBytes=4096`. It checks the scan is claimed and matches Spark, that it is one data file, and that the native scan has more than one partition. Forcing `maxPartitionBytes` back to the default makes that last check fail with a single partition, so it is exercising the split path.
Issue #5655 is updated to say splits happen today and that the work there is making each split cheaper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4081163000
##########
spark/pom.xml:
##########
@@ -222,6 +222,27 @@ under the License.
</dependencies>
<profiles>
+ <profile>
+ <!-- Only the contrib build needs the test-jar: contrib/delta-spark reuses CometTestBase
+ and the shared test utilities in its suites. Binding it here keeps the jar out of
+ every other profile's install and out of the release upload. -->
+ <id>delta</id>
Review Comment:
> Please activate `delta` for the prerequisite installs and the README's matching command, or use a build sequence that produces the current test JAR.
Done on 15 September. Both prerequisite installs in `delta_contrib_test.yml` and the README's first command run with `-Pspark-X,delta`, so the install produces this checkout's test jar before the contrib suites resolve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4133876921
##########
native/core/src/parquet/datetime_rebase.rs:
##########
@@ -0,0 +1,2802 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Per-file datetime calendar-rebase handling for the parquet scan.
+//!
+//! Spark 2.4 and earlier wrote dates and timestamps in the hybrid Julian + Gregorian calendar;
+//! Spark 3.0+ uses the proleptic Gregorian calendar and records the calendar policy of every
+//! file it writes in the parquet footer's key-value metadata (`org.apache.spark.version`,
+//! `org.apache.spark.legacyDateTime`, `org.apache.spark.legacyINT96`,
+//! `org.apache.spark.timeZone`). Spark's reader resolves the rebase policy from EACH FILE's
+//! writer metadata (`DataSourceUtils.datetimeRebaseSpec` / `int96RebaseSpec`) -- the session's
+//! `spark.sql.parquet.datetimeRebaseModeInRead` conf only applies to files whose metadata does
+//! not decide the policy on its own -- so a reader that ignores the metadata silently returns
+//! values shifted by up to ten days for dates before 1582-10-15 (e.g. `1500-01-01` reads as
+//! `1500-01-10`).
+//!
+//! This module mirrors that per-file resolution: [`resolve_file_rebase_policies`] computes the
+//! date / INT64-timestamp / INT96-timestamp policies from a file's arrow schema metadata (the
+//! parquet key-value pairs survive the parquet -> arrow schema conversion), and
+//! [`wrap_datetime_rebase`] wraps the per-file rewritten expressions' column references in a
+//! [`SparkDatetimeRebaseExpr`] that rebases values exactly where that is possible without the
+//! JVM's historical timezone tables (dates always; timestamps for a fixed UTC writer zone) and
+//! refuses -- rather than silently corrupting -- ancient values it cannot rebase. Nested
+//! columns are rebuilt leaf by leaf (struct / list / map / fixed-size list / dictionary), each
+//! leaf under its own policy, with nulls and offsets preserved. Modern values are always the
+//! identity under every policy: from 1582-10-15 onward for dates, and from
+//! [`LAST_SWITCH_JULIAN_TS_SECONDS`] (1900-01-01T00:00:00Z, Spark's
+//! `RebaseDateTime.lastSwitchJulianTs`) onward for timestamps.
+//!
+//! Spark applies `datetimeRebaseSpec` to INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns
+//! and `int96RebaseSpec` to INT96 columns. The two physical types are indistinguishable in the
+//! arrow schema DataFusion hands the expression adapter (both surface as `Timestamp(us, "UTC")`
+//! after INT96 coercion), so Comet's parquet reader factory stamps the file's INT96 leaf
+//! ordinals -- taken from the parquet footer's own `SchemaDescriptor` -- into the key-value
+//! metadata under [`INT96_LEAVES_METADATA_KEY`] before the arrow schema is derived (see
+//! [`stamp_int96_leaves`] and `eager_page_index_reader_factory.rs`), and the adapter attributes
+//! every timestamp leaf to its spec from that stamp. Without a stamp, the two specs are merged:
+//! agreement decides, disagreement degrades to [`RebasePolicy::CheckAncient`].
+//!
+//! The wrapper sits BENEATH the schema adapter's nested narrowing (the struct -> struct convert
+//! that keeps only the requested children), which is what keeps those ordinals physical -- but
+//! it means the wrapper sees every physical child, requested or not. Spark only ever decodes
+//! the requested nested schema, so [`FileRebasePolicies::restrict_to_requested`] marks the
+//! physical leaves the narrowing drops as the identity: an unrequested ancient `s.ts` never
+//! blocks `select s.d`, exactly as in Spark.
+//!
+//! Currently only enabled by the Delta scan arms via
+//! `SparkParquetOptions::rebase_from_file_metadata`, which also carries the session read modes
+//! ([`SessionRebaseModes`], forwarded from the JVM) that decide the policy for files without
+//! Spark writer metadata; the plain NativeScan keeps its documented no-rebase behavior (see
+//! the compatibility guide and issue #5010).
+
+use std::collections::HashMap;
+use std::fmt::{self, Display};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Date32Array, FixedSizeListArray, GenericListArray, MapArray,
+ OffsetSizeTrait, PrimitiveArray, RecordBatch, StructArray,
+};
+use arrow::datatypes::{
+ ArrowPrimitiveType, ArrowTimestampType, DataType, Date32Type, FieldRef, Schema, SchemaRef,
+ TimeUnit, TimestampMicrosecondType, TimestampMillisecondType, TimestampNanosecondType,
+ TimestampSecondType,
+};
+use arrow::error::ArrowError;
+use datafusion::common::tree_node::{Transformed, TreeNode};
+use datafusion::common::{DataFusionError, Result as DataFusionResult};
+use datafusion::physical_expr::expressions::Column;
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::ColumnarValue;
+use parquet::basic::Type as ParquetPhysicalType;
+use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaData};
+use parquet::schema::types::SchemaDescriptor;
+
+use super::name_fold::fold_names;
+use super::parquet_support::field_id;
+
+/// Footer key naming the Spark release that wrote the file; absent for non-Spark writers.
+const SPARK_VERSION_METADATA_KEY: &str = "org.apache.spark.version";
+/// Present (empty value) when the file's dates and INT64 timestamps were written with
+/// `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_DATETIME_KEY: &str = "org.apache.spark.legacyDateTime";
+/// Present (empty value) when the file's INT96 timestamps were written with
+/// `spark.sql.parquet.int96RebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_INT96_KEY: &str = "org.apache.spark.legacyINT96";
+/// The writer session's time zone, stamped alongside either legacy flag.
+const SPARK_TIMEZONE_KEY: &str = "org.apache.spark.timeZone";
+
+/// Key-value metadata entry Comet's parquet reader factory adds to a file's footer metadata
+/// (in memory only, never written back) so the expression adapter can tell INT96 timestamp
+/// columns from INT64 ones after both have been coerced to the same arrow type. Value:
+/// `"<leaf count>:<comma-separated INT96 leaf ordinals>"`, where leaves are the file's
+/// primitive columns in `SchemaDescriptor::columns()` order -- the same depth-first order
+/// parquet-rs assigns arrow leaves, so an arrow-side depth-first walk lines up with it. The
+/// leaf count lets the reader detect a stamp that does not describe the schema it is paired
+/// with (see [`Int96Attribution::from_schema`]).
+pub(crate) const INT96_LEAVES_METADATA_KEY: &str = "comet.int96_leaf_columns";
+
+/// Day of the Gregorian cutover (1582-10-15) as days since the epoch; rebasing is the identity
+/// from this day onward. Same value as Spark's `RebaseDateTime.lastSwitchJulianDay`.
+const LAST_SWITCH_JULIAN_DAY: i32 = -141427;
+
+/// Spark's `RebaseDateTime.lastSwitchJulianTs` (and `lastSwitchGregorianTs`) in seconds since
+/// the epoch: 1900-01-01T00:00:00Z. Spark derives it as the latest switch instant across every
+/// zone in its `julian-gregorian-rebase-micros.json` table (`getLastSwitchTs`, which also
+/// asserts the calendars' difference is zero for every zone from then on): most zones ran on
+/// local mean time before 1900, so the last instant at which rebasing changes a value in ANY
+/// zone is 1900-01-01T00:00:00Z, not the 1582 cutover. `createTimestampRebaseFuncInRead`
+/// under `EXCEPTION` throws exactly for `micros < lastSwitchJulianTs` (after converting
+/// `TIMESTAMP_MILLIS` to micros), and `rebaseJulianToGregorianMicros` is the identity from it
+/// onward in every zone. The value is in seconds so it scales exactly to any timestamp unit.
+pub(crate) const LAST_SWITCH_JULIAN_TS_SECONDS: i64 = -2_208_988_800;
+
+/// The per-century differences between the Julian and proleptic Gregorian calendars, and the
+/// Julian-calendar switch days at which each difference starts to apply. Copied verbatim from
+/// Spark's `RebaseDateTime.julianGregDiffs` / `julianGregDiffSwitchDay` (which Spark generated
+/// from `localRebaseJulianToGregorianDays`); `rebase_julian_to_gregorian_days` must stay
+/// value-for-value equal to Spark's `rebaseJulianToGregorianDays`.
+const JULIAN_GREG_DIFFS: [i32; 14] = [2, 1, 0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, 0];
+const JULIAN_GREG_DIFF_SWITCH_DAY: [i32; 14] = [
+ -719164, -682945, -646420, -609895, -536845, -500320, -463795, -390745, -354220, -317695,
+ -244645, -208120, -171595, -141427,
+];
+
+/// Proleptic-Gregorian days since 1970-01-01 for a nominal civil date, via Howard Hinnant's
+/// `days_from_civil`. `d` may exceed the month's length; the excess rolls into the following
+/// month exactly like `LocalDate.of(y, m, 1).plusDays(d - 1)` in Spark's
+/// `localRebaseJulianToGregorianDays` (how the non-existent proleptic date `1000-02-29`,
+/// valid in the Julian calendar, lands on `1000-03-01`).
+fn days_from_civil(y: i64, m: i64, d: i64) -> i64 {
+ let y = if m <= 2 { y - 1 } else { y };
+ let era = y.div_euclid(400);
+ let yoe = y - era * 400; // [0, 399]
+ let mp = (m + 9) % 12; // [0, 11], March = 0
+ let doy = (153 * mp + 2) / 5 + d - 1;
+ let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
+ era * 146097 + doe - 719468
+}
+
+/// Julian-calendar civil date `(year, month, day)` for a day count since 1970-01-01 that labels
+/// days in the Julian calendar (astronomical year numbering: 1 BCE is year 0). Standard
+/// Julian-day-number conversion (E.G. Richards' algorithm), exact for any day.
+fn julian_day_to_civil(days: i64) -> (i64, i64, i64) {
+ // Integer (noon) Julian Day Number of this civil day: 1970-01-01 is JDN 2440588.
+ let jdn = days + 2_440_588;
+ let f = jdn + 1401;
+ let e = 4 * f + 3;
+ let g = e.rem_euclid(1461) / 4;
+ let h = 5 * g + 2;
+ let day = h.rem_euclid(153) / 5 + 1;
+ let month = (h / 153 + 2).rem_euclid(12) + 1;
+ let year = e.div_euclid(1461) - 4716 + (14 - month) / 12;
+ (year, month, day)
+}
+
+/// Exact port of Spark's `RebaseDateTime.rebaseJulianToGregorianDays`: reinterprets a day count
+/// written in the hybrid Julian + Gregorian calendar as the proleptic Gregorian day count of the
+/// same nominal civil date. Identity for days from 1582-10-15 onward. Days before the tables'
+/// range (before Julian `0001-01-01`) take the calendar-arithmetic path, mirroring Spark's
+/// `localRebaseJulianToGregorianDays` fallback.
+pub(crate) fn rebase_julian_to_gregorian_days(days: i32) -> i32 {
+ if days < JULIAN_GREG_DIFF_SWITCH_DAY[0] {
+ let (y, m, d) = julian_day_to_civil(days as i64);
+ (days_from_civil(y, m, 1) + (d - 1)) as i32
+ } else {
+ // Spark's rebaseDays: linear search from the most recent switch day.
+ let mut i = JULIAN_GREG_DIFF_SWITCH_DAY.len();
+ loop {
+ i -= 1;
+ if i == 0 || days >= JULIAN_GREG_DIFF_SWITCH_DAY[i] {
+ break;
+ }
+ }
+ days + JULIAN_GREG_DIFFS[i]
+ }
+}
+
+/// Timezone strings from `org.apache.spark.timeZone` that denote a fixed zero-offset zone in
+/// both `java.util.TimeZone` and `java.time`. Only for these is timestamp rebasing the pure
+/// nominal-date shift [`SparkDatetimeRebaseExpr::rebase_timestamp_utc`] computes; any other (or
+/// absent) zone needs the JVM's historical timezone tables and stays on the
+/// refuse-ancient-values path.
+const UTC_EQUIVALENT_TIMEZONES: [&str; 6] = ["UTC", "Etc/UTC", "GMT", "Etc/GMT", "Z", "+00:00"];
+
+/// How the writer's session time zone (if recorded) affects timestamp rebasing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum WriterTimeZone {
+ /// A fixed zero-offset zone: rebasing reduces to the exact nominal-date shift.
+ Utc,
+ /// Any other zone, or none recorded (pre-3.0 files): ancient values cannot be rebased
+ /// without the JVM's historical timezone data.
+ OtherOrUnknown,
+}
+
+/// One session-level datetime rebase read mode (a `LegacyBehaviorPolicy` value of
+/// `spark.sql.parquet.datetimeRebaseModeInRead` / `int96RebaseModeInRead`), consulted by
+/// [`resolve_file_rebase_policies`] ONLY for files whose footer metadata does not decide the
+/// policy on its own -- exactly the `getOrElse` fallback in Spark's
+/// `DataSourceUtils.getRebaseSpec`. Files that carry `org.apache.spark.version` ignore these
+/// modes entirely, on every Spark version.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) enum RebaseReadMode {
+ /// Refuse ancient values (Spark raises `SparkUpgradeException`); maps to
+ /// [`RebasePolicy::CheckAncient`]. The default mirrors the conservative posture used
+ /// before the conf was plumbed through (and Spark 3.x's own conf default).
+ #[default]
+ Exception,
+ /// Read values as proleptic Gregorian without rebasing.
+ Corrected,
+ /// Rebase from the hybrid Julian + Gregorian calendar.
+ Legacy,
+}
+
+impl RebaseReadMode {
+ /// Parses a `LegacyBehaviorPolicy` conf value. `SQLConf` validates and upper-cases the
+ /// session conf, but a per-relation `datetimeRebaseMode` option arrives verbatim, so the
+ /// match is case-insensitive. Anything unrecognized -- including the empty string a proto
+ /// producer that predates the field sends -- falls back to [`RebaseReadMode::Exception`],
+ /// which refuses ancient values rather than silently corrupting them.
+ pub(crate) fn from_conf_value(value: &str) -> Self {
+ match value.to_ascii_uppercase().as_str() {
+ "CORRECTED" => RebaseReadMode::Corrected,
+ "LEGACY" => RebaseReadMode::Legacy,
+ _ => RebaseReadMode::Exception,
+ }
+ }
+}
+
+/// The session's effective datetime rebase read modes, one per spec class (INT64
+/// dates/timestamps vs INT96 timestamps), forwarded from the JVM at planning time.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) struct SessionRebaseModes {
+ /// `spark.sql.parquet.datetimeRebaseModeInRead` (or the relation's `datetimeRebaseMode`).
+ pub datetime: RebaseReadMode,
+ /// `spark.sql.parquet.int96RebaseModeInRead` (or the relation's `int96RebaseMode`).
+ pub int96: RebaseReadMode,
+}
+
+/// Calendar policy of one file's date or timestamp columns, resolved from writer metadata the
+/// same way Spark's `DataSourceUtils.getRebaseSpec` resolves it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum RebasePolicy {
+ /// Written in the proleptic Gregorian calendar; values pass through untouched.
+ Corrected,
+ /// Written in the hybrid Julian + Gregorian calendar; values must be rebased.
+ Legacy(WriterTimeZone),
+ /// Policy could not be pinned down (contradictory flags, or a non-Spark writer under the
+ /// `EXCEPTION` read mode): modern values -- identical under either calendar -- pass,
+ /// ancient values raise. Mirrors Spark's `EXCEPTION` behavior (`SparkUpgradeException`).
+ CheckAncient,
+}
+
+/// Which of a file's leaf columns are physically INT96, from the stamp the parquet reader
+/// factory adds under [`INT96_LEAVES_METADATA_KEY`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) enum Int96Attribution {
+ /// No stamp, or a stamp whose leaf count does not match the schema it arrived with: the
+ /// INT64 and INT96 timestamp specs cannot be told apart per column and are merged.
+ Unknown,
+ /// Sorted leaf ordinals (depth-first over the file schema's primitive columns) that are
+ /// INT96; every other timestamp leaf is INT64.
+ Known(Vec<usize>),
+}
+
+impl Int96Attribution {
+ /// Parses the stamp out of `schema`'s metadata and validates its leaf count against the
+ /// schema's own depth-first leaf count, so a stamp that does not describe this schema (a
+ /// crafted footer key, or a cached-metadata mismatch) degrades to [`Self::Unknown`].
+ fn from_schema(schema: &Schema) -> Self {
+ let Some(stamp) = schema.metadata().get(INT96_LEAVES_METADATA_KEY) else {
+ return Int96Attribution::Unknown;
+ };
+ let Some((count, ordinals)) = stamp.split_once(':') else {
+ return Int96Attribution::Unknown;
+ };
+ let schema_leaves: usize = schema
+ .fields()
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ if count.parse::<usize>().ok() != Some(schema_leaves) {
+ return Int96Attribution::Unknown;
+ }
+ let parsed: Option<Vec<usize>> = if ordinals.is_empty() {
+ Some(Vec::new())
+ } else {
+ ordinals
+ .split(',')
+ .map(|o| o.parse::<usize>().ok().filter(|o| *o < schema_leaves))
+ .collect()
+ };
+ match parsed {
+ Some(mut leaves) => {
+ leaves.sort_unstable();
+ Int96Attribution::Known(leaves)
+ }
+ None => Int96Attribution::Unknown,
+ }
+ }
+
+ /// `Some(true)` / `Some(false)` when the leaf is known to be INT96 / INT64, `None` when
+ /// the attribution is unknown.
+ fn is_int96(&self, leaf: usize) -> Option<bool> {
+ match self {
+ Int96Attribution::Unknown => None,
+ Int96Attribution::Known(leaves) => Some(leaves.binary_search(&leaf).is_ok()),
+ }
+ }
+}
+
+/// The [`INT96_LEAVES_METADATA_KEY`] value describing `schema`: its leaf count and the
+/// ordinals of its INT96 primitive columns.
+pub(crate) fn int96_leaf_stamp(schema: &SchemaDescriptor) -> String {
+ let ordinals: Vec<String> = schema
+ .columns()
+ .iter()
+ .enumerate()
+ .filter(|(_, column)| column.physical_type() == ParquetPhysicalType::INT96)
+ .map(|(ordinal, _)| ordinal.to_string())
+ .collect();
+ format!("{}:{}", schema.num_columns(), ordinals.join(","))
+}
+
+/// Returns a copy of `metadata` whose key-value metadata carries the [`int96_leaf_stamp`] of
+/// its own schema, or `None` when it already does (the common case after the first open of a
+/// file, since the caller caches the stamped copy). Any pre-existing entry under the key --
+/// a file cannot legitimately carry one -- is replaced, never trusted. Only the file-level
+/// key-value list changes; row groups and page indexes are carried over as-is. The parquet
+/// API cannot carry a file decryptor, nor `FileMetaData`'s crate-private encryption fields
+/// (encryption algorithm, footer signing key metadata), across this rebuild, so callers must
+/// not stamp opens that supply decryption properties -- and the only consumer, the Delta
+/// scan, declines every encrypted-parquet configuration before planning, so a parquet
+/// modular encryption file never reaches this path with or without those properties.
+pub(crate) fn stamp_int96_leaves(metadata: &ParquetMetaData) -> Option<ParquetMetaData> {
+ let file_metadata = metadata.file_metadata();
+ let stamp = int96_leaf_stamp(file_metadata.schema_descr());
+ let existing = file_metadata
+ .key_value_metadata()
+ .and_then(|kvs| kvs.iter().find(|kv| kv.key == INT96_LEAVES_METADATA_KEY))
+ .and_then(|kv| kv.value.as_deref());
+ if existing == Some(stamp.as_str()) {
+ return None;
+ }
+ let mut key_values: Vec<KeyValue> = file_metadata
+ .key_value_metadata()
+ .map(|kvs| {
+ kvs.iter()
+ .filter(|kv| kv.key != INT96_LEAVES_METADATA_KEY)
+ .cloned()
+ .collect()
+ })
+ .unwrap_or_default();
+ key_values.push(KeyValue::new(INT96_LEAVES_METADATA_KEY.to_string(), stamp));
+ let stamped_file_metadata = FileMetaData::new(
+ file_metadata.version(),
+ file_metadata.num_rows(),
+ file_metadata.created_by().map(str::to_string),
+ Some(key_values),
+ file_metadata.schema_descr_ptr(),
+ file_metadata.column_orders().cloned(),
+ );
+ Some(
+ ParquetMetaData::new(stamped_file_metadata, metadata.row_groups().to_vec())
+ .into_builder()
+ .set_column_index(metadata.column_index().cloned())
+ .set_offset_index(metadata.offset_index().cloned())
+ .build(),
+ )
+}
+
+/// Per-file rebase policies for the three affected column classes, plus the INT96
+/// attribution that selects between the two timestamp specs per leaf.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct FileRebasePolicies {
+ /// `DATE` columns, governed by `org.apache.spark.legacyDateTime` alone.
+ pub date: RebasePolicy,
+ /// INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns: the datetime spec (same
+ /// resolution as `date`), as Spark's `ParquetVectorUpdaterFactory` selects for INT64.
+ pub int64_timestamp: RebasePolicy,
+ /// INT96 columns: the INT96 spec (`org.apache.spark.legacyINT96`, min version 3.1.0).
+ pub int96_timestamp: RebasePolicy,
+ /// Which timestamp leaves are INT96. See [`Int96Attribution`].
+ pub int96_leaves: Int96Attribution,
+ /// Sorted depth-first leaf ordinals -- over the physical file schema, the same ordinals
+ /// `int96_leaves` uses -- that the query does not read: nested children the schema
+ /// adapter's struct narrowing drops before any value leaves the scan. Spark never decodes
+ /// them either, so their policy is the identity whatever the file's calendar. Empty until
+ /// [`Self::restrict_to_requested`] runs (every leaf requested).
+ pub unrequested_leaves: Vec<usize>,
+}
+
+impl FileRebasePolicies {
+ /// True when some policy is not the plain proleptic-Gregorian pass-through, i.e. when the
+ /// per-column wrap in [`wrap_datetime_rebase`] can install anything at all.
+ pub(crate) fn any_rebase_needed(&self) -> bool {
+ self.date != RebasePolicy::Corrected
+ || self.int64_timestamp != RebasePolicy::Corrected
+ || self.int96_timestamp != RebasePolicy::Corrected
+ }
+
+ fn is_requested(&self, leaf: usize) -> bool {
+ self.unrequested_leaves.binary_search(&leaf).is_err()
+ }
+
+ /// The policy of the `Date32` leaf at depth-first ordinal `leaf`: the file's date policy,
+ /// or the identity when the query does not read that leaf.
+ fn date_policy(&self, leaf: usize) -> RebasePolicy {
+ if self.is_requested(leaf) {
+ self.date
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+
+ /// The policy of the timezone-carrying timestamp leaf at depth-first ordinal `leaf`: the
+ /// identity when the query does not read it; otherwise its physical type's spec when the
+ /// attribution is known, or else the two specs merged -- agreement decides, disagreement
+ /// degrades to [`RebasePolicy::CheckAncient`], which still passes every modern value and
+ /// refuses only ancient ones.
+ fn timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ Some(false) => self.int64_timestamp,
+ None if self.int64_timestamp == self.int96_timestamp => self.int64_timestamp,
+ None => RebasePolicy::CheckAncient,
+ }
+ }
+
+ /// These policies with every physical leaf the query does not read marked the identity.
+ /// `requested` pairs each top-level field of `physical_schema` (by position) with the type
+ /// of the logical field the schema adapter narrows it to -- `None` for a column without a
+ /// logical counterpart, whose leaves are left as they are (no expression reads it anyway).
+ /// Nested children pair the way the adapter's struct convert selects them (see
+ /// [`push_unrequested_leaves`]); the INT96 attribution is untouched, since the ordinals
+ /// stay physical. `requested` is parallel to the schema's fields; should a caller pass a
+ /// shorter slice, the trailing columns simply keep every leaf (the safe direction).
+ pub(crate) fn restrict_to_requested(
+ mut self,
+ physical_schema: &Schema,
+ requested: &[Option<&DataType>],
+ case_sensitive: bool,
+ use_field_id: bool,
+ ) -> Self {
+ debug_assert_eq!(requested.len(), physical_schema.fields().len());
+ let matching = FieldMatching {
+ case_sensitive,
+ use_field_id,
+ };
+ let mut next_leaf = 0;
+ let mut unrequested = Vec::new();
+ for (field, requested) in physical_schema.fields().iter().zip(requested) {
+ match requested {
+ Some(logical) => push_unrequested_leaves(
+ field.data_type(),
+ logical,
+ &mut next_leaf,
+ matching,
+ &mut unrequested,
+ ),
+ None => next_leaf += leaf_count(field.data_type()),
+ }
+ }
+ // Emitted in depth-first order, so already sorted for `is_requested`'s binary search.
+ self.unrequested_leaves = unrequested;
+ self
+ }
+}
+
+/// The field-matching rules of the schema adapter's nested narrowing
+/// (`parquet_convert_struct_to_struct`): names fold per `case_sensitive`, and Parquet field ids
+/// select fields when `use_field_id` is set.
+#[derive(Debug, Clone, Copy)]
+struct FieldMatching {
+ case_sensitive: bool,
+ use_field_id: bool,
+}
+
+/// Appends to `out` the depth-first leaf ordinals of `physical` (counting from `next_leaf`,
+/// which advances past every leaf of `physical`) that reading it as `requested` drops.
+///
+/// Recurses through exactly the pairings `parquet_convert_array` narrows, and no others: a
+/// struct child is dropped only when NO requested child selects it by either rule the struct
+/// convert uses -- folded name, or Parquet field id when ids are in play -- and an ambiguous
+/// child (several requested children select it) is kept; `List` pairs with `List` by element
+/// type, and `Map` with a `Map` of the same key ordering by its entries, positionally. Any
+/// other pairing -- a leaf, a `LargeList` / `FixedSizeList` / dictionary, a map whose ordering
+/// differs, or a shape mismatch -- is handed to arrow's cast or passed through whole by the
+/// convert, so it keeps every leaf. Keeping a superset of what the narrowing reads is always
+/// safe (a spurious check at worst); dropping a leaf the narrowing reads would skip its
+/// rebase, so every doubt resolves to "requested".
+fn push_unrequested_leaves(
+ physical: &DataType,
+ requested: &DataType,
+ next_leaf: &mut usize,
+ matching: FieldMatching,
+ out: &mut Vec<usize>,
+) {
+ match (physical, requested) {
+ (DataType::Struct(physical_fields), DataType::Struct(requested_fields)) => {
+ let names: Vec<&str> = physical_fields
+ .iter()
+ .chain(requested_fields.iter())
+ .map(|f| f.name().as_str())
+ .collect();
+ // A fold failure means the names could not be compared at all; keeping every leaf
+ // requested is the safe superset, the same as the pass-through pairings below.
+ let Ok(folded) = fold_names(&names, matching.case_sensitive) else {
+ *next_leaf += leaf_count(physical);
+ return;
+ };
+ let (physical_folded, requested_folded) = folded.split_at(physical_fields.len());
+ for (i, child) in physical_fields.iter().enumerate() {
+ let child_id = if matching.use_field_id {
+ field_id(child)
+ } else {
+ None
+ };
+ let mut selectors = requested_fields.iter().enumerate().filter(|(j, r)| {
+ requested_folded[*j] == physical_folded[i]
+ || (child_id.is_some() && field_id(r) == child_id)
+ });
+ match (selectors.next(), selectors.next()) {
+ (None, _) => {
+ let n = leaf_count(child.data_type());
+ out.extend(*next_leaf..*next_leaf + n);
+ *next_leaf += n;
+ }
+ (Some((_, requested_child)), None) => push_unrequested_leaves(
+ child.data_type(),
+ requested_child.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (Some(_), Some(_)) => *next_leaf += leaf_count(child.data_type()),
+ }
+ }
+ }
+ (DataType::List(physical_item), DataType::List(requested_item)) => push_unrequested_leaves(
+ physical_item.data_type(),
+ requested_item.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (
+ DataType::Map(physical_entries, physical_sorted),
+ DataType::Map(requested_entries, requested_sorted),
+ ) if physical_sorted == requested_sorted => {
+ match (physical_entries.data_type(), requested_entries.data_type()) {
+ (DataType::Struct(physical_kv), DataType::Struct(requested_kv))
+ if physical_kv.len() == requested_kv.len() =>
+ {
+ for (p, r) in physical_kv.iter().zip(requested_kv.iter()) {
+ push_unrequested_leaves(
+ p.data_type(),
+ r.data_type(),
+ next_leaf,
+ matching,
+ out,
+ );
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+}
+
+/// The writer time zone recorded in `metadata`, classified for timestamp rebasing. Mirrors the
+/// `Option(lookupFileMeta(SPARK_TIMEZONE_METADATA_KEY))` lookup Spark's `getRebaseSpec` performs
+/// for every LEGACY resolution, conf-fallback included; Spark substitutes the JVM default zone
+/// when the key is absent (`RebaseSpec.timeZone`), which is unavailable natively, so an absent or
+/// non-UTC zone classifies as [`WriterTimeZone::OtherOrUnknown`] (dates still rebase fully --
+/// the day rebase is zone-free -- while ancient timestamps refuse rather than guess).
+fn writer_time_zone(metadata: &HashMap<String, String>) -> WriterTimeZone {
+ match metadata.get(SPARK_TIMEZONE_KEY) {
+ Some(tz) if UTC_EQUIVALENT_TIMEZONES.contains(&tz.as_str()) => WriterTimeZone::Utc,
+ _ => WriterTimeZone::OtherOrUnknown,
+ }
+}
+
+/// One spec resolution, mirroring Spark's `DataSourceUtils.getRebaseSpec` exactly: a Spark
+/// version below `min_version` (lexicographic comparison, same as the Scala `String.<`) or a
+/// present legacy flag means LEGACY; a Spark version at/after `min_version` without the flag
+/// means CORRECTED; no Spark version at all falls back to `conf_mode`, the session read conf
+/// forwarded from the JVM (`getRebaseSpec`'s `modeByConfig` fallback, its ONLY use of the
+/// conf): CORRECTED passes values through, LEGACY rebases (with the writer zone from the
+/// file's `org.apache.spark.timeZone` key, same lookup as the metadata-driven LEGACY path),
+/// and EXCEPTION refuses ancient values as [`RebasePolicy::CheckAncient`].
+fn resolve_spec(
+ metadata: &HashMap<String, String>,
+ min_version: &str,
+ legacy_key: &str,
+ conf_mode: RebaseReadMode,
+) -> RebasePolicy {
+ match metadata.get(SPARK_VERSION_METADATA_KEY) {
+ None => match conf_mode {
+ RebaseReadMode::Corrected => RebasePolicy::Corrected,
+ RebaseReadMode::Legacy => RebasePolicy::Legacy(writer_time_zone(metadata)),
+ RebaseReadMode::Exception => RebasePolicy::CheckAncient,
+ },
+ Some(version) => {
+ if version.as_str() < min_version || metadata.contains_key(legacy_key) {
+ RebasePolicy::Legacy(writer_time_zone(metadata))
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+ }
+}
+
+/// Resolves the per-file rebase policies from a file's arrow schema: the parquet footer's
+/// key-value pairs in its metadata decide the specs (the datetime spec uses min version
+/// `3.0.0` and the INT96 spec `3.1.0`, matching `DataSourceUtils.datetimeRebaseSpec` /
+/// `int96RebaseSpec`; `session_modes` supplies the per-spec conf fallback for files without
+/// Spark writer metadata), and the reader factory's INT96 stamp -- validated against the
+/// schema's leaf structure -- attributes each timestamp leaf to its spec.
+pub(crate) fn resolve_file_rebase_policies(
+ physical_file_schema: &Schema,
+ session_modes: SessionRebaseModes,
+) -> FileRebasePolicies {
+ let metadata = physical_file_schema.metadata();
+ let datetime_spec = resolve_spec(
+ metadata,
+ "3.0.0",
+ SPARK_LEGACY_DATETIME_KEY,
+ session_modes.datetime,
+ );
+ let int96_spec = resolve_spec(
+ metadata,
+ "3.1.0",
+ SPARK_LEGACY_INT96_KEY,
+ session_modes.int96,
+ );
+ FileRebasePolicies {
+ date: datetime_spec,
+ int64_timestamp: datetime_spec,
+ int96_timestamp: int96_spec,
+ int96_leaves: Int96Attribution::from_schema(physical_file_schema),
+ unrequested_leaves: Vec::new(),
+ }
+}
+
+/// Number of primitive leaves `dt` contains in a depth-first walk -- the same count and order
+/// parquet-rs uses when it maps the file's `SchemaDescriptor` columns onto the arrow schema, so
+/// arrow-side leaf ordinals line up with [`int96_leaf_stamp`]'s.
+fn leaf_count(dt: &DataType) -> usize {
+ match dt {
+ DataType::Struct(fields) => fields.iter().map(|f| leaf_count(f.data_type())).sum(),
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::Map(f, _) => leaf_count(f.data_type()),
+ DataType::Dictionary(_, value) => leaf_count(value),
+ DataType::RunEndEncoded(_, value) => leaf_count(value.data_type()),
+ DataType::Union(fields, _) => fields.iter().map(|(_, f)| leaf_count(f.data_type())).sum(),
+ _ => 1,
+ }
+}
+
+/// Appends the policy of every leaf of `dt`, in depth-first order, to `out`, consuming leaf
+/// ordinals from `next_leaf` (exactly [`leaf_count`] of them). Only `Date32` and
+/// timezone-carrying timestamps have a policy to apply, and only when the query reads the
+/// leaf; timezone-free timestamps are `TIMESTAMP_NTZ`, which Spark never rebases, and every
+/// other leaf is the identity ([`RebasePolicy::Corrected`]).
+fn leaf_policies(
+ dt: &DataType,
+ next_leaf: &mut usize,
+ policies: &FileRebasePolicies,
+ out: &mut Vec<RebasePolicy>,
+) {
+ match dt {
+ DataType::Date32 => {
+ out.push(policies.date_policy(*next_leaf));
+ *next_leaf += 1;
+ }
+ DataType::Timestamp(_, Some(_)) => {
Review Comment:
Fixed in 4b7ad02. A leaf read as `TIMESTAMP_NTZ` now takes no rebase policy whatever its physical type, so an INT96 or adjusted INT64 column read as NTZ returns the value unchanged under every mode. DATE read as `TIMESTAMP_NTZ` keeps the date policy, as `DateToTimestampNTZWithRebaseUpdater` does on 4.x.
Your case is `timestamps_requested_as_ntz_are_never_rebased` in `parquet_exec.rs` (INT96 words `(0, 0, 2378497)`, requested `Timestamp(Microsecond, None)`, `EXCEPTION`, `CORRECTED` and `LEGACY` all return `1800-01-01T00:00:00`), and the Delta test "INT96 and adjusted INT64 timestamps read as TIMESTAMP_NTZ are never rebased" runs it through `CONVERT TO DELTA` on Spark 4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4166646868
##########
native/core/src/parquet/datetime_rebase.rs:
##########
@@ -0,0 +1,3192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Per-file datetime calendar-rebase handling for the parquet scan.
+//!
+//! Spark 2.4 and earlier wrote dates and timestamps in the hybrid Julian + Gregorian calendar;
+//! Spark 3.0+ uses the proleptic Gregorian calendar and records the calendar policy of every
+//! file it writes in the parquet footer's key-value metadata (`org.apache.spark.version`,
+//! `org.apache.spark.legacyDateTime`, `org.apache.spark.legacyINT96`,
+//! `org.apache.spark.timeZone`). Spark's reader resolves the rebase policy from EACH FILE's
+//! writer metadata (`DataSourceUtils.datetimeRebaseSpec` / `int96RebaseSpec`) -- the session's
+//! `spark.sql.parquet.datetimeRebaseModeInRead` conf only applies to files whose metadata does
+//! not decide the policy on its own -- so a reader that ignores the metadata silently returns
+//! values shifted by up to ten days for dates before 1582-10-15 (e.g. `1500-01-01` reads as
+//! `1500-01-10`).
+//!
+//! This module mirrors that per-file resolution: [`resolve_file_rebase_policies`] computes the
+//! date / INT64-timestamp / INT96-timestamp policies from a file's arrow schema metadata (the
+//! parquet key-value pairs survive the parquet -> arrow schema conversion), and
+//! [`wrap_datetime_rebase`] wraps the per-file rewritten expressions' column references in a
+//! [`SparkDatetimeRebaseExpr`] that rebases values exactly where that is possible without the
+//! JVM's historical timezone tables (dates always; timestamps for a fixed UTC writer zone) and
+//! refuses -- rather than silently corrupting -- ancient values it cannot rebase. Nested
+//! columns are rebuilt leaf by leaf (struct / list / map / fixed-size list / dictionary), each
+//! leaf under its own policy, with nulls and offsets preserved. Modern values are always the
+//! identity under every policy: from 1582-10-15 onward for dates, and from
+//! [`LAST_SWITCH_JULIAN_TS_SECONDS`] (1900-01-01T00:00:00Z, Spark's
+//! `RebaseDateTime.lastSwitchJulianTs`) onward for timestamps.
+//!
+//! Spark applies `datetimeRebaseSpec` to INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns
+//! and `int96RebaseSpec` to INT96 columns. The two physical types are indistinguishable in the
+//! arrow schema DataFusion hands the expression adapter (both surface as `Timestamp(us, "UTC")`
+//! after INT96 coercion), so Comet's parquet reader factory stamps the file's INT96 leaf
+//! ordinals -- taken from the parquet footer's own `SchemaDescriptor` -- into the key-value
+//! metadata under [`INT96_LEAVES_METADATA_KEY`] before the arrow schema is derived (see
+//! [`stamp_int96_leaves`] and `eager_page_index_reader_factory.rs`), and the adapter attributes
+//! every timestamp leaf to its spec from that stamp. Without a stamp, the two specs are merged:
+//! agreement decides, disagreement degrades to [`RebasePolicy::CheckAncient`].
+//!
+//! The wrapper sits BENEATH the schema adapter's nested narrowing (the struct -> struct convert
+//! that keeps only the requested children), which is what keeps those ordinals physical -- but
+//! it means the wrapper sees every physical child, requested or not. Spark only ever decodes
+//! the requested nested schema, so [`FileRebasePolicies::restrict_to_requested`] marks the
+//! physical leaves the narrowing drops as the identity: an unrequested ancient `s.ts` never
+//! blocks `select s.d`, exactly as in Spark.
+//!
+//! The same pairing decides a timestamp leaf's policy by the type the query READS it as, since
+//! `ParquetVectorUpdaterFactory.getUpdater` keys on the requested Spark type, not the parquet
+//! annotation: a leaf read as `TIMESTAMP_NTZ` never rebases (INT96 or INT64; Spark 4.x's
+//! `BinaryToSQLTimestampUpdater` / `LongUpdater` consult no mode, and Spark 3.x refuses the
+//! INT96 and adjusted-INT64 pairings outright, which Comet's `allow_timestamp_ltz_to_ntz` gate
+//! reproduces); a leaf read as `TIMESTAMP` rebases under the datetime spec even when the file
+//! declares it `isAdjustedToUTC=false` (`isTimestampTypeMatched` checks the unit only); and a
+//! `DATE` leaf keeps the date policy whether read as `DATE` or, on Spark 4.x, as
+//! `TIMESTAMP_NTZ` (`DateToTimestampNTZWithRebaseUpdater`).
+//!
+//! Currently only enabled by the Delta scan arms via
+//! `SparkParquetOptions::rebase_from_file_metadata`, which also carries the session read modes
+//! ([`SessionRebaseModes`], forwarded from the JVM) that decide the policy for files without
+//! Spark writer metadata; the plain NativeScan keeps its documented no-rebase behavior (see
+//! the compatibility guide and issue #5010).
+
+use std::collections::HashMap;
+use std::fmt::{self, Display};
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Date32Array, FixedSizeListArray, GenericListArray, MapArray,
+ OffsetSizeTrait, PrimitiveArray, RecordBatch, StructArray,
+};
+use arrow::datatypes::{
+ ArrowPrimitiveType, ArrowTimestampType, DataType, Date32Type, FieldRef, Schema, SchemaRef,
+ TimeUnit, TimestampMicrosecondType, TimestampMillisecondType, TimestampNanosecondType,
+ TimestampSecondType,
+};
+use arrow::error::ArrowError;
+use datafusion::common::tree_node::{Transformed, TreeNode};
+use datafusion::common::{DataFusionError, Result as DataFusionResult};
+use datafusion::physical_expr::expressions::Column;
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::ColumnarValue;
+use parquet::basic::Type as ParquetPhysicalType;
+use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaData};
+use parquet::schema::types::SchemaDescriptor;
+
+use super::name_fold::fold_names;
+use super::parquet_support::field_id;
+
+/// Footer key naming the Spark release that wrote the file; absent for non-Spark writers.
+const SPARK_VERSION_METADATA_KEY: &str = "org.apache.spark.version";
+/// Present (empty value) when the file's dates and INT64 timestamps were written with
+/// `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_DATETIME_KEY: &str = "org.apache.spark.legacyDateTime";
+/// Present (empty value) when the file's INT96 timestamps were written with
+/// `spark.sql.parquet.int96RebaseModeInWrite=LEGACY`.
+const SPARK_LEGACY_INT96_KEY: &str = "org.apache.spark.legacyINT96";
+/// The writer session's time zone, stamped alongside either legacy flag.
+const SPARK_TIMEZONE_KEY: &str = "org.apache.spark.timeZone";
+
+/// Key-value metadata entry Comet's parquet reader factory adds to a file's footer metadata
+/// (in memory only, never written back) so the expression adapter can tell INT96 timestamp
+/// columns from INT64 ones after both have been coerced to the same arrow type. Value:
+/// `"<leaf count>:<comma-separated INT96 leaf ordinals>"`, where leaves are the file's
+/// primitive columns in `SchemaDescriptor::columns()` order -- the same depth-first order
+/// parquet-rs assigns arrow leaves, so an arrow-side depth-first walk lines up with it. The
+/// leaf count lets the reader detect a stamp that does not describe the schema it is paired
+/// with (see [`Int96Attribution::from_schema`]).
+pub(crate) const INT96_LEAVES_METADATA_KEY: &str = "comet.int96_leaf_columns";
+
+/// Day of the Gregorian cutover (1582-10-15) as days since the epoch; rebasing is the identity
+/// from this day onward. Same value as Spark's `RebaseDateTime.lastSwitchJulianDay`.
+const LAST_SWITCH_JULIAN_DAY: i32 = -141427;
+
+/// Spark's `RebaseDateTime.lastSwitchJulianTs` (and `lastSwitchGregorianTs`) in seconds since
+/// the epoch: 1900-01-01T00:00:00Z. Spark derives it as the latest switch instant across every
+/// zone in its `julian-gregorian-rebase-micros.json` table (`getLastSwitchTs`, which also
+/// asserts the calendars' difference is zero for every zone from then on): most zones ran on
+/// local mean time before 1900, so the last instant at which rebasing changes a value in ANY
+/// zone is 1900-01-01T00:00:00Z, not the 1582 cutover. `createTimestampRebaseFuncInRead`
+/// under `EXCEPTION` throws exactly for `micros < lastSwitchJulianTs` (after converting
+/// `TIMESTAMP_MILLIS` to micros), and `rebaseJulianToGregorianMicros` is the identity from it
+/// onward in every zone. The value is in seconds so it scales exactly to any timestamp unit.
+pub(crate) const LAST_SWITCH_JULIAN_TS_SECONDS: i64 = -2_208_988_800;
+
+/// The per-century differences between the Julian and proleptic Gregorian calendars, and the
+/// Julian-calendar switch days at which each difference starts to apply. Copied verbatim from
+/// Spark's `RebaseDateTime.julianGregDiffs` / `julianGregDiffSwitchDay` (which Spark generated
+/// from `localRebaseJulianToGregorianDays`); `rebase_julian_to_gregorian_days` must stay
+/// value-for-value equal to Spark's `rebaseJulianToGregorianDays`.
+const JULIAN_GREG_DIFFS: [i32; 14] = [2, 1, 0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, 0];
+const JULIAN_GREG_DIFF_SWITCH_DAY: [i32; 14] = [
+ -719164, -682945, -646420, -609895, -536845, -500320, -463795, -390745, -354220, -317695,
+ -244645, -208120, -171595, -141427,
+];
+
+/// Proleptic-Gregorian days since 1970-01-01 for a nominal civil date, via Howard Hinnant's
+/// `days_from_civil`. `d` may exceed the month's length; the excess rolls into the following
+/// month exactly like `LocalDate.of(y, m, 1).plusDays(d - 1)` in Spark's
+/// `localRebaseJulianToGregorianDays` (how the non-existent proleptic date `1000-02-29`,
+/// valid in the Julian calendar, lands on `1000-03-01`).
+fn days_from_civil(y: i64, m: i64, d: i64) -> i64 {
+ let y = if m <= 2 { y - 1 } else { y };
+ let era = y.div_euclid(400);
+ let yoe = y - era * 400; // [0, 399]
+ let mp = (m + 9) % 12; // [0, 11], March = 0
+ let doy = (153 * mp + 2) / 5 + d - 1;
+ let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
+ era * 146097 + doe - 719468
+}
+
+/// Julian-calendar civil date `(year, month, day)` for a day count since 1970-01-01 that labels
+/// days in the Julian calendar (astronomical year numbering: 1 BCE is year 0). Standard
+/// Julian-day-number conversion (E.G. Richards' algorithm), exact for any day.
+fn julian_day_to_civil(days: i64) -> (i64, i64, i64) {
+ // Integer (noon) Julian Day Number of this civil day: 1970-01-01 is JDN 2440588.
+ let jdn = days + 2_440_588;
+ let f = jdn + 1401;
+ let e = 4 * f + 3;
+ let g = e.rem_euclid(1461) / 4;
+ let h = 5 * g + 2;
+ let day = h.rem_euclid(153) / 5 + 1;
+ let month = (h / 153 + 2).rem_euclid(12) + 1;
+ let year = e.div_euclid(1461) - 4716 + (14 - month) / 12;
+ (year, month, day)
+}
+
+/// Exact port of Spark's `RebaseDateTime.rebaseJulianToGregorianDays`: reinterprets a day count
+/// written in the hybrid Julian + Gregorian calendar as the proleptic Gregorian day count of the
+/// same nominal civil date. Identity for days from 1582-10-15 onward. Days before the tables'
+/// range (before Julian `0001-01-01`) take the calendar-arithmetic path, mirroring Spark's
+/// `localRebaseJulianToGregorianDays` fallback.
+pub(crate) fn rebase_julian_to_gregorian_days(days: i32) -> i32 {
+ if days < JULIAN_GREG_DIFF_SWITCH_DAY[0] {
+ let (y, m, d) = julian_day_to_civil(days as i64);
+ (days_from_civil(y, m, 1) + (d - 1)) as i32
+ } else {
+ // Spark's rebaseDays: linear search from the most recent switch day.
+ let mut i = JULIAN_GREG_DIFF_SWITCH_DAY.len();
+ loop {
+ i -= 1;
+ if i == 0 || days >= JULIAN_GREG_DIFF_SWITCH_DAY[i] {
+ break;
+ }
+ }
+ days + JULIAN_GREG_DIFFS[i]
+ }
+}
+
+/// Timezone strings from `org.apache.spark.timeZone` that denote a fixed zero-offset zone in
+/// both `java.util.TimeZone` and `java.time`. Only for these is timestamp rebasing the pure
+/// nominal-date shift [`SparkDatetimeRebaseExpr::rebase_timestamp_utc`] computes; any other (or
+/// absent) zone needs the JVM's historical timezone tables and stays on the
+/// refuse-ancient-values path.
+const UTC_EQUIVALENT_TIMEZONES: [&str; 6] = ["UTC", "Etc/UTC", "GMT", "Etc/GMT", "Z", "+00:00"];
+
+/// How the writer's session time zone (if recorded) affects timestamp rebasing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum WriterTimeZone {
+ /// A fixed zero-offset zone: rebasing reduces to the exact nominal-date shift.
+ Utc,
+ /// Any other zone, or none recorded (pre-3.0 files): ancient values cannot be rebased
+ /// without the JVM's historical timezone data.
+ OtherOrUnknown,
+}
+
+/// One session-level datetime rebase read mode (a `LegacyBehaviorPolicy` value of
+/// `spark.sql.parquet.datetimeRebaseModeInRead` / `int96RebaseModeInRead`), consulted by
+/// [`resolve_file_rebase_policies`] ONLY for files whose footer metadata does not decide the
+/// policy on its own -- exactly the `getOrElse` fallback in Spark's
+/// `DataSourceUtils.getRebaseSpec`. Files that carry `org.apache.spark.version` ignore these
+/// modes entirely, on every Spark version.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) enum RebaseReadMode {
+ /// Refuse ancient values (Spark raises `SparkUpgradeException`); maps to
+ /// [`RebasePolicy::CheckAncient`]. The default mirrors the conservative posture used
+ /// before the conf was plumbed through (and Spark 3.x's own conf default).
+ #[default]
+ Exception,
+ /// Read values as proleptic Gregorian without rebasing.
+ Corrected,
+ /// Rebase from the hybrid Julian + Gregorian calendar.
+ Legacy,
+}
+
+impl RebaseReadMode {
+ /// Parses a `LegacyBehaviorPolicy` conf value. `SQLConf` validates and upper-cases the
+ /// session conf, but a per-relation `datetimeRebaseMode` option arrives verbatim, so the
+ /// match is case-insensitive. Anything unrecognized -- including the empty string a proto
+ /// producer that predates the field sends -- falls back to [`RebaseReadMode::Exception`],
+ /// which refuses ancient values rather than silently corrupting them.
+ pub(crate) fn from_conf_value(value: &str) -> Self {
+ match value.to_ascii_uppercase().as_str() {
+ "CORRECTED" => RebaseReadMode::Corrected,
+ "LEGACY" => RebaseReadMode::Legacy,
+ _ => RebaseReadMode::Exception,
+ }
+ }
+}
+
+/// The session's effective datetime rebase read modes, one per spec class (INT64
+/// dates/timestamps vs INT96 timestamps), forwarded from the JVM at planning time.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub(crate) struct SessionRebaseModes {
+ /// `spark.sql.parquet.datetimeRebaseModeInRead` (or the relation's `datetimeRebaseMode`).
+ pub datetime: RebaseReadMode,
+ /// `spark.sql.parquet.int96RebaseModeInRead` (or the relation's `int96RebaseMode`).
+ pub int96: RebaseReadMode,
+}
+
+/// Calendar policy of one file's date or timestamp columns, resolved from writer metadata the
+/// same way Spark's `DataSourceUtils.getRebaseSpec` resolves it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum RebasePolicy {
+ /// Written in the proleptic Gregorian calendar; values pass through untouched.
+ Corrected,
+ /// Written in the hybrid Julian + Gregorian calendar; values must be rebased.
+ Legacy(WriterTimeZone),
+ /// Policy could not be pinned down (contradictory flags, or a non-Spark writer under the
+ /// `EXCEPTION` read mode): modern values -- identical under either calendar -- pass,
+ /// ancient values raise. Mirrors Spark's `EXCEPTION` behavior (`SparkUpgradeException`).
+ CheckAncient,
+}
+
+/// Which of a file's leaf columns are physically INT96, from the stamp the parquet reader
+/// factory adds under [`INT96_LEAVES_METADATA_KEY`].
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) enum Int96Attribution {
+ /// No stamp, or a stamp whose leaf count does not match the schema it arrived with: the
+ /// INT64 and INT96 timestamp specs cannot be told apart per column and are merged.
+ Unknown,
+ /// Sorted leaf ordinals (depth-first over the file schema's primitive columns) that are
+ /// INT96; every other timestamp leaf is INT64.
+ Known(Vec<usize>),
+}
+
+impl Int96Attribution {
+ /// Parses the stamp out of `schema`'s metadata and validates its leaf count against the
+ /// schema's own depth-first leaf count, so a stamp that does not describe this schema (a
+ /// crafted footer key, or a cached-metadata mismatch) degrades to [`Self::Unknown`].
+ fn from_schema(schema: &Schema) -> Self {
+ let Some(stamp) = schema.metadata().get(INT96_LEAVES_METADATA_KEY) else {
+ return Int96Attribution::Unknown;
+ };
+ let Some((count, ordinals)) = stamp.split_once(':') else {
+ return Int96Attribution::Unknown;
+ };
+ let schema_leaves: usize = schema
+ .fields()
+ .iter()
+ .map(|f| leaf_count(f.data_type()))
+ .sum();
+ if count.parse::<usize>().ok() != Some(schema_leaves) {
+ return Int96Attribution::Unknown;
+ }
+ let parsed: Option<Vec<usize>> = if ordinals.is_empty() {
+ Some(Vec::new())
+ } else {
+ ordinals
+ .split(',')
+ .map(|o| o.parse::<usize>().ok().filter(|o| *o < schema_leaves))
+ .collect()
+ };
+ match parsed {
+ Some(mut leaves) => {
+ leaves.sort_unstable();
+ Int96Attribution::Known(leaves)
+ }
+ None => Int96Attribution::Unknown,
+ }
+ }
+
+ /// `Some(true)` / `Some(false)` when the leaf is known to be INT96 / INT64, `None` when
+ /// the attribution is unknown.
+ fn is_int96(&self, leaf: usize) -> Option<bool> {
+ match self {
+ Int96Attribution::Unknown => None,
+ Int96Attribution::Known(leaves) => Some(leaves.binary_search(&leaf).is_ok()),
+ }
+ }
+}
+
+/// The [`INT96_LEAVES_METADATA_KEY`] value describing `schema`: its leaf count and the
+/// ordinals of its INT96 primitive columns.
+pub(crate) fn int96_leaf_stamp(schema: &SchemaDescriptor) -> String {
+ let ordinals: Vec<String> = schema
+ .columns()
+ .iter()
+ .enumerate()
+ .filter(|(_, column)| column.physical_type() == ParquetPhysicalType::INT96)
+ .map(|(ordinal, _)| ordinal.to_string())
+ .collect();
+ format!("{}:{}", schema.num_columns(), ordinals.join(","))
+}
+
+/// Returns a copy of `metadata` whose key-value metadata carries the [`int96_leaf_stamp`] of
+/// its own schema, or `None` when it already does (the common case after the first open of a
+/// file, since the caller caches the stamped copy). Any pre-existing entry under the key --
+/// a file cannot legitimately carry one -- is replaced, never trusted. Only the file-level
+/// key-value list changes; row groups and page indexes are carried over as-is. The parquet
+/// API cannot carry a file decryptor, nor `FileMetaData`'s crate-private encryption fields
+/// (encryption algorithm, footer signing key metadata), across this rebuild, so callers must
+/// not stamp opens that supply decryption properties -- and the only consumer, the Delta
+/// scan, declines every encrypted-parquet configuration before planning, so a parquet
+/// modular encryption file never reaches this path with or without those properties.
+pub(crate) fn stamp_int96_leaves(metadata: &ParquetMetaData) -> Option<ParquetMetaData> {
+ let file_metadata = metadata.file_metadata();
+ let stamp = int96_leaf_stamp(file_metadata.schema_descr());
+ let existing = file_metadata
+ .key_value_metadata()
+ .and_then(|kvs| kvs.iter().find(|kv| kv.key == INT96_LEAVES_METADATA_KEY))
+ .and_then(|kv| kv.value.as_deref());
+ if existing == Some(stamp.as_str()) {
+ return None;
+ }
+ let mut key_values: Vec<KeyValue> = file_metadata
+ .key_value_metadata()
+ .map(|kvs| {
+ kvs.iter()
+ .filter(|kv| kv.key != INT96_LEAVES_METADATA_KEY)
+ .cloned()
+ .collect()
+ })
+ .unwrap_or_default();
+ key_values.push(KeyValue::new(INT96_LEAVES_METADATA_KEY.to_string(), stamp));
+ let stamped_file_metadata = FileMetaData::new(
+ file_metadata.version(),
+ file_metadata.num_rows(),
+ file_metadata.created_by().map(str::to_string),
+ Some(key_values),
+ file_metadata.schema_descr_ptr(),
+ file_metadata.column_orders().cloned(),
+ );
+ Some(
+ ParquetMetaData::new(stamped_file_metadata, metadata.row_groups().to_vec())
+ .into_builder()
+ .set_column_index(metadata.column_index().cloned())
+ .set_offset_index(metadata.offset_index().cloned())
+ .build(),
+ )
+}
+
+/// Per-file rebase policies for the three affected column classes, plus the INT96
+/// attribution that selects between the two timestamp specs per leaf.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct FileRebasePolicies {
+ /// `DATE` columns, governed by `org.apache.spark.legacyDateTime` alone.
+ pub date: RebasePolicy,
+ /// INT64 `TIMESTAMP_MICROS` / `TIMESTAMP_MILLIS` columns, adjusted to UTC or not: the
+ /// datetime spec (same resolution as `date`), as Spark's `ParquetVectorUpdaterFactory`
+ /// selects for INT64 read as `TIMESTAMP`.
+ pub int64_timestamp: RebasePolicy,
+ /// INT96 columns: the INT96 spec (`org.apache.spark.legacyINT96`, min version 3.1.0).
+ pub int96_timestamp: RebasePolicy,
+ /// Which timestamp leaves are INT96. See [`Int96Attribution`].
+ pub int96_leaves: Int96Attribution,
+ /// Sorted depth-first leaf ordinals -- over the physical file schema, the same ordinals
+ /// `int96_leaves` uses -- that the query does not read: nested children the schema
+ /// adapter's struct narrowing drops before any value leaves the scan. Spark never decodes
+ /// them either, so their policy is the identity whatever the file's calendar. Empty until
+ /// [`Self::restrict_to_requested`] runs (every leaf requested).
+ pub unrequested_leaves: Vec<usize>,
+ /// Sorted physical leaf ordinals of timezone-carrying timestamps (INT96, or INT64 with
+ /// `isAdjustedToUTC=true`) the query reads as `TIMESTAMP_NTZ`. Spark decodes those with
+ /// `BinaryToSQLTimestampUpdater` / `LongUpdater`, which never rebase, so their policy is
+ /// the identity whatever the file's calendar. Filled by [`Self::restrict_to_requested`].
+ pub ntz_requested_leaves: Vec<usize>,
+ /// Sorted physical leaf ordinals of timezone-free INT64 timestamps
+ /// (`isAdjustedToUTC=false`) the query reads as `TIMESTAMP`. Spark's INT64 branch checks
+ /// only the unit (`isTimestampTypeMatched`) and hands a `TimestampType` request to
+ /// `LongWithRebaseUpdater` under the datetime spec, adjusted or not, so these leaves take
+ /// the same policy as adjusted INT64 leaves. Filled by [`Self::restrict_to_requested`].
+ pub ltz_requested_leaves: Vec<usize>,
+}
+
+/// The leaf ordinals [`push_unrequested_leaves`] records while pairing a physical type with
+/// the type the query reads it as. Each list is emitted in depth-first order, so it is already
+/// sorted for the binary searches in [`FileRebasePolicies`].
+#[derive(Debug, Default)]
+struct LeafPairing {
+ unrequested: Vec<usize>,
+ ntz_requested: Vec<usize>,
+ ltz_requested: Vec<usize>,
+}
+
+impl FileRebasePolicies {
+ /// True when some policy is not the plain proleptic-Gregorian pass-through, i.e. when the
+ /// per-column wrap in [`wrap_datetime_rebase`] can install anything at all.
+ pub(crate) fn any_rebase_needed(&self) -> bool {
+ self.date != RebasePolicy::Corrected
+ || self.int64_timestamp != RebasePolicy::Corrected
+ || self.int96_timestamp != RebasePolicy::Corrected
+ }
+
+ fn is_requested(&self, leaf: usize) -> bool {
+ self.unrequested_leaves.binary_search(&leaf).is_err()
+ }
+
+ /// The policy of the `Date32` leaf at depth-first ordinal `leaf`: the file's date policy,
+ /// or the identity when the query does not read that leaf.
+ fn date_policy(&self, leaf: usize) -> RebasePolicy {
+ if self.is_requested(leaf) {
+ self.date
+ } else {
+ RebasePolicy::Corrected
+ }
+ }
+
+ /// The policy of the timezone-carrying timestamp leaf at depth-first ordinal `leaf`: the
+ /// identity when the query does not read it or reads it as `TIMESTAMP_NTZ` (Spark's NTZ
+ /// updaters never rebase; on Spark 3.x the pairing is refused before any rebase decision,
+ /// which Comet's `allow_timestamp_ltz_to_ntz` gate reproduces); otherwise its physical
+ /// type's spec when the attribution is known, or else the two specs merged -- agreement
+ /// decides, disagreement degrades to [`RebasePolicy::CheckAncient`], which still passes
+ /// every modern value and refuses only ancient ones.
+ fn timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) || self.ntz_requested_leaves.binary_search(&leaf).is_ok() {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ Some(false) => self.int64_timestamp,
+ None if self.int64_timestamp == self.int96_timestamp => self.int64_timestamp,
+ None => RebasePolicy::CheckAncient,
+ }
+ }
+
+ /// The policy of the timezone-free timestamp leaf at depth-first ordinal `leaf` (INT64 with
+ /// `isAdjustedToUTC=false`): the identity unless the query reads it as `TIMESTAMP`, which
+ /// Spark decodes with `LongWithRebaseUpdater` under the datetime spec exactly like an
+ /// adjusted INT64 leaf. The stamp is still consulted so a leaf it names INT96 follows the
+ /// INT96 spec; without a stamp the physical type itself proves INT64, so the two specs are
+ /// not merged.
+ fn tz_free_timestamp_policy(&self, leaf: usize) -> RebasePolicy {
+ if !self.is_requested(leaf) || self.ltz_requested_leaves.binary_search(&leaf).is_err() {
+ return RebasePolicy::Corrected;
+ }
+ match self.int96_leaves.is_int96(leaf) {
+ Some(true) => self.int96_timestamp,
+ _ => self.int64_timestamp,
+ }
+ }
+
+ /// These policies with every physical leaf the query does not read marked the identity,
+ /// and every timestamp leaf whose requested type differs from its physical one in timezone
+ /// presence recorded, so [`leaf_policies`] can pick the policy Spark's
+ /// `ParquetVectorUpdaterFactory.getUpdater` picks for the REQUESTED type.
+ /// `requested` pairs each top-level field of `physical_schema` (by position) with the type
+ /// of the logical field the schema adapter narrows it to -- `None` for a column without a
+ /// logical counterpart, whose leaves are left as they are (no expression reads it anyway).
+ /// Nested children pair the way the adapter's struct convert selects them (see
+ /// [`push_unrequested_leaves`]); the INT96 attribution is untouched, since the ordinals
+ /// stay physical. `requested` is parallel to the schema's fields; should a caller pass a
+ /// shorter slice, the trailing columns simply keep every leaf (the safe direction).
+ pub(crate) fn restrict_to_requested(
+ mut self,
+ physical_schema: &Schema,
+ requested: &[Option<&DataType>],
+ case_sensitive: bool,
+ use_field_id: bool,
+ ) -> Self {
+ debug_assert_eq!(requested.len(), physical_schema.fields().len());
+ let matching = FieldMatching {
+ case_sensitive,
+ use_field_id,
+ };
+ let mut next_leaf = 0;
+ let mut pairing = LeafPairing::default();
+ for (field, requested) in physical_schema.fields().iter().zip(requested) {
+ match requested {
+ Some(logical) => push_unrequested_leaves(
+ field.data_type(),
+ logical,
+ &mut next_leaf,
+ matching,
+ &mut pairing,
+ ),
+ None => next_leaf += leaf_count(field.data_type()),
+ }
+ }
+ // Emitted in depth-first order, so already sorted for the binary searches.
+ self.unrequested_leaves = pairing.unrequested;
+ self.ntz_requested_leaves = pairing.ntz_requested;
+ self.ltz_requested_leaves = pairing.ltz_requested;
+ self
+ }
+}
+
+/// The field-matching rules of the schema adapter's nested narrowing
+/// (`parquet_convert_struct_to_struct`): names fold per `case_sensitive`, and Parquet field ids
+/// select fields when `use_field_id` is set.
+#[derive(Debug, Clone, Copy)]
+struct FieldMatching {
+ case_sensitive: bool,
+ use_field_id: bool,
+}
+
+/// Appends to `out.unrequested` the depth-first leaf ordinals of `physical` (counting from
+/// `next_leaf`, which advances past every leaf of `physical`) that reading it as `requested`
+/// drops, and records in `out.ntz_requested` / `out.ltz_requested` the timestamp leaves whose
+/// requested type has the opposite timezone presence (a timezone-carrying leaf read as
+/// `TIMESTAMP_NTZ`, a timezone-free leaf read as `TIMESTAMP`); the unit is irrelevant to
+/// either, as it is to Spark's `isTimestampTypeMatched`.
+///
+/// Recurses through exactly the pairings `parquet_convert_array` narrows, and no others: a
+/// struct child is dropped only when NO requested child selects it by either rule the struct
+/// convert uses -- folded name, or Parquet field id when ids are in play -- and an ambiguous
+/// child (several requested children select it) is kept; `List` pairs with `List` by element
+/// type, and `Map` with a `Map` of the same key ordering by its entries, positionally. Any
+/// other pairing -- a `LargeList` / `FixedSizeList` / dictionary, a map whose ordering
+/// differs, or a shape mismatch -- is handed to arrow's cast or passed through whole by the
+/// convert, so it keeps every leaf under its physical type's policy. Keeping a superset of
+/// what the narrowing reads is always safe (a spurious check at worst); dropping a leaf the
+/// narrowing reads would skip its rebase, so every doubt resolves to "requested". Timestamp
+/// leaves inside those pass-through shapes are never recorded either, so they keep the
+/// physical rule (a spurious check for an NTZ request, no rebase for a `TIMESTAMP` request of
+/// a timezone-free leaf); Spark's requested schemas never take those arrow shapes.
+fn push_unrequested_leaves(
+ physical: &DataType,
+ requested: &DataType,
+ next_leaf: &mut usize,
+ matching: FieldMatching,
+ out: &mut LeafPairing,
+) {
+ match (physical, requested) {
+ (DataType::Timestamp(_, Some(_)), DataType::Timestamp(_, None)) => {
+ out.ntz_requested.push(*next_leaf);
+ *next_leaf += 1;
+ }
+ (DataType::Timestamp(_, None), DataType::Timestamp(_, Some(_))) => {
+ out.ltz_requested.push(*next_leaf);
+ *next_leaf += 1;
+ }
+ (DataType::Struct(physical_fields), DataType::Struct(requested_fields)) => {
+ let names: Vec<&str> = physical_fields
+ .iter()
+ .chain(requested_fields.iter())
+ .map(|f| f.name().as_str())
+ .collect();
+ // A fold failure means the names could not be compared at all; keeping every leaf
+ // requested is the safe superset, the same as the pass-through pairings below.
+ let Ok(folded) = fold_names(&names, matching.case_sensitive) else {
+ *next_leaf += leaf_count(physical);
+ return;
+ };
+ let (physical_folded, requested_folded) = folded.split_at(physical_fields.len());
+ for (i, child) in physical_fields.iter().enumerate() {
+ let child_id = if matching.use_field_id {
+ field_id(child)
+ } else {
+ None
+ };
+ let mut selectors = requested_fields.iter().enumerate().filter(|(j, r)| {
+ requested_folded[*j] == physical_folded[i]
+ || (child_id.is_some() && field_id(r) == child_id)
+ });
+ match (selectors.next(), selectors.next()) {
+ (None, _) => {
+ let n = leaf_count(child.data_type());
+ out.unrequested.extend(*next_leaf..*next_leaf + n);
+ *next_leaf += n;
+ }
+ (Some((_, requested_child)), None) => push_unrequested_leaves(
+ child.data_type(),
+ requested_child.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (Some(_), Some(_)) => *next_leaf += leaf_count(child.data_type()),
+ }
+ }
+ }
+ (DataType::List(physical_item), DataType::List(requested_item)) => push_unrequested_leaves(
+ physical_item.data_type(),
+ requested_item.data_type(),
+ next_leaf,
+ matching,
+ out,
+ ),
+ (
+ DataType::Map(physical_entries, physical_sorted),
+ DataType::Map(requested_entries, requested_sorted),
+ ) if physical_sorted == requested_sorted => {
+ match (physical_entries.data_type(), requested_entries.data_type()) {
+ (DataType::Struct(physical_kv), DataType::Struct(requested_kv))
+ if physical_kv.len() == requested_kv.len() =>
+ {
+ for (p, r) in physical_kv.iter().zip(requested_kv.iter()) {
+ push_unrequested_leaves(
+ p.data_type(),
+ r.data_type(),
+ next_leaf,
+ matching,
+ out,
+ );
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
+ }
+ }
+ _ => *next_leaf += leaf_count(physical),
Review Comment:
Fixed in 5031c2ecd. A leaf that Spark reads through a numeric updater now gets no rebase policy: a DATE requested as TINYINT, SMALLINT, INT, BIGINT or DOUBLE, and an INT64 timestamp of any unit requested as BIGINT, matching `ParquetVectorUpdaterFactory` on 3.5.9, 4.0.4 and 4.1.3. Temporal requests keep their policy. The regression writes a metadata-free file with NANOS, MICROS and DATE columns at 1800-01-01, runs `CONVERT TO DELTA ... NO STATISTICS`, and reads them as BIGINT and INT with `nanosAsLong=true` under EXCEPTION.
One gap this leaves as it was: on 3.5 Spark refuses a DATE read as BIGINT or DOUBLE, while the native scan returns the integer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3881818485
##########
contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala:
##########
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning}
+import org.apache.spark.sql.execution.{FileSourceScanExec, InSubqueryExec, ReusedSubqueryExec, ScalarSubquery, SparkPlan, SubqueryAdaptiveBroadcastExec}
+import org.apache.spark.sql.execution.datasources.HadoopFsRelation
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.contrib.delta.DeltaSparkScanEnvelope
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+
+/**
+ * Native scan node for Delta Lake tables (contrib). Delta's own planning (log replay, snapshot
+ * resolution, partition pruning) has already run inside delta-spark by the time this node is
+ * created from the DSv1 [[FileSourceScanExec]]; file listing and split planning are delegated to
+ * a [[CometScanExec]] helper, and data reads execute through Comet's native DataFusion parquet
+ * machinery, inheriting row-group and page-index pruning.
+ *
+ * DPP: `runtimeFilters` is a constructor field included in equality, so its rewrite (via
+ * [[CometScanWithPlanData]]) survives plan copies -- a transient field would be dropped by
+ * `TreeNode.makeCopy` on MERGE re-planning (the CometIcebergNativeScanExec lesson).
+ */
+case class CometDeltaNativeScanExec(
+ override val nativeOp: Operator,
+ override val output: Seq[Attribute],
+ requiredSchema: StructType,
+ runtimeFilters: Seq[Expression],
+ dataFilters: Seq[Expression],
+ @transient relation: HadoopFsRelation,
+ originalPlan: FileSourceScanExec,
+ override val serializedPlanOpt: SerializedPlan,
+ sourceKey: String)
+ extends CometLeafExec
+ with CometScanWithPlanData {
+
+ override val nodeName: String = s"CometDeltaNativeScan $relation"
+
+ // Derived from (originalPlan, runtimeFilters), never stored: any copy of this node
+ // automatically gets a helper consistent with ITS runtimeFilters, avoiding the #3510 class of
+ // bug where a stored helper field desyncs from rewritten filters. Costs one extra file listing
+ // per executed instance; correctness over the duplicate driver-side listing.
+ //
+ // Forcing invariant: this lazy val is forced by the `metrics` override below, and AQE's UI
+ // plan-walk calls `.metrics` on every node MID-PLANNING, including while a DPP subquery is
+ // still an adaptive placeholder or a partition filter holds an unresolved ScalarSubquery (see
+ // `hasUnevaluableSubqueryFilter` below). That's safe ONLY because constructing `scanHelper` is a
+ // cheap case-class build with no file listing, and core's `CometScanExec.metrics` touches only
+ // `wrapped.driverMetrics` (populated by Spark's own planning) plus a static metric-node
+ // constructor -- neither file listing nor subquery resolution. If core's `metrics` ever touches
+ // either, forcing `scanHelper` here would resurrect the AQE mid-planning crashes this invariant
+ // prevents.
+ @transient private lazy val scanHelper: CometScanExec =
+ CometDeltaNativeScanExec.planningHelper(originalPlan, runtimeFilters)
+
+ // NOT lazy val: while a DPP subquery is still an adaptive placeholder, or a partition filter
+ // holds an unresolved scalar subquery, this returns a temporary value that must not be
+ // memoized -- after CometPlanAdaptiveDynamicPruningFilters rewrites the filters (DPP case) or
+ // AQE resolves the subquery (scalar case), later reads must see the real post-pruning
+ // partition count.
+ override def outputPartitioning: Partitioning =
+ if (hasUnevaluableSubqueryFilter) UnknownPartitioning(0)
+ else UnknownPartitioning(perPartitionData.length)
+
+ // runtimeFilters IS scanHelper.partitionFilters element-for-element, so checking runtimeFilters
+ // here avoids constructing/forcing the derived scanHelper just to read partitioning. The
+ // InSubqueryExec placeholder shapes mirror
+ // CometPlanAdaptiveDynamicPruningFilters.extractSABData + hasWrappedSAB -- keep in sync. The
+ // ScalarSubquery case is probed rather than treated as permanently unevaluable: Spark exposes no
+ // public finished/updated flag on ExecSubqueryExpression, but `eval()` doubles as one -- it only
+ // reads the cached `result` behind a `require(updated, ...)` guard, while the subquery is
+ // actually run by `updateResult()` (invoked separately during prepare/AQE), never by `eval()`.
+ // Once resolved, outputPartitioning below reports the real perPartitionData.length instead of
+ // staying at zero -- a fused native parent's buildNativeContext requires that count to match.
+ private def hasUnevaluableSubqueryFilter: Boolean =
+ runtimeFilters.exists(_.exists {
+ // Match `e: InSubqueryExec` and dispatch on e.plan rather than unapplying InSubqueryExec
+ // directly: its unapply arity differs across Spark versions and this module ships no
+ // version shim.
+ case e: InSubqueryExec => isAdaptivePlaceholder(e.plan)
+ case s: ScalarSubquery => !isScalarSubqueryResolved(s)
+ case _ => false
+ })
+
+ // `eval()` never triggers the subquery's execution: on a resolved subquery it is a pure cached
+ // read of `result` (verified against bytecode: `Predef.require(updated(), ...)` then a plain
+ // field read), so this probe is safe to call repeatedly, including from AQE's mid-planning plan
+ // walks. Pre-resolution, the ONLY throw is `require`'s `IllegalArgumentException`; catch exactly
+ // that, since anything else escaping is a genuine bug we must not mask as unpartitioned.
+ private def isScalarSubqueryResolved(s: ScalarSubquery): Boolean =
+ try {
+ s.eval()
+ true
+ } catch {
+ case _: IllegalArgumentException => false
+ }
+
+ private def isAdaptivePlaceholder(p: SparkPlan): Boolean = p match {
+ case ReusedSubqueryExec(inner) => isAdaptivePlaceholder(inner)
+ case _: CometSubqueryAdaptiveBroadcastExec => true
+ case _: SubqueryAdaptiveBroadcastExec => true
+ case _ => false
+ }
+
+ override lazy val outputOrdering: Seq[SortOrder] = originalPlan.outputOrdering
+
+ override def dynamicPruningFilters: Seq[Expression] = runtimeFilters
+
+ override def withDynamicPruningFilters(filters: Seq[Expression]): SparkPlan = {
+ // A real copy: runtimeFilters is a constructor field included in equality, so the copy
+ // survives enclosing-block rebuilds, and the derived scanHelper picks up the rewritten
+ // filters automatically.
+ copy(runtimeFilters = filters)
+ }
+
+ /**
+ * Lazy split-mode serialization, mirroring CometNativeScanExec: common data was serialized at
+ * planning; per-partition file lists serialize here, at execution time.
+ */
+ @transient private lazy val serializedPartitionData
+ : (Array[Byte], Array[Array[Byte]], Array[Seq[String]]) = {
+ // Resolve the helper's DPP subqueries: it holds its own InSubqueryExec instances that
+ // Spark's expressions walk does not see (the helper is derived, not a child).
+ scanHelper.partitionFilters.foreach {
+ case DynamicPruningExpression(e: InSubqueryExec) if e.values().isEmpty =>
+ e.updateResult()
+ case _ =>
+ }
+
+ val commonBytes = {
+ val deltaScan = DeltaSparkScanEnvelope.unpack(nativeOp)
+ // Scalar subqueries in dataFilters were unresolved at planning; resolve them now and
+ // append them as pushed filters, as CometNativeScanExec.serializedPartitionData does.
+ val subqueryFilters = org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .resolvedSubqueryFilters(dataFilters, output, requiredSchema, conf)
+ val common = if (subqueryFilters.isEmpty) {
+ deltaScan.getCommon
+ } else {
+ val builder = deltaScan.getCommon.toBuilder
+ subqueryFilters.foreach(builder.addDataFilters)
+ builder.build()
+ }
+ OperatorOuterClass.DeltaSparkScan
+ .newBuilder()
+ .setCommon(common)
+ .setDeltaCommon(deltaScan.getDeltaCommon)
+ .build()
+ .toByteArray
+ }
+
+ val filePartitions = scanHelper.getFilePartitions()
+
+ val tableRoot = DeltaSparkScanEnvelope.unpack(nativeOp).getDeltaCommon.getTableRoot
+ val perPartitionBytes = filePartitions.map { filePartition =>
+ org.apache.comet.contrib.delta.CometDeltaNativeScan
+ .serializePartition(filePartition, originalPlan, tableRoot)
+ }.toArray
+
+ val perPartitionPaths = filePartitions.map(_.files.map(_.filePath.toString).toSeq).toArray
+
+ (commonBytes, perPartitionBytes, perPartitionPaths)
+ }
+
+ override def commonData: Array[Byte] = serializedPartitionData._1
+
+ override def perPartitionData: Array[Array[Byte]] = serializedPartitionData._2
+
+ def perPartitionFilePaths: Array[Seq[String]] = serializedPartitionData._3
+
+ override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+ val nativeMetrics = CometMetricNode.fromCometPlan(this)
+ val serializedPlan = CometExec.serializeNativePlan(nativeOp)
+
+ new CometExecRDD(
+ sparkContext,
+ Seq.empty,
+ Map(sourceKey -> commonData),
+ Map(sourceKey -> perPartitionData),
+ serializedPlan,
+ perPartitionData.length,
+ output.length,
+ nativeMetrics,
+ Seq.empty,
+ None,
+ Seq.empty,
+ perPartitionFilePaths = perPartitionFilePaths)
Review Comment:
Wired up. The RDD registers the input-metrics callback now and the fused path recognizes contrib scans through the shared trait rather than the concrete class, so future contribs inherit it. Task-listener regressions cover the standalone and fused reads, both were zero before the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4156598511
##########
native/core/src/parquet/objectstore/s3.rs:
##########
@@ -1002,10 +1037,97 @@ impl CredentialProviderMetadata {
#[cfg(test)]
mod tests {
+ use std::collections::BTreeSet;
use std::sync::atomic::{AtomicI32, Ordering};
use super::*;
+ /// Discovery-harness test (see `NATIVE_S3A_CONFIG_PROPERTIES`'s doc): mechanically re-derives
+ /// the set of `fs.s3a.*` property suffixes this file actually resolves by scanning this
+ /// file's OWN source text (via `include_str!`) for every `get_config(configs, bucket, ...)`/
+ /// `get_config_trimmed(configs, bucket, ...)` call site, resolving an identifier argument
+ /// (e.g. `PROVIDER_CLASS_PROPERTY`) through its own `const NAME: &str = "..."` definition, and
+ /// asserts the result is EXACTLY `NATIVE_S3A_CONFIG_PROPERTIES`. This fails loudly the moment
+ /// a call site is added, removed, or its literal changes without updating that constant --
+ /// which is exactly the class of bug (a config key silently added to one side of the
+ /// Scala/Rust boundary but not the other) that let a Hadoop-side resolution rule diverge
+ /// unnoticed in the round-15 SSE-C finding.
+ ///
+ /// The `configs, property` call inside `get_config_trimmed`'s own body (a passthrough of its
+ /// own `property` parameter, not a call site naming a fixed config key) is deliberately
+ /// excluded by name.
+ #[test]
+ fn native_s3a_config_properties_matches_call_sites() {
Review Comment:
This test scans `s3.rs` as text and fails when a `get_config` call site is added without updating `NATIVE_S3A_CONFIG_PROPERTIES`. `DeltaScanContribSuite` then parses the same constant and requires `DeltaScanSupport.AllS3ConfigKeys` to cover it. So someone who adds an S3 option in core fixes the Rust test, gets a green PR run, and then fails in the merge queue where `delta_3_5` first runs, until they also edit the Delta contrib. That evicts their PR and blocks the queue for others. Would it make sense to run that one discovery-harness test in the PR tier whenever `native/core/src/parquet/objectstore/**` changes, so the first signal is not the queue? Or could the contrib comparator be made advisory for keys it does not need to compare?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r4246896564
##########
dev/ci/compute-changes.py:
##########
@@ -577,6 +604,12 @@
"iceberg_1_9": ["nightly", "label:run-iceberg-tests"],
"iceberg_1_10": ["nightly", "label:run-iceberg-tests"],
"iceberg_1_11": ["queue", "label:run-iceberg-tests"],
+ # Same shape for the Delta contrib suites: Spark 3.5 is the profile the
+ # MinIO and feature-off jobs already pin, so it is the one the queue runs;
+ # 4.0 and 4.1 run nightly. One label opts a pull request into all three.
+ "delta_3_5": ["queue", "label:run-delta-tests"],
Review Comment:
> Could `delta_3_5` move to the nightly tier with 4.0 and 4.1, still behind `run-delta-tests`? And could the Spark 3.5 strict warnings compile in `pr_build_linux.yml` add the `delta` profile, so a change that breaks the contrib build fails on its own PR?
Both in 3062796c6. `delta_3_5` is nightly now, still behind `run-delta-tests`, and the strict warnings job runs `-Pspark-3.5,delta -Pstrict-warnings`. Since nothing in the queue runs the contrib any more, `contrib/delta-spark/**` also routes to the Linux PR build, so a change under the contrib alone gets that compile too. The cost is that a contrib-only PR runs the whole Linux pipeline; a narrower output that gates only the strict job would avoid that if you'd rather. The workflows README, `ci.md` and `check-ci-config.py` follow the new tiering, and `ci.md` lists `run-delta-tests` among the labels to reach for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-6122235744
> The `CometNativeScan` one needs a decision on your side, since `buildNativeScanCommon` and `populateScanConfFlags` still take `firstFileUri` while `convert` now uses `extractScanObjectStoreOptions` over every file.
Merged main in e2d6deb3b. `buildNativeScanCommon` and `populateScanConfFlags` now take the resolved object store options instead of `firstFileUri`, so each caller decides. `CometNativeScan.convert` does what main does: `extractScanObjectStoreOptions` over every selected file, and a fallback with the reason on both the scan and the wrapped scan. The Delta scan runs the same check over its data files. Delta's existing gate already declines data files and deletion vectors whose authority carries userinfo, which covers every ABFS path, so the only Azure authority that can reach the Delta option union is the table root. The union's comment now says so, since #6059 made the ABFS keys global. The Delta suites pass on Spark 3.5 with strict warnings, the MinIO ones included, and so do the object store and ABFS suites on 3.5 and 4.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add native Delta Lake scan contrib module (page/row-group pruning) [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-6050465820
@andygrove @sunchao @comphead hi all. Do you know when we target getting this merged?
There are some follow on tasks I am hoping to work on as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org